# What a bug report states

Testkit files each bug as an issue in GitHub or Gitea. The issue has five sections, names the test that fails, and ends with a facts block.

## From a bug record to an issue

A bug report starts as a bug record. The bug record is a JSON file that your coding agent writes during a [campaign](https://turnstone.cloud/docs/campaigns.md), or that you write yourself. Three commands turn it into an issue.

| Command | What it does |
| --- | --- |
| `testkit bug guide` | Prints the bug record format, the severities, and the cause, impact, and escape lists |
| `testkit bug record <file>` | Checks a bug record, stores it as a draft under `.testkit/bugs/`, and prints the draft id |
| `testkit bug file <id>` | Runs the draft's test and files the bug as an issue |

The commands work with or without an active campaign. Bug filing needs no Turnstone account.

## The five sections

Every issue that Testkit files has five sections.

| Section | What it answers | Where it comes from |
| --- | --- | --- |
| Expected behaviour | What should the product do? | `expected` in the bug record |
| Authority | Where is that written down? | `authority` in the bug record |
| Observed behaviour | What does the product do instead? | `observed` in the bug record |
| Root cause | Which code makes it happen? | `rootCause` in the bug record |
| Reproduce | Which command shows it? | Testkit writes the command from `test.file` |

A report with all five can be checked by someone who was not there. Every field of the bug record is required, so no report lacks one.

## Write the bug record

```sh
npx testkit bug guide
```

The command prints the fields of a bug record, the severities, and the lists of cause, impact, and escape ids.

| Field | Value |
| --- | --- |
| `title` | One line of at most 200 characters that names the broken behaviour. It becomes the issue title. |
| `severity` | `critical`, `high`, `medium`, or `low` |
| `expected` | What the product should do |
| `authority` | The source that says so: documentation, a specification, an API contract, or the code's own stated intent. Name it and quote the part that applies. |
| `observed` | What the product does instead |
| `rootCause` | Why the code behaves this way |
| `test.file` | The Testkit test file that fails because of the bug |
| `test.title` | The failure's title, in one line of at most 200 characters |
| `fault.path` | The source file that holds the mistake |
| `fault.lines` | The first and last faulty line as `[start, end]`, counted from 1. One faulty line is `[41, 41]`. |
| `cause` | One cause id |
| `impact` | One impact id |
| `confidence.cause` | `high` or `low`. How sure the writer is of the cause. |
| `confidence.impact` | `high` or `low`. How sure the writer is of the impact. |
| `escape` | One escape id |

```json title=bug.json
{
	"title": "A refund pays out twice",
	"severity": "high",
	"expected": "The second POST /orders/{id}/refunds for the same order returns 409 and pays nothing.",
	"authority": "`docs/api/refunds.md`, \"A refund is idempotent per order.\"",
	"observed": "The second request returns 201 and creates a second payout.",
	"rootCause": "`src/checkout/refunds.ts` retries the payout call without an idempotency key.",
	"test": {
		"file": "src/checkout/__testkit__/refund.contract.testkit.ts",
		"title": "a refund pays out once"
	},
	"fault": { "path": "src/checkout/refunds.ts", "lines": [41, 58] },
	"cause": "mishandled-error",
	"impact": "financial-error",
	"confidence": { "cause": "high", "impact": "high" },
	"escape": "no-test"
}
```

These rules apply to the fields.

- Write `test.file` and `fault.path` relative to the product directory, the directory that holds `testkit.config.ts`.
- `test.title` must be the failure's title exactly as Testkit reports it. In a Vitest or Playwright file, that is the names of the `describe` blocks around the test and the test's own name, joined by single spaces. In a k6 file, that is the label of the failed assertion.
- A k6 script that fails as a whole has no failure title. Write in `test.title` what the script tests. Testkit then files one issue that covers the whole file.
- Keep credentials, tokens, and passwords out of the record.

### Severities

| Severity | Meaning |
| --- | --- |
| `critical` | The product is unsafe to run or to trust. |
| `high` | A core behaviour is broken with no acceptable workaround. |
| `medium` | A behaviour is broken and a workaround exists, or the reach is limited. |
| `low` | A minor defect with little consequence. |

### Cause, impact, and escape

Each of these fields takes one id from a list. `testkit bug guide` prints each id's definition and the rule that separates it from its neighbours.

| Field | Question | Ids |
| --- | --- | --- |
| `cause` | What kind of mistake is in the code? | `missing-validation`, `mishandled-error`, `race-condition`, `unbounded-or-inefficient-work`, `incorrect-query-or-selection`, `missing-access-control`, `null-or-default-mishandling`, `stale-state`, `interface-mismatch`, `omitted-step`, `incorrect-logic`, `other` |
| `impact` | What is at stake for the business? | `unauthorized-access`, `financial-error`, `audit-or-compliance-gap`, `data-loss-or-corruption`, `workflow-breakdown`, `availability-or-performance`, `blocked-operation`, `wrong-information-shown`, `other` |
| `escape` | Why did the existing tests not catch the bug? | `no-test`, `untested-input`, `untested-interleaving`, `wrong-assertion`, `test-double`, `other` |

A bug gets one id for each field. The impact list is in precedence order, and the first id that applies wins. Use `other` when none fits.

## The test that comes with it

The bug record names one test. The test's name states the expected behaviour, so the name is true once the product is fixed.

```ts title=src/checkout/__testkit__/refund.contract.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { contractScenario, expect } from "@gfdelarue/testkit/node-vitest/contract";

export const testkit = defineFile({ engine: "node-vitest" });

contractScenario("a refund pays out once", async ({ http }) => {
	const first = await http().post("/orders/8841/refunds");
	const second = await http().post("/orders/8841/refunds");
	expect(first.status).toBe(201);
	expect(second.status).toBe(409);
});
```

Against the broken product, the test fails on the line that states the expected behaviour.

```console $ npx testkit run --file src/checkout/__testkit__/refund.contract.testkit.ts
[1/1] FAIL api contract src/checkout/__testkit__/refund.contract.testkit.ts 0s
    AssertionError: expected 201 to be 409 // Object.is equality at
    /repo/src/checkout/__testkit__/refund.contract.testkit.ts:10:24
```

## The issue Testkit files

The issue's title is the `title` of the bug record.

### Labels

| Label | Meaning |
| --- | --- |
| `testkit` | Filed by Testkit with a regression test. Every issue that Testkit files carries it. |
| `severity:<level>` | The bug's severity, such as `severity:high` |
| `area:<id>` | The product area that holds the fault. The label's description is the area's name. |

Testkit creates a label that does not exist yet and never changes one that does. An issue carries an `area:<id>` label when [`areas`](https://turnstone.cloud/docs/configuration.md#areas) is set.

Severity and area are labels because people re-triage both. To change a bug's severity or area, replace the label. An issue with two `severity:` labels counts as having none, and the same holds for `area:` labels.

### Body

The body has the five sections, three lines that locate the bug, and the facts block.

````text title=The body of the issue for bug.json
## Expected behaviour

The second POST /orders/{id}/refunds for the same order returns 409 and pays nothing.

## Authority

`docs/api/refunds.md`, "A refund is idempotent per order."

## Observed behaviour

The second request returns 201 and creates a second payout.

## Root cause

`src/checkout/refunds.ts` retries the payout call without an idempotency key.

## Reproduce

```sh
npx testkit run --file src/checkout/__testkit__/refund.contract.testkit.ts
```

Test: `src/checkout/__testkit__/refund.contract.testkit.ts`, "a refund pays out once".
Found at `4f0c2d1e9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d`.
Fault: `src/checkout/refunds.ts`, lines 41 to 58.

<details>
<summary>Testkit facts</summary>

<!-- testkit-facts:start -->

```json
{
  "v": 1,
  "cause": "mishandled-error",
  "impact": "financial-error",
  "confidence": { "cause": "high", "impact": "high" },
  "escape": "no-test",
  "caughtBy": "campaign",
  "campaign": "refunds-2026-10",
  "testType": "contract",
  "test": {
    "file": "src/checkout/__testkit__/refund.contract.testkit.ts",
    "title": "a refund pays out once"
  },
  "foundAt": "4f0c2d1e9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d",
  "fault": { "path": "src/checkout/refunds.ts", "lines": [41, 58] },
  "area": { "id": "checkout", "name": "Checkout", "share": 0.14 },
  "introducedBy": "9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b",
  "introducedMethod": "blame"
}
```

<!-- testkit-facts:end -->

</details>
````

The file names, the commits, and the campaign in this body are examples.

The reproduce command runs from the repo root. When `testkit.config.ts` is in a subdirectory, the command carries `--dir <directory>`, and the `--file` path is relative to that directory. The `Test:` and `Fault:` lines always give paths from the repo root. A fault on one line reads `line 41`.

### Facts block

The facts block is one JSON object between two marker comments. It sits in a collapsed section named **Testkit facts**, and tools read it. [Facts](https://turnstone.cloud/docs/facts.md) lists what Turnstone reads from it.

| Field | Written | Meaning |
| --- | --- | --- |
| `v` | Always | The format version, `1` |
| `cause` | Always | The kind of mistake in the code |
| `impact` | Always | What is at stake for the business |
| `confidence.cause` | Always | The writer's confidence in `cause`, `high` or `low` |
| `confidence.impact` | Always | The writer's confidence in `impact`, `high` or `low` |
| `escape` | Always | Why the existing tests did not catch the bug |
| `caughtBy` | Always | `campaign` when a campaign was active when the bug was recorded, and `manual` otherwise |
| `campaign` | With `caughtBy: "campaign"` | The id of the campaign that caught the bug |
| `testType` | Always | The type of the regression test, from its file suffix |
| `test.file` | Always | The regression test file |
| `test.title` | When the issue covers one test | The failure's title |
| `foundAt` | Always | The commit the test failed at |
| `fault.path` | Always | The source file that holds the mistake |
| `fault.lines` | Always | The first and last faulty line at `foundAt`, counted from 1 |
| `area.id` | When an area claims `fault.path` | The area's key in `areas` |
| `area.name` | With `area.id` | The area's name when the bug was filed |
| `area.share` | With `area.id` | The area's share of the repo's source bytes at `foundAt`, from 0 to 1 |
| `introducedBy` | When the blame resolves | The newest commit among the fault lines at `foundAt` |
| `introducedMethod` | With `introducedBy` | How Testkit found `introducedBy`. The value is `blame`. |

All paths in the block use `/` and are relative to the repo root. Commits are full commit ids.

The package ships the full format, with the rules for a tool that reads it, as `node_modules/@gfdelarue/testkit/docs/bug-records.md`.

## What `testkit bug file` checks

`testkit bug file` runs the test at the checked-out commit and files the issue only when the test fails. It files nothing in these cases.

| `testkit bug file` stops when | What to do |
| --- | --- |
| Tracked files have uncommitted changes | Commit or discard the changes. Then run the command again. |
| No remote branch contains the commit | Push the commit. Then run the command again. |
| An open issue already covers the test | Read that issue. Testkit keeps one open bug for each test. |
| `areas` is set, and no area claims the fault's file or two areas do | The message names the file. Change `areas` in `testkit.config.ts` so that one area claims the file. |
| `test.title` is not one of the failure titles that the file reports | The message lists the titles that a bug record can use. |

The token needs permission to create issues and labels. [Set the token](https://turnstone.cloud/docs/campaigns.md#set-the-token) lists the variables that Testkit reads it from.

### Secrets

Testkit redacts the secrets it recognises in the title, in every section, and in `test.title`. It does so when it records the bug and again when it files the issue. Testkit cannot recognise every secret, so keep credentials, tokens, and passwords out of the record.

Testkit does not rewrite a path or an area's name. It stops in these cases.

- `testkit bug record` rejects a record whose `test.file` or `fault.path` holds text that Testkit redacts.
- `testkit bug file` stops when the area's name or the reproduce command holds such text.
- Testkit files no bug under a failure title that holds such text. Rename the test, or give the assertion a label.

## Known failures

A known failure is a failing test that an open issue with the `testkit` label covers. `bugs.knownFailures` in `testkit.config.ts` decides what a known failure does to a run.

| Value | What a run does |
| --- | --- |
| `"fail"` | Every failing test fails the run. This is the default. |
| `"allow"` | A run whose failures are all known passes. One new failure fails the run. |

```ts title=testkit.config.ts
export default defineConfig({
	bugs: { knownFailures: "allow" },
});
```

Use `"allow"` to keep CI passing while a filed bug waits for its fix. These rules apply.

- An issue covers a failure when its facts block gives the failing file as `test.file` and the failure's title as `test.title`. An issue with no `test.title` covers the whole file.
- Closing an issue makes its test count again.
- A failure that names no test is always new. A crash, a timeout, and a file that does not load are such failures.
- When Testkit cannot read the issues, every failure is new.
- A run that fails outside its tests fails under both values.

With `"allow"`, the run reads the open issues, so `testkit run` needs a token. [Pass a run whose failures are all filed](https://turnstone.cloud/docs/ci.md#pass-a-run-whose-failures-are-all-filed) shows the CI step.

## Where the evidence lives

The issue points to evidence where it already is. Do not copy it somewhere new.

| Evidence | Where it lives |
| --- | --- |
| The failing test | Your repo, at the commit on the issue's `Found at` line |
| The command that shows the failure | The issue's **Reproduce** section |
| The faulty lines | Your repo, at the path and lines on the issue's `Fault:` line |
| The full run | The CI job's uploaded `.testkit/results` |

[Read a run's results](https://turnstone.cloud/docs/results.md) shows how to find a file's output in a run.

Turnstone holds none of this evidence. [What Turnstone keeps](https://turnstone.cloud/docs/cloud.md) lists the facts that Turnstone stores.

## What a report never does

- It never skips the failing test.
- It never asserts the broken behaviour so that the test passes.
- It never reports a failure without a cause. A failing test with no root cause is a lead, and the agent keeps working on it.