# Read the manual.

Testkit is a test runner and CLI for coding agents and CI. These pages cover Testkit 0.4.1.

## Pages

### Start

- [Get started](https://turnstone.cloud/quickstart): Five steps from install to a merge gate.

### Write tests

- [Test files](https://turnstone.cloud/docs/test-files.md): Where a test lives and what its name means.
- [Contract tests](https://turnstone.cloud/docs/contract-tests.md): Call your running service from Node.
- [HTTP tests](https://turnstone.cloud/docs/http-tests.md): Drive routes with k6, signed in as named actors.
- [Browser tests](https://turnstone.cloud/docs/browser-tests.md): Assert on a real page in Chromium.
- [Database tests](https://turnstone.cloud/docs/database-tests.md): Run SQL against a Postgres that Testkit builds.

### Run

- [CI](https://turnstone.cloud/docs/ci.md): One command decides whether a pull request merges.
- [Results](https://turnstone.cloud/docs/results.md): Where a run's evidence lands and how to read it.
- [CLI](https://turnstone.cloud/docs/cli.md): Every command and flag.
- [Configuration](https://turnstone.cloud/docs/configuration.md): Every field of `testkit.config.ts`.

### Hunt bugs

- [Campaigns](https://turnstone.cloud/docs/campaigns.md): Give your agent a target and collect proven bugs.
- [Bug reports](https://turnstone.cloud/docs/bug-reports.md): The five things every issue states.

### Turnstone

- [Turnstone](https://turnstone.cloud/docs/cloud.md): What the dashboard keeps, and what it never stores.
- [Connect GitHub](https://turnstone.cloud/docs/connect-github.md): Connect a repository, and see the access Turnstone asks for.
- [Connect Gitea](https://turnstone.cloud/docs/connect-gitea.md): Add your Gitea host, then connect a repository.
- [Sync](https://turnstone.cloud/docs/sync.md): When Turnstone reads your issues, and what it reads.
- [Facts](https://turnstone.cloud/docs/facts.md): Each fact about a bug, and where Turnstone reads it.

## Four parts, and one is optional

- **Your repo** holds the tests and the config. Policy goes in `testkit.config.ts`.
- **The CLI** runs in your repo. Your coding agent uses it to write tests and hunt bugs, and CI uses it to run them. It works without a server.
- **GitHub or Gitea** gets everything a person acts on: issues, checks and statuses, and pull requests that add tests.
- **Turnstone** is an optional, read-only dashboard. It receives run facts from GitHub or Gitea and charts them.

The first three are free. None of them asks for an account, a licence key, or a payment. Turnstone has a free tier.

## What you need

| Requirement | Version |
| --- | --- |
| Node | `>=22.15.0 <23.0.0` or `>=24.14.1 <25.0.0` |
| npm | `>=11.6.0 <12.0.0` |
| Docker | Only for a service that declares `database.postgres(...)` |

```sh
npm install --save-dev @gfdelarue/testkit
```

If Testkit is not in your repo yet, start with [Get started](https://turnstone.cloud/quickstart). It has five steps and ends with a merge gate.

## Hand these pages to your agent

Every page here has a Markdown copy. Add `.md` to the page's URL, or press **Copy page** at the top of the page.

| URL | Contents |
| --- | --- |
| [`/llms.txt`](https://turnstone.cloud/llms.txt) | Every page, with a one-line summary of each |
| [`/llms-full.txt`](https://turnstone.cloud/llms-full.txt) | Every page in one file |
| [`/docs/ci.md`](https://turnstone.cloud/docs/ci.md) | One page, in this case the CI guide |

The package ships its own reference too. Your agent can read these files without a network call.

| File | Contents |
| --- | --- |
| `node_modules/@gfdelarue/testkit/README.md` | The CLI, the engines, and the CI gate |
| `node_modules/@gfdelarue/testkit/testkit-config-schema.md` | Every field of `testkit.config.ts` |
| `node_modules/@gfdelarue/testkit/docs/bug-records.md` | The format of the issues that Testkit files |

---

# Test files

Where a test file lives, what its name means, and the one declaration every file needs.

## Put tests beside the code

Testkit discovers test files in `__testkit__` directories. Put each `__testkit__` directory beside the code it tests.

```text title=A service with four suites
src/
├── __testkit__/
│   └── health.contract.testkit.ts
├── checkout/
│   └── __testkit__/
│       ├── health.int.testkit.ts
│       └── refund.contract.testkit.ts
└── db/
    └── __testkit__/
        └── refunds.dal.testkit.ts
```

Keep each `__testkit__` directory flat. When a narrower piece of code needs its own tests, add another `__testkit__` directory beside that code. Shared helpers can live in `__testkit__/helpers/`.

The suite name comes from the directory that holds `__testkit__`, and the test type goes in front of it. In the tree above, `refund.contract.testkit.ts` belongs to the suite `contract:checkout`, and `refunds.dal.testkit.ts` belongs to `dal:db`.

## The filename picks the test type

| Suffix | Type | Use it for |
| --- | --- | --- |
| `*.contract.testkit.ts` | `contract` | A service boundary that needs Node APIs, fixtures, or file IO |
| `*.int.testkit.ts` | `int` | A route, service, queue, or job at its runtime boundary |
| `*.e2e.testkit.ts` | `e2e` | A flow that spans services without a browser |
| `*.ui.testkit.ts` | `ui` | A page or interaction where the browser is part of the assertion |
| `*.dal.testkit.ts` | `dal` | Data access and persistence |
| `*.data.testkit.ts` | `data` | The shape, cost, volume, or latency of data in a real flow |
| `*.load.testkit.ts` | `load` | Load, soak, or pressure |

## Every file declares its engine

```ts
import { defineFile } from "@gfdelarue/testkit";

export const testkit = defineFile({ engine: "node-vitest" });
```

Import `defineFile` from the root package and pass it an object literal. Testkit never infers the engine from the filename or from the imports. Discovery fails on a missing declaration, on a computed engine value, and on a pair that this table does not list.

| Test type | Engine | Runner profile | Guide |
| --- | --- | --- | --- |
| `contract` | `node-vitest` | `node-vitest-contract` | [Write a contract test](https://turnstone.cloud/docs/contract-tests.md) |
| `int`, `e2e`, `load` | `k6` | `k6-http` | [Write an HTTP test](https://turnstone.cloud/docs/http-tests.md) |
| `ui` | `playwright` | `playwright-ui` | [Write a browser test](https://turnstone.cloud/docs/browser-tests.md) |
| `dal` | `k6` | `k6-dal` | [Test the database](https://turnstone.cloud/docs/database-tests.md) |
| `data` | `node-vitest` | `node-vitest-data` | [Data tests](#data-tests) |

## See what Testkit found

`testkit discover` lists every test file without running one.

```console $ npx testkit discover
Summary 6 files · 6 active · 0 skipped · 6 suites · 1 services

api (6 files)
├── UI UI
│   └── Web (1 files · playwright · playwright-ui)
│       └── Checkout
├── INT Integration
│   ├── Account (1 files · k6 · k6-http)
│   │   └── Profile
│   └── Checkout (1 files · k6 · k6-http)
│       └── Health
├── CONTRACT Contract
│   ├── Checkout (1 files · node-vitest · node-vitest-contract)
│   │   └── Refund
│   └── Src (1 files · node-vitest · node-vitest-contract)
│       └── Health
└── DAL DAL
    └── Db (1 files · k6 · k6-dal)
        └── Refunds
```

Add `--output-mode verbose` for paths and file ids, or `--json` for a document a script can read.

## Skip a file with a reason

```ts
export const testkit = defineFile({
	engine: "node-vitest",
	skip: "The payout sandbox is down until Friday",
});
```

The run prints the file and the reason.

```console $ npx testkit run contract
SKIP api contract src/checkout/__testkit__/payout.contract.testkit.ts 0s The payout sandbox is down until Friday
[1/2] PASS api contract src/checkout/__testkit__/refund.contract.testkit.ts 1s
[2/2] PASS api contract src/__testkit__/health.contract.testkit.ts 1s

Summary
Result: FAILED
Passed: 2
Failed: 0
Skipped: 1
Not run: 0
Files: 3
Duration: 10s
Run ID: 1791890889522-1447900-822d9269
```

> In Testkit 0.4.1, a run that skips a file exits 1, even when every other file passes. To get a passing run, select the other files with `--file`, `--suite`, or a type.

To skip a whole suite, or to keep skip rules in one place, use the service's [`skip`](https://turnstone.cloud/docs/configuration.md#skip) field. `testkit run --ignore-skip-rules` runs skipped files anyway.

## Keep two files apart with a lock

```ts
export const testkit = defineFile({
	engine: "k6",
	locks: ["global-worker-loop"],
});
```

Files that name the same lock do not run at the same time. Use a lock when two files change the same shared state and `--task-concurrency` is above 1.

## Run Node code before a file

A test file can have a sibling setup module with the same name and a `.setup.mjs` ending. Testkit runs its `setup` export before it starts the engine.

```js title=src/checkout/__testkit__/refund.contract.testkit.setup.mjs
export async function setup({ sidecars }) {
	const capture = await sidecars.httpCapture({ path: "/webhook" });
	return {
		data: { webhookUrl: capture.url, requestsUrl: capture.requestsUrl },
	};
}
```

The setup function receives `deadlineMs`, `signal`, `env`, `leaseDir`, `productDir`, `serviceName`, `testFile`, and `sidecars`. Return plain JSON, or `{ data, cleanup }`. Testkit passes `data` to the engine as `setupData` and calls `cleanup` when the file finishes. `data` can hold at most 1 MiB of JSON.

If setup throws, the file fails and the run names the setup module.

```console $ npx testkit run --file src/checkout/__testkit__/refund.contract.testkit.ts
[1/1] FAIL api contract src/checkout/__testkit__/refund.contract.testkit.ts 0s
    Runtime node setup "src/checkout/__testkit__/refund.contract.testkit.setup.mjs" failed: The
    payment sandbox refused the key
```

Setup runs in a Node worker thread with the permissions of the host process. It is not a sandbox.

## Data tests

A data test measures data moving through a real flow: its shape, volume, cost, and latency. It runs in Node under Vitest.

```ts
import { defineFile } from "@gfdelarue/testkit";
import { dataScenario } from "@gfdelarue/testkit/node-vitest/data";
import { expect } from "vitest";

export const testkit = defineFile({ engine: "node-vitest" });
```

Register every case with `dataScenario(...)`. The data runner rejects a case registered any other way, and `testkit lint` reports the import. A data test must record evidence through the context that `dataScenario` provides. A lone `count(...)` or `event(...)` call is not enough. For plain Node assertions, [write a contract test](https://turnstone.cloud/docs/contract-tests.md).

---

# Write a contract test

A contract test runs in Node under Vitest. Testkit starts your service, gives the test an HTTP client that points at it, and stops the service when the run ends.

## Before you start

You need one service in `testkit.config.ts`. [Get started](https://turnstone.cloud/quickstart) adds it in step two.

## Write the test

Create the file in a `__testkit__` directory beside the code it tests.

```ts title=src/checkout/__testkit__/refund.contract.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { contractScenario, expect } from "@gfdelarue/testkit/node-vitest/contract";

export const testkit = defineFile({ engine: "node-vitest" });

contractScenario("a refund pays out once", async ({ http }) => {
	const first = await http().post("/orders/8841/refunds");
	const second = await http().post("/orders/8841/refunds");
	expect(first.status).toBe(201);
	expect(second.status).toBe(409);
});
```

`contractScenario` registers one case. Name the case after the behaviour you expect, so the name is true when the product is right.

## Run it

```console $ npx testkit run contract
[1/2] PASS api contract src/checkout/__testkit__/refund.contract.testkit.ts 1s
[2/2] PASS api contract src/__testkit__/health.contract.testkit.ts 1s

Summary
Result: PASSED
Passed: 2
Failed: 0
Skipped: 0
Not run: 0
Files: 2
Duration: 8s
Run ID: 1791890142106-1343177-e5c04590
```

To run one file, pass `--file src/checkout/__testkit__/refund.contract.testkit.ts`.

## Read a failure

When the service pays the refund twice, the second assertion fails. The run prints the assertion and the line that made it, then the rest of the stack.

```console $ npx testkit run contract
[1/2] FAIL api contract src/checkout/__testkit__/refund.contract.testkit.ts 1s
    AssertionError: expected 201 to be 409 // Object.is equality at
    /repo/src/checkout/__testkit__/refund.contract.testkit.ts:10:24
[2/2] PASS api contract src/__testkit__/health.contract.testkit.ts 1s

Summary
Result: FAILED
Passed: 1
Failed: 1
Skipped: 0
Not run: 0
Files: 2
Duration: 10s
Run ID: 1791890523760-1353387-5113dd97
```

The command exits 1. Leave the test failing until the product is fixed. A test that fails for a real bug is the evidence a [bug report](https://turnstone.cloud/docs/bug-reports.md) needs.

## What a scenario receives

| Field | What it is |
| --- | --- |
| `http(options?)` | Returns an HTTP client for the service. `options` takes `baseUrl` and `defaultHeaders`. |
| `baseUrl` | The URL of the running service |
| `env` | The service's declared environment values, read-only |
| `fixture(name)` | A fixture file, with `text()`, `json()`, `buffer()`, `sha256`, and `sizeBytes` |

The client has `get`, `post`, `put`, `patch`, `delete`, and `request(method, path, body?, headers?)`. `get` takes a path and optional headers. The others take a path, an optional body, and optional headers.

Every response has the same fields.

| Field | Type |
| --- | --- |
| `status` | `number` |
| `headers` | `Record<string, string>` |
| `body` | `string` |
| `json()` | Parses `body` |
| `method`, `path`, `url` | `string` |

## Use Vitest where you need it

`expect` is Vitest's `expect`. A contract file is a native Vitest suite, so `test.skip(...)` and `test.todo(...)` from `vitest` work beside `contractScenario`.

To give one scenario its own time limit, pass options as the second argument.

```ts
contractScenario("the export finishes", { timeoutMs: 20_000 }, async ({ http }) => {
	const response = await http().post("/exports");
	expect(response.status).toBe(202);
});
```

---

# Write an HTTP test

An HTTP test drives your running service with k6. Use it for int, e2e, and load files.

## Write the test

```ts title=src/checkout/__testkit__/health.int.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { defineHttpSuite } from "@gfdelarue/testkit/k6";
import { check } from "k6";

export const testkit = defineFile({ engine: "k6" });

const suite = defineHttpSuite(({ rawReq }) => {
	const response = rawReq.get("/health");
	check(response, {
		"health returns 200": (r) => r.status === 200,
	});
});

export const options = suite.options;
export const setup = suite.setup;
export default suite.exec;
```

The last three lines are required. A k6 module exports `options`, `setup`, and a default function, and Testkit does not write them for you.

`rawReq` sends a request with no session. The path is relative to the service's base URL.

## Run it

```console $ npx testkit run int
[1/2] PASS api int src/account/__testkit__/profile.int.testkit.ts 0s
[2/2] PASS api int src/checkout/__testkit__/health.int.testkit.ts 0s

Summary
Result: PASSED
Passed: 2
Failed: 0
Skipped: 0
Not run: 0
Files: 2
Duration: 5s
Run ID: 1791890151637-1344059-9ba3838f
```

## Name the check that failed

A failed k6 `check` fails the file, but the run prints only the threshold.

```console $ npx testkit run int
[1/2] PASS api int src/account/__testkit__/profile.int.testkit.ts 0s
[2/2] FAIL api int src/checkout/__testkit__/health.int.testkit.ts 0s
    k6 thresholds failed: checks(rate==1.0)
```

Testkit's `expect.value` takes a value, a predicate, and a label. When the predicate returns false, the run prints the label.

```ts
import { defineHttpSuite, expect } from "@gfdelarue/testkit/k6";

const suite = defineHttpSuite(({ rawReq }) => {
	const response = rawReq.get("/health");
	expect.value(response.status, (status) => status === 200, "health returns 200");
});
```

```console $ npx testkit run int
[1/2] PASS api int src/account/__testkit__/profile.int.testkit.ts 0s
[2/2] FAIL api int src/checkout/__testkit__/health.int.testkit.ts 0s
    health returns 200
    k6 thresholds failed: checks(rate==1.0)
```

## Sign in as named actors

Put the sign-in code in a helper beside the tests. An auth adapter has two functions. `setup` signs every actor in once and returns their sessions. `headers` turns a session into request headers.

```ts title=src/account/__testkit__/helpers/session.ts
import type { AuthAdapter, AuthSessionBundle } from "@gfdelarue/testkit/k6";

export function sessionAuth(actorNames: string[]): AuthAdapter<AuthSessionBundle> {
	return {
		setup({ env, http }) {
			const actors = Object.fromEntries(
				actorNames.map((actorName, actorIndex) => {
					const login = http.post(`${env.BASE}/login`, JSON.stringify({ name: actorName }));
					const token = String(JSON.parse(String(login.body)).token);
					return [actorName, { actorIndex, actorName, session: { token } }];
				}),
			);
			return { actors, primaryActor: actorNames[0] };
		},
		headers(bundle, { actor }) {
			const session = bundle?.actors[actor ?? bundle.primaryActor ?? ""]?.session;
			return session ? { Authorization: `Bearer ${session.token}` } : {};
		},
	};
}
```

Pass the adapter to `defineHttpSuite`. `req` sends requests as the primary actor, and `req.as(name)` sends them as another one.

```ts title=src/account/__testkit__/profile.int.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { defineHttpSuite } from "@gfdelarue/testkit/k6";
import { check } from "k6";

import { sessionAuth } from "./helpers/session.js";

export const testkit = defineFile({ engine: "k6" });

const suite = defineHttpSuite({ auth: sessionAuth(["reviewer", "outsider"]) }, ({ rawReq, req }) => {
	check(req.get("/me"), {
		"the reviewer reads their profile": (r) => r.status === 200,
	});
	check(req.as("outsider").get("/me"), {
		"an outsider is refused": (r) => r.status === 403,
	});
	check(rawReq.get("/me"), {
		"a request with no session is refused": (r) => r.status === 401,
	});
});

export const options = suite.options;
export const setup = suite.setup;
export default suite.exec;
```

Three checks cover the signed-in user, the wrong user, and no user. Most access bugs sit in the second one.

## Wait for work to finish

`waitFor` calls a function until a predicate accepts its result. It spends the file's time budget, so do not set a timeout of your own.

```ts
import { parse, waitFor } from "@gfdelarue/testkit/k6";

const response = waitFor(
	() => req.get("/api/v1/jobs/123"),
	(res) => parse.json(res).data?.status === "completed",
	{ description: "job 123 to complete" },
);
```

Set the budget for every file with [`execution.fileTimeoutSeconds`](https://turnstone.cloud/docs/configuration.md#execution), or for one run with `--file-timeout-seconds`.

## Use k6 directly

Native k6 imports work in every HTTP test.

```ts
import { check, group } from "k6";
import http from "k6/http";
import { Rate, Trend } from "k6/metrics";
```

To set k6 options such as `vus` and `iterations`, name an HTTP profile in [`profiles.http`](https://turnstone.cloud/docs/configuration.md#profileshttp) and pass `{ profile: "name" }` to `defineHttpSuite`. A `load` file uses the same suite shape as an `int` file.

---

# Write a browser test

A browser test opens your running app in Chromium with Playwright. Use it when the browser is part of the assertion.

## Install the browser once

Testkit runs browser tests in a Chromium that it manages. A test run never downloads a browser, so install it first.

```sh
npx testkit setup playwright
```

On CI and on a fresh host, add `--with-deps`. It installs the system libraries that Chromium needs.

## Write the test

```ts title=src/web/__testkit__/checkout.ui.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { resolveFrontendBaseUrl } from "@gfdelarue/testkit/playwright";
import { expect, test } from "@playwright/test";

export const testkit = defineFile({ engine: "playwright" });

test("the checkout page shows the order total", async ({ page }) => {
	await page.goto(new URL("/checkout", resolveFrontendBaseUrl()).href);
	await expect(page.getByRole("heading", { name: "Checkout" })).toBeVisible();
	await expect(page.getByText("Total")).toBeVisible();
});
```

`resolveFrontendBaseUrl()` returns the URL of the service that Testkit started.

Import `test` and `expect` from `@playwright/test`. You do not install that package. Testkit ships it and pins its version, so a copy of Playwright in your own `node_modules` does not change what the test runs on.

## Run it

```console $ npx testkit run ui
[1/1] PASS api ui src/web/__testkit__/checkout.ui.testkit.ts 4s

Summary
Result: PASSED
Passed: 1
Failed: 0
Skipped: 0
Not run: 0
Files: 1
Duration: 12s
Run ID: 1791890876609-1444310-4b62c131
```

## Read a failure

When the page loses its heading, the run prints Playwright's error and the locator that found nothing.

```console $ npx testkit run ui
[1/1] FAIL api ui src/web/__testkit__/checkout.ui.testkit.ts 9s
    Error: expect(locator).toBeVisible() failed Locator: getByRole('heading', { name: 'Checkout' })
    Expected: visible Timeout: 5000ms Error: element(s) not found Call log:  - Expect "toBeVisible
```

The run keeps more than it prints. List what it kept for the file.

```sh
npx testkit artifacts src/web/__testkit__/checkout.ui.testkit.ts
```

For a failed browser test, the list includes `playwright-failure-details` and Playwright's `error-context` attachment. [Read a run's results](https://turnstone.cloud/docs/results.md) shows how to print one.

## Point the test at another target

`resolveFrontendBaseUrl()` takes the first target it finds, in this order.

| Order | Browser target | API target |
| --- | --- | --- |
| 1 | The `frontendBaseUrl` option you pass | The `backendBaseUrl` option you pass |
| 2 | `ui.frontendBaseUrl` in `testkit.config.ts` | `ui.backendBaseUrl` in `testkit.config.ts` |
| 3 | `TESTKIT_FRONTEND_BASE_URL` | `TESTKIT_BACKEND_BASE_URL` |
| 4 | `BASE_URL`, which Testkit sets for the run | `API_BASE_URL` |

`resolveBackendBaseUrl()` reads the API column.

Testkit refuses a target that is not on loopback. To allow one, set `ui.allowRemote` to `true`.

## Helpers

Import these from `@gfdelarue/testkit/playwright`.

| Helper | Returns |
| --- | --- |
| `resolveFrontendBaseUrl(options?)` | The browser target |
| `resolveBackendBaseUrl(options?)` | The API target |
| `createUiSandbox(options?)` | `{ test, expect }`, where `test` adds a `testkitSandbox` fixture with `id`, `page`, `frontendBaseUrl`, and `backendBaseUrl` |
| `createAuthedApiClient(request?, options?)` | A Playwright `APIRequestContext` for the API target |
| `waitForUiSettled(page, options?)` | A promise that resolves when the page has settled |
| `waitForNetworkIdle(page, options?)` | A promise that resolves when the network is idle |
| `waitForElementStable(page, selector, options?)` | A promise that resolves when the element stops changing |
| `waitForImagesLoaded(page, containerSelector?)` | A promise that resolves when the images have loaded |

The three `waitFor...` helpers that take `options` accept `frames` and `timeout`.

## Keep each file small

`testkit lint` checks the shape of browser test files. By default, a file can have at most 220 lines and 8 tests. Change the limits with `lint.ui.maxLines` and `lint.ui.maxTests` in [`testkit.config.ts`](https://turnstone.cloud/docs/configuration.md#lint).

---

# Test the database

A database test runs SQL against a local Postgres that Testkit builds from your schema. Use it for constraints, queries, and anything else the database decides.

## Before you start

Start Docker. Testkit runs Postgres in a container. It runs `psql` and `pg_dump` in a container too, so you do not need Postgres on the host.

## Declare the database

Add `database.postgres(...)` to the service that owns the schema.

```ts title=testkit.config.ts
import { app, database, defineConfig } from "@gfdelarue/testkit/config";

export default defineConfig({
	services: {
		api: app.node({
			build: null,
			start: "npm start",
			port: 4392,
			database: database.postgres({
				template: {
					inputs: ["db/schema.sql"],
					migrate: [{ kind: "sql-file", path: "db/schema.sql" }],
				},
			}),
		}),
	},
});
```

```sql title=db/schema.sql
CREATE TABLE refunds (
	id text PRIMARY KEY,
	order_id text NOT NULL UNIQUE,
	amount_cents integer NOT NULL
);
```

Testkit builds a template database from the `migrate` steps and caches it. It rebuilds the template when a file in `inputs` changes.

The service process receives its database as `DATABASE_URL`.

## Write the test

```ts title=src/db/__testkit__/refunds.dal.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { defineDalSuite, expect } from "@gfdelarue/testkit/k6";

export const testkit = defineFile({ engine: "k6" });

const suite = defineDalSuite(({ db }) => {
	db.exec("INSERT INTO refunds (id, order_id, amount_cents) VALUES ('r1', '8841', 1200)");
	const error = expect.captureError(() => {
		db.exec("INSERT INTO refunds (id, order_id, amount_cents) VALUES ('r2', '8841', 1200)");
	});
	expect.error.captured(
		error,
		(caught) => String(caught).includes("refunds_order_id_key"),
		"the database rejects a second refund",
	);
	const rows = db.query("SELECT id FROM refunds WHERE order_id = '8841'");
	expect.rowCount(rows, 1, "one refund is stored");
});

export const options = suite.options;
export const setup = suite.setup;
export default suite.exec;
```

`db.exec(sql)` runs a statement. `db.query(sql)` returns the rows.

`expect.captureError` runs a function and returns what it threw. `expect.error.captured` takes that error, a predicate, and a label. The predicate must be a function.

The last three lines are required, as in an [HTTP test](https://turnstone.cloud/docs/http-tests.md).

## Run it

```console $ npx testkit run dal
[1/1] PASS api dal src/db/__testkit__/refunds.dal.testkit.ts 0s

Summary
Result: PASSED
Passed: 1
Failed: 0
Skipped: 0
Not run: 0
Files: 1
Duration: 4s
Run ID: 1791890158174-1344714-af8dd695
```

By default, Testkit runs the `dal` files for one service database one at a time.

## Read a failure

When an `expect` helper fails, the run prints what it expected and what it got.

```console $ npx testkit run dal
[1/1] FAIL api dal src/db/__testkit__/refunds.dal.testkit.ts 0s
    k6 thresholds failed: checks(rate==1.0)
    expected 2 row(s), got 1
```

| Helper | Passes when |
| --- | --- |
| `expect.rowCount(rows, n, label)` | The query returned `n` rows |
| `expect.singleRow(rows, label)` | The query returned one row. It returns that row. |
| `expect.noRows(rows, label)` | The query returned no rows |
| `expect.error.captured(error, predicate, label)` | The predicate accepts the captured error |

## Seed rows with fixtures

`defineDalFixtures` gives a file named fixtures. Each fixture seeds its rows once and returns the values the test needs.

```ts title=src/db/__testkit__/seeded.dal.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { defineDalFixtures, defineDalSuite, expect } from "@gfdelarue/testkit/k6";

export const testkit = defineFile({ engine: "k6" });

const fixtures = defineDalFixtures(({ db, fixtureScope }) => ({
	refund() {
		return fixtureScope.seed("refund", "primary", { amountCents: 1200 }, () => {
			const refundId = fixtureScope.id("refund");
			const orderId = fixtureScope.id("order");
			db.exec(`
				INSERT INTO refunds (id, order_id, amount_cents)
				VALUES ('${refundId}', '${orderId}', 1200)
			`);
			return { refundId };
		});
	},
}));

const suite = defineDalSuite({ fixtures }, ({ db, fixtures }) => {
	const refund = fixtures.refund();
	const rows = db.query(`SELECT amount_cents FROM refunds WHERE id = '${refund.refundId}'`);
	expect.rowCount(rows, 1, "the seeded refund is stored");
});

export const options = suite.options;
export const setup = suite.setup;
export default suite.exec;
```

`fixtureScope.seed(kind, key, signature, create)` identifies a fixture by its `kind` and `key`.

- A second call with the same `kind`, `key`, and `signature` returns the first call's value.
- A second call with a different `signature` fails the file.
- Testkit saves the seeded records with the run as a `testkit.dal-fixtures` artifact.

`fixtureScope` also makes values that are stable for a label: `id(label)`, `uuid(label)`, `slug(label)`, `email(label)`, `string(label, options)`, and `token(label)`.

## Remove the database

`testkit destroy` removes the Postgres container of each service. It keeps `.testkit/results`, so earlier run records stay readable.

```console $ npx testkit destroy
Destroyed api
```

The next run that needs the database builds it again.

---

# Gate merges in CI

Run Testkit on every pull request, block the merge when a test fails, and report each run to GitHub or Gitea as a commit status and a pull request comment.

## Add the script

```json title=package.json
{
	"scripts": {
		"testkit": "testkit run"
	}
}
```

## Add the workflow

```yaml title=.github/workflows/testkit.yml
name: Testkit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  testkit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true
```

Three lines need a reason.

- `setup playwright --with-deps` installs Chromium for browser tests. If the repo has no `ui` files, delete the step.
- `if: always()` uploads the results when the run fails. A failed run is the one you want to read.
- `include-hidden-files: true` is required because `.testkit` starts with a dot. Without it, `actions/upload-artifact@v4` uploads nothing.

`ubuntu-latest` has Docker, which [database tests](https://turnstone.cloud/docs/database-tests.md) need.

## What fails the gate

`npm run testkit` exits `0` only when at least one file completed and every required file passed. Each of these makes it exit `1`.

- A test fails.
- A [setup module](https://turnstone.cloud/docs/test-files.md#run-node-code-before-a-file) throws. The summary adds a `Runtime errors` line.
- A file is skipped, even when every other file passes.
- A file is interrupted or does not run.
- The selection matches no file.

```console $ npx testkit run e2e
SKIP api no matching files (types=e2e suites=all files=all)

Summary
Result: FAILED
Passed: 0
Failed: 0
Skipped: 0
Not run: 0
Files: 0
Duration: 0s
Run ID: 1791891434722-1572097-fbd9051c
```

If a job's selection is empty on purpose, add `--allow-empty` to that job. Do not add it to the default command. A gate that passes with no tests is not a gate.

```sh
npm run testkit -- e2e --allow-empty
```

One setting changes the first case. With [`bugs.knownFailures`](https://turnstone.cloud/docs/bug-reports.md#known-failures) set to `"allow"`, a failing test does not fail the run while an open issue covers it.

## Require the check

A failing job does not block a merge until the forge requires it.

- **GitHub.** In the repo's settings, add a branch protection rule or a ruleset for `main`. Turn on **Require status checks to pass**, and select the `testkit` job.
- **Gitea.** In the repo's settings, open **Branches** and edit the protection rule for `main`. Turn on **Enable Status Check**, and select the check.

Gitea Actions uses the same workflow syntax. Put the file in `.gitea/workflows/`, and check that your Gitea version supports each action the workflow uses.

## Report the run to GitHub or Gitea

`testkit ci report` reads the last run in the checkout and posts it in three places.

| Place | What Testkit posts |
| --- | --- |
| The commit | One commit status |
| The job | The step summary |
| The pull request | One comment, which Testkit updates in place |

The report needs no Turnstone account. In GitHub Actions, add the `permissions` block and the report step to the job that runs your tests.

```yaml title=.github/workflows/testkit.yml
jobs:
  testkit:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      issues: read
      pull-requests: write
      statuses: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - run: npx testkit ci report
        if: always()
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true
```

Four parts need a reason.

- The `npm run testkit` step decides whether the job fails. `ci report` exits `0` when it posts the report, and that includes the report of a failed run.
- `if: always()` runs the report step after a failed run.
- The `env` lines pass the job's token to the report. GitHub Actions passes the job's token to a `run` step only when the step sets it. Gitea Actions accepts the same lines.
- `permissions` lets the job's token read the code and the open issues, post the commit status, and write the pull request comment.

After a failed run on a pull request, the report step prints lines like these.

```console $ npx testkit ci report
Status testkit on 4f0c2d1e9b7a: failure, 1 failed: 1 new, 0 known
Pull request comment: created https://github.com/acme/shop/pull/7#issuecomment-31
Step summary: written
```

The commit, the repo, and the comment in this output are examples.

### Commit status

`testkit ci report` posts one commit status on the checked-out commit. When CI checks out a pull request's merge commit, the status goes on the pull request's head commit instead.

| Part | Value |
| --- | --- |
| Context | `testkit`, or the value of `--context` |
| State | `success` when the run passed, or when `bugs.knownFailures` is `"allow"` and every failure is known. `failure` otherwise. |
| Target URL | The CI run's URL, when the CI provider gives one |
| Description | One of the three forms below, in at most 140 characters |

| Run | Description |
| --- | --- |
| Every file passed | `142 files passed` |
| Tests failed | `3 failed: 1 new, 2 known` |
| The run failed outside its tests | `run failed: <reason>` |

The counts are failures. A failure is known when an open issue with the `testkit` label covers it. [Known failures](https://turnstone.cloud/docs/bug-reports.md#known-failures) gives the rule.

### Pull request comment

`testkit ci report` keeps one comment on the run's pull request and updates it in place. The comment names each failure by its file and its title.

The comment's first line is `<!-- testkit-ci-report -->`. Testkit finds its comment by that line.

### Pass the token

Testkit reads the token from the environment.

| Host | Variable |
| --- | --- |
| GitHub | `GITHUB_TOKEN` or `GH_TOKEN` |
| Gitea | `GITEA_TOKEN` first. In a Gitea Actions job, Testkit reads the job's `GITHUB_TOKEN` second. |

With no token set, Testkit asks `git credential fill` for one. The token that `actions/checkout` keeps in the checkout does not answer that request, so set the variable on the step.

### Pull requests from forks

GitHub gives a pull request from a fork a read-only token. `ci report` then writes the step summary and fails when it posts the status. If the repo takes pull requests from forks, add `continue-on-error: true` to the step.

```yaml
- run: npx testkit ci report
  if: always()
  continue-on-error: true
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

### No results for the commit

When the checkout holds no results for its commit, `ci report` posts a `failure` status with this description.

```text
run failed: Testkit found no results for this commit
```

If the commit status reads that way, run `testkit run` before the report in the same checkout.

### Report more than one run

A job that runs Testkit more than once passes `--context <name>` to each report, so each run keeps its own status and its own comment. The default context is `testkit`.

The jobs in [Split a slow gate](#split-a-slow-gate) report on the same commit and the same pull request. Give each of those jobs its own context too.

```yaml
- run: npx testkit ci report --context testkit-${{ matrix.type }}
  if: always()
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

With a context other than `testkit`, the comment's first line is `<!-- testkit-ci-report:<context> -->`.

## Pass a run whose failures are all filed

Every failing test fails the run by default. To keep CI passing while a filed bug waits for its fix, set [`bugs.knownFailures`](https://turnstone.cloud/docs/bug-reports.md#known-failures) to `"allow"`. A run then passes when an open issue with the `testkit` label covers each failure. A new failure still fails the run.

With `"allow"`, the run reads the open issues. Pass the job's token to the `npm run testkit` step as well.

```yaml
- run: npm run testkit
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

The job also needs the `issues: read` permission.

## Split a slow gate

Run one test type per job. Each job passes its type to the script.

```yaml title=.github/workflows/testkit.yml
jobs:
  testkit:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        type: [contract, int, ui, dal]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
        if: matrix.type == 'ui'
      - run: npm run testkit -- ${{ matrix.type }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results-${{ matrix.type }}
          path: .testkit/results
          include-hidden-files: true
```

List only the types the repo has. A type with no files is an empty selection, and it fails its job.

If these jobs run `testkit ci report`, give each job its own context. [Report more than one run](#report-more-than-one-run) shows the step.

To run more than one file at a time inside a job, raise [`execution.taskConcurrency`](https://turnstone.cloud/docs/configuration.md#execution).

## Read a failed job

The job log holds the same lines as a local run. The `testkit-results` artifact holds everything else: service logs, engine output, and each file's artifacts. [Read a run's results](https://turnstone.cloud/docs/results.md) shows how to find a file in it.

---

# Read a run's results

Every run prints a summary and seals a record under .testkit/results. This page shows how to read both.

## Read the summary

```console $ npx testkit run
[1/6] PASS api ui src/web/__testkit__/checkout.ui.testkit.ts 2s
[2/6] PASS api contract src/checkout/__testkit__/refund.contract.testkit.ts 0s
[3/6] PASS api contract src/__testkit__/health.contract.testkit.ts 0s
[4/6] PASS api int src/account/__testkit__/profile.int.testkit.ts 0s
[5/6] PASS api dal src/db/__testkit__/refunds.dal.testkit.ts 0s
[6/6] PASS api int src/checkout/__testkit__/health.int.testkit.ts 0s

Summary
Result: PASSED
Passed: 6
Failed: 0
Skipped: 0
Not run: 0
Files: 6
Duration: 5s
Run ID: 1791891435785-1572397-5c629188
```

A file line gives the file's position, its result, its service, its test type, its path, and how long it took.

| Line | What it counts |
| --- | --- |
| `Passed`, `Failed` | Files that ran |
| `Skipped` | Files that a skip rule held back |
| `Not run` | Files that the run selected and did not run |
| `Files` | Every file the run selected |
| `Runtime errors` | Files that failed outside their tests, such as in a setup module. The line appears only when the count is above zero. |
| `Run ID` | The name of the run's directory under `.testkit/results/runs/` |

The run prints one line per file and a short block per failure. Service logs and engine output go to disk.

## Find a run

```console $ npx testkit artifacts --list
1791891435785-1572397-5c629188 passed 2026-10-13T11:37:20.929Z
1791891435201-1572245-010ad857 passed 2026-10-13T11:37:15.203Z
1791891434722-1572097-fbd9051c failed 2026-10-13T11:37:14.724Z
```

The list is newest first. Every other `artifacts` command reads the latest run unless you pass `--run <id>`.

## See what a file left behind

Pass a test file to list its artifacts. Each entry has a name, a kind in brackets, a summary, and a path.

```console $ npx testkit artifacts src/checkout/__testkit__/refund.contract.testkit.ts
testkit-vitest-report [engine.testkit-vitest-report]
  Testkit native Vitest report
  .testkit/results/runs/1791891435785-1572397-5c629188/files/artifacts/api/task-4-01-testkit-vitest-report.json
task-engine-native [engine.task.native]
  task engine-native evidence
  .testkit/results/runs/1791891435785-1572397-5c629188/files/artifacts/api/task-4-01-task-engine-native.json
invocation-engine-native [engine.invocation.native]
  invocation engine-native evidence
  .testkit/results/runs/1791891435785-1572397-5c629188/files/artifacts/api/task-4-01-invocation-engine-native.json
```

The list does not read the files. To print one, name its kind and ask for text.

```console $ npx testkit artifacts src/checkout/__testkit__/refund.contract.testkit.ts --kind engine.testkit-vitest-report --format text
{"schemaVersion":1,"engineId":"node-vitest","invocationId":"1791891435785-1572397-5c629188:lease-4-2:4","taskId":"4","sourceFile":"src/checkout/__testkit__/refund.contract.testkit.ts","result":{"summary":{"numFailedTestSuites":0,"numFailedTests":0,"numPassedTestSuites":1,"numPassedTests":1,"numPendingTestSuites":0,"numPendingTests":0,"numTodoTests":0,"numTotalTestSuites":1,"numTotalTests":1,"success":true},"failures":[],"errors":[]}}
```

The kinds depend on the engine.

| Test | Kinds worth knowing |
| --- | --- |
| Contract | `engine.testkit-vitest-report` |
| HTTP and database | `engine.summary` for the k6 summary, `engine.output` for k6's stderr |
| Database with fixtures | `testkit.dal-fixtures` for the seeded records |
| Browser, after a failure | `testkit.failure-details` and `engine.playwright-attachment` |

## Check that nothing changed

Testkit seals a run when it ends. `--verify` recomputes every checksum and compares it with the run's manifest.

```console $ npx testkit artifacts --verify
Verified 1791891435785-1572397-5c629188: 186 files (73ad31bf7d5b7b25b930cac190ed1003309dd875c391d0dc966c7e521132d0d4)
```

## Where the files are

```text title=.testkit/results/
.testkit/results/
├── latest.json
├── objects/
│   └── sha256/
└── runs/
    └── 1791891435785-1572397-5c629188/
        ├── run.json
        ├── provenance.json
        ├── manifest.json
        └── files/
            ├── artifacts/
            │   └── api/
            └── logs/
                └── api-runtime-1__api.log
```

| Path | Holds |
| --- | --- |
| `runs/<run-id>/run.json` | The run's record: its services, files, results, and timings |
| `runs/<run-id>/provenance.json` | What produced the run: the CI job, the Git commit, and the Testkit and Node versions |
| `runs/<run-id>/manifest.json` | Every file in the run and its SHA-256 checksum |
| `runs/<run-id>/files/logs/` | Service logs. `api-runtime-1__api.log` is the output of the `api` service. |
| `runs/<run-id>/files/artifacts/<service>/` | Each file's artifacts |
| `objects/sha256/` | The payloads, stored once per checksum |
| `latest.json` | A copy of the newest run's record. Read it for convenience, and treat `runs/` as the record. |

Add `.testkit/` to `.gitignore`.

## Read results from a script

`--json` prints one document for the whole run.

```sh
npx testkit run --json > run.json
```

| Key | Holds |
| --- | --- |
| `exitCode` | The exit code of the run |
| `runId` | The run's id |
| `results` | One entry per service, with its suites and files |
| `runArtifact` | The run's record. `runArtifact.summary.files` has `passed`, `failed`, `skipped`, `notRun`, and `total`. |
| `outputMode` | The reporter mode |

In Node, `@gfdelarue/testkit/workspace` exports `listRunBundles`, `loadRunBundleManifest`, `loadRunBundleArtifact`, and `verifyRunBundle`.

## Keep fewer runs

Testkit keeps every run by default. To set a limit, add [`artifacts.retention`](https://turnstone.cloud/docs/configuration.md#artifacts).

```ts title=testkit.config.ts
export default defineConfig({
	artifacts: {
		retention: {
			maxRuns: 50,
			maxAgeDays: 30,
		},
	},
});
```

## What leaves your machine

Testkit 0.4.1 has no command that uploads a run's record. The record stays in `.testkit/results` until you or your CI job copies it somewhere. [Gate merges in CI](https://turnstone.cloud/docs/ci.md#add-the-workflow) uploads it as a workflow artifact.

Two commands send a summary to GitHub or Gitea. [`testkit ci report`](https://turnstone.cloud/docs/ci.md#report-the-run-to-github-or-gitea) posts a run's result and names each failure by its file and its title. [`testkit bug file`](https://turnstone.cloud/docs/bug-reports.md#the-issue-testkit-files) files one bug as an issue.

---

# CLI reference

Every command and flag in Testkit 0.4.1.

## How to run a command

Run `testkit` through `npx` from the repo root, or from a package script.

```console $ npx testkit --version
@gfdelarue/testkit/0.4.1 linux-x64 node-v24.19.0
```

### Shared flags

| Flag | Meaning |
| --- | --- |
| `--dir <path>` | The product directory. The default is the current directory. |
| `--service <name>` | Limit the command to one service. |
| `--json` | Print one JSON document instead of text. |
| `-h, --help` | Print the command's flags. |
| `-v, --version` | Print the Testkit version. Root command only. |

`doctor`, `lint`, `typecheck`, `setup playwright`, `ci report`, `bug record`, `bug file`, and the three `campaign` commands take `--dir` and `--json` but not `--service`. `bug guide` takes `--json` only. `guard unsafe-entrypoint` takes none of the three.

The tables below list only the flags that belong to one command.

## Run tests

### `testkit run`

Runs test files. With no argument, it runs every file that Testkit discovers.

```usage
testkit run [type] [flags]
```

`type` is `ui`, `e2e`, `int`, `contract`, `data`, `dal`, `load`, or `all`.

| Flag | Meaning |
| --- | --- |
| `-t, --type <type>` | Run one test type. Repeat the flag to add more. |
| `-s, --suite <suite>` | Run one suite. `contract:checkout` selects one suite, and `checkout` selects that name in every type. Repeat the flag to add more. |
| `-f, --file <path>` | Run one file. Repeat the flag to add more. |
| `--task-concurrency <n>` | The most test tasks that run at once. Overrides [`execution.taskConcurrency`](https://turnstone.cloud/docs/configuration.md#execution). |
| `--file-timeout-seconds <n>` | The wall-clock limit for each file, in seconds. |
| `--allow-empty` | Let the run pass when no file runs. |
| `--ignore-skip-rules` | Run files that a skip rule holds back. |
| `--skip-schema-source-verify` | Skip the check that the local schema matches the source database schema. |
| `--output-mode <mode>` | `compact`, `debug`, or `events`. `events` prints one JSON object per line. |
| `--debug` | The same as `--output-mode debug`. |

`testkit run` exits `0` only when at least one file completed and every required file passed. [What fails the gate](https://turnstone.cloud/docs/ci.md#what-fails-the-gate) lists the other cases.

### `testkit run status`

Shows the runs that are active.

```console $ npx testkit run status
Testkit Run Status
  product: /repo
  local runs: none
```

| Flag | Meaning |
| --- | --- |
| `--run <id>` | Show one run. |
| `--full` | Include the full live status in the JSON output. |

### `testkit run stop`

Stops the newest active run.

| Flag | Meaning |
| --- | --- |
| `--run <id>` | Stop this run. |
| `--all` | Stop every active run for the product. |
| `--reason <text>` | A reason, recorded with the cancelled run. |

### `testkit stop`

Stops every run that Testkit owns in the product.

| Flag | Meaning |
| --- | --- |
| `--reason <text>` | A reason, recorded with the cancelled runs. |

### `testkit ci report`

Posts the last run in the checkout as one commit status, the job's step summary, and one pull request comment. [Report the run to GitHub or Gitea](https://turnstone.cloud/docs/ci.md#report-the-run-to-github-or-gitea) shows the workflow step and the token it needs.

| Flag | Meaning |
| --- | --- |
| `--context <name>` | The commit status context. The default is `testkit`. |

`testkit ci report` exits `0` when it posts the report, and that includes the report of a failed run.

## Hunt bugs

These commands need no Turnstone account. [Run a campaign](https://turnstone.cloud/docs/campaigns.md) shows them in order.

### `testkit campaign start`

Starts a campaign and prints the agent's brief.

| Flag | Meaning |
| --- | --- |
| `--target <text>` | Required. What the agent should hunt, in one line of at most 200 characters. |
| `--id <id>` | The campaign id. The default is made from the target and the date. |

### `testkit campaign status`

Shows the active campaign, the bugs it has recorded, and how much of the source the area map covers.

```console $ npx testkit campaign status
No campaign is active.
Last finished: refunds-2026-10, finished 2026-10-14T00:37:22.377Z
Area map: 3.9% of source bytes unclaimed, 0 double-claimed files.
```

### `testkit campaign finish`

Finishes the active campaign, writes its summary to `.testkit/campaigns/<id>/summary.md`, and prints the summary. The summary is for the pull request that adds the tests.

### `testkit bug guide`

Prints the bug record format, the severities, and the lists of cause, impact, and escape ids.

### `testkit bug record`

Checks a bug record, stores it as a draft under `.testkit/bugs/`, and prints the draft id. [Write the bug record](https://turnstone.cloud/docs/bug-reports.md#write-the-bug-record) lists the fields.

```usage
testkit bug record <file> [flags]
```

`file` is a bug record JSON file, or `-` to read standard input.

### `testkit bug file`

Runs a draft's test at the checked-out commit, and files the bug as an issue only when the test fails. [What `testkit bug file` checks](https://turnstone.cloud/docs/bug-reports.md#what-testkit-bug-file-checks) lists the cases where it files nothing.

```usage
testkit bug file <id> [flags]
```

`id` is the draft id that `testkit bug record` printed.

## Inspect

### `testkit discover`

Lists test files and their metadata without running one. [Test files](https://turnstone.cloud/docs/test-files.md#see-what-testkit-found) shows the output.

| Flag | Meaning |
| --- | --- |
| `-t, --type <type>` | List one test type. Repeat the flag to add more. |
| `-s, --suite <suite>` | List one suite. Repeat the flag to add more. |
| `-f, --file <path>` | List one file. Repeat the flag to add more. |
| `--runnable-only` | List only the files that no skip rule holds back. |
| `--strict` | Exit non-zero on a discovery or config diagnostic. |
| `--output <path>` | Write the discovery JSON to a file. |
| `--output-mode <mode>` | `compact` or `verbose`. The default is `compact`. `verbose` adds each file's path, id, and status. |

### `testkit artifacts`

Lists, verifies, or prints what a run kept. [Read a run's results](https://turnstone.cloud/docs/results.md) shows each use.

```usage
testkit artifacts [file] [flags]
```

| Flag | Meaning |
| --- | --- |
| `--list` | List the runs that Testkit kept. |
| `--run <id>` | Read this run. The default is the latest run. |
| `--verify` | Check the run's manifest and checksums. |
| `--kind <kind>` | List one kind of artifact. |
| `--format <format>` | `list` or `text`. The default is `list`. `text` prints the artifact's content. |

### `testkit status`

Shows Testkit's local state for the product: its services, its active and stale runs, its runtime instances, and its caches.

### `testkit workspace`

Prints a summary of the product.

```console $ npx testkit workspace
shop · Testkit workspace
  1 services · 6 suites · 6 files
  latest passed · 6/6 files passed
```

| Flag | Meaning |
| --- | --- |
| `--target-limit <n>` | The most quick-run targets to preview. The default is `24`. |
| `--no-status` | Skip the runtime status. |

## Check the repo

### `testkit doctor`

Runs the config, discovery, lint, and typecheck checks in one command.

```console $ npx testkit doctor
testkit doctor passed for /repo
PASS config-load Loaded testkit config and service config
PASS discovery Discovered 6 testkit files without diagnostics
PASS lint No Testkit lint violations across 6 suite file(s)
PASS typecheck Typechecked 4 generated program(s)
```

| Flag | Meaning |
| --- | --- |
| `--no-typecheck` | Skip the TypeScript checks. |

### `testkit lint`

Runs Testkit's repository hygiene rules. [`lint`](https://turnstone.cloud/docs/configuration.md#lint) lists the rules and their defaults.

```console $ npx testkit lint
testkit lint passed for /repo
Checked 11 source file(s), 6 testkit suite file(s).
```

### `testkit typecheck`

Typechecks `testkit.config.ts`, helpers, and test files. Testkit checks each engine's files as a separate program.

```console $ npx testkit typecheck
Typechecked 4 testkit program(s) in /repo
PASS config /repo/.testkit/_typecheck/config.tsconfig.json
PASS k6 /repo/.testkit/_typecheck/k6.repo.tsconfig.json
PASS node-vitest /repo/.testkit/_typecheck/node-vitest.repo.tsconfig.json
PASS playwright /repo/.testkit/_typecheck/playwright.repo.tsconfig.json
```

## Set up and clean up

### `testkit setup playwright`

Installs the Chromium that browser tests run in.

| Flag | Meaning |
| --- | --- |
| `--with-deps` | Also install the host libraries that Chromium needs. |
| `--force` | Repair the browser installation. |

### `testkit db schema refresh`

Refreshes the cached schema of a source database. It applies to a service whose [`database`](https://turnstone.cloud/docs/configuration.md#database) sets `sourceSchema`.

### `testkit db schema verify`

Checks that the local schema replay matches the source database schema.

| Flag | Meaning |
| --- | --- |
| `--skip-schema-source-verify` | Skip the check against the source database. |

### `testkit cleanup`

Removes stale local state.

```console $ npx testkit cleanup --dry-run
No stale runs to clean.
```

| Flag | Meaning |
| --- | --- |
| `--dry-run` | Print what the command would remove. |
| `--cache <cache>` | Also clear a cache: `runtime`, `bundles`, or `all`. Repeat the flag to add more. |
| `--resources` | Also remove Docker resources that Testkit owns and that are stale or orphaned. |

### `testkit destroy`

Removes the Postgres container of each service, and prints one line for each service. It keeps `.testkit/results`. The next run that needs the database builds it again.

```console $ npx testkit destroy
Destroyed api
```

### `testkit guard unsafe-entrypoint`

Fails with a message that names a replacement command. Use it as the body of a package script that people must run through Testkit.

| Flag | Meaning |
| --- | --- |
| `-c, --command <name>` | Required. The disabled command, shown in the error. |
| `-u, --use <command>` | Required. The replacement command, shown in the error. |

```console $ npx testkit guard unsafe-entrypoint --command "npm run test:e2e" --use "npm run testkit -- e2e"
Error: npm run test:e2e is disabled because DB-touching tests must run through the isolated testkit harness.
Use npm run testkit -- e2e instead.
```

---

# Configuration reference

The fields of testkit.config.ts in Testkit 0.4.1. The file lives at the repo root and holds the repo's test policy.

## The file

```ts title=testkit.config.ts
import { app, defineConfig } from "@gfdelarue/testkit/config";

export default defineConfig({
	services: {
		api: app.node({
			build: null,
			start: "npm start",
			port: 4392,
		}),
	},
});
```

Import the builders from `@gfdelarue/testkit/config`.

| Export | Returns |
| --- | --- |
| `defineConfig(config)` | The config, typed |
| `app.node(options)` | A Node service |
| `app.next(options)` | A Next.js service |
| `database.postgres(options)` | A Postgres database for a service |
| `database.fixture(options)` | A service that is only a database |
| `toolchain.node(options)` | A rule for the Node and npm versions |

> Testkit 0.4.1 ignores a top-level key that it does not know, and prints no warning. A [`skip`](#skip) block placed beside `services` does nothing. To confirm that a rule took effect, run `testkit discover` and read the `skipped` count.

## Top-level fields

### `execution`

| Field | Default | Meaning |
| --- | --- | --- |
| `taskConcurrency` | `1` | The most test tasks that run at once. `--task-concurrency` overrides it. |
| `fileTimeoutSeconds` | `60` | The wall-clock limit for each file. `waitFor()` spends from the same limit. |

```ts
execution: {
	taskConcurrency: 4,
	fileTimeoutSeconds: 120,
},
```

### `services`

A map from a service name to a service. The name appears in run output, and `--service` takes it. Build each service with [`app.node(...)`](#appnode), [`app.next(...)`](#appnext), or `database.fixture(...)`.

### `profiles.http`

Named k6 options. An [HTTP test](https://turnstone.cloud/docs/http-tests.md#use-k6-directly) selects one with `defineHttpSuite({ profile: "raw" }, ...)`.

```ts
profiles: {
	http: {
		raw: {
			options: { vus: 1, iterations: 1 },
		},
	},
},
```

When a run selects a k6 file, every profile must be plain JSON. A profile that holds a function fails config validation. Put auth and header functions in the test file or in a helper beside it.

### `ui`

Targets for [browser tests](https://turnstone.cloud/docs/browser-tests.md#point-the-test-at-another-target).

| Field | Default | Meaning |
| --- | --- | --- |
| `frontendBaseUrl` | The run's base URL | The browser target |
| `backendBaseUrl` | None | The API target for `createAuthedApiClient` |
| `allowRemote` | `false` | Permit a target that is not on loopback |

Options that a test passes to `createUiSandbox` win over this block.

### `lint`

Settings for `testkit lint`. Omit the block to run the default rules.

| Rule | Default |
| --- | --- |
| `engineImports` | On |
| `configImports` | On |
| `dataSuiteContract` | On |
| `dalParallelSafety` | On |
| `uiSpecShape` | On |
| `unitTestBoundaries` | Off |
| `testkitBoundary` | Off |

Set a rule to `true`, `false`, or an options object. The options are `enabled`, `include`, and `exclude`. `testkitBoundary` also takes `allowed` and `trackedOnly`.

`lint.ui` sets the limits for browser test files.

| Field | Default | Meaning |
| --- | --- | --- |
| `ui.maxLines` | `220` | The most lines in one `ui` file |
| `ui.maxTests` | `8` | The most tests in one `ui` file |

```ts
lint: {
	rules: {
		unitTestBoundaries: true,
	},
	ui: { maxLines: 300 },
},
```

A finding from an enabled rule fails the command. There are no severity levels.

### `areas`

A map from an area id to a named part of the product. `testkit bug file` uses it to name the area that holds a fault, and puts an `area:<id>` label on the issue.

| Field | Meaning |
| --- | --- |
| The key | The area id. It uses lowercase letters, digits, and single hyphens, and has at most 40 characters. |
| `name` | One line of 1 to 60 characters. It becomes the label's description. |
| `paths` | At least one glob, relative to the repo root and written with `/`. Testkit rejects a leading `!`, a leading `/`, and a `..` segment. |

```ts
areas: {
	checkout: { name: "Checkout", paths: ["src/checkout/**"] },
	billing: { name: "Billing", paths: ["src/billing/**", "src/invoices/**"] },
},
```

Each source file belongs to one area at most. `testkit campaign start` lists the source files that no area claims and the files that two areas claim. `testkit bug file` stops when no area claims the fault's file, or when two areas do, and names the file. Without `areas`, Testkit files bugs with no area.

An area's share is its part of the repo's source bytes at the commit that holds the bug. Tests, generated files, vendored code, and migrations do not count as source.

### `bugs`

Settings for bug filing, and for runs that meet a filed bug.

| Field | Default | Meaning |
| --- | --- | --- |
| `knownFailures` | `"fail"` | `"fail"` or `"allow"`. With `"fail"`, every failing test fails the run. With `"allow"`, a run passes when an open issue with the `testkit` label covers each failure. |
| `repository` | The `origin` remote | The repo that holds the bug issues. Write `owner/name` for a repo on the same host as the `origin` remote, or a URL such as `https://host/owner/name`. |

```ts
bugs: {
	knownFailures: "allow",
	repository: "acme/shop-bugs",
},
```

[Known failures](https://turnstone.cloud/docs/bug-reports.md#known-failures) gives the rule for `knownFailures`. With `"allow"`, the run reads the open issues, so the `testkit run` step in CI [needs a token](https://turnstone.cloud/docs/ci.md#pass-a-run-whose-failures-are-all-filed).

`testkit bug file` files issues in `repository`, and `testkit run` and `testkit ci report` read the open issues from it. Commit statuses and pull request comments always go to the `origin` repo. If a Turnstone project reads this repo's bugs, [enter the same repo in the project](https://turnstone.cloud/docs/sync.md#issues-repository).

### `artifacts`

| Field | Meaning |
| --- | --- |
| `retention.maxRuns` | The most runs to keep |
| `retention.maxAgeDays` | The oldest run to keep, in days |
| `retention.preserveSuites` | Suites whose runs retention does not remove, such as `ui:performance` |
| `publication.directory` | A directory that Testkit copies each finished run into |
| `publication.required` | When `true`, a failed copy fails the command. The local run stays in place. |

With no `retention` block, Testkit keeps every run. It records retention decisions under `.testkit/results/retention` and each publication under `.testkit/results/publications`.

### `fingerprints`

Testkit hashes the repo's files to decide when a cache is stale. `exclude` removes generated files from the hash, and `include` puts a path back below an excluded directory.

```ts
fingerprints: {
	exclude: ["next-env.d.ts"],
},
```

Patterns use `/` as the only separator. Testkit always excludes `.testkit/` and `.next-testkit/`.

### `toolchains`

Named rules for the Node and npm versions that run a service's commands. A service selects one with `runtime.toolchain`.

```ts
toolchains: {
	frontendNode: toolchain.node({
		cwd: "frontend",
		detect: "auto",
		install: "download",
	}),
},
```

| Option | Values | Meaning |
| --- | --- | --- |
| `cwd` | A path, relative to the repo root | The directory that holds the version files |
| `detect` | `"auto"`, `"off"` | `"auto"` reads the versions from the repo |
| `install` | `"require-host"`, `"download"` | `"download"` fetches Node into `.testkit/_toolchains/node` and checks it against a trusted release manifest. It supports glibc hosts only. |
| `node`, `npm` | A version or a range | The version to use |

With `detect: "auto"`, Testkit reads the Node version from `package.json#volta.node`, `.nvmrc`, `.node-version`, `.tool-versions`, and `package.json#engines.node`. It reads the npm version from `package.json#volta.npm`, `package.json#packageManager`, and `package.json#engines.npm`.

## Services

### `app.node(...)`

| Option | Meaning |
| --- | --- |
| `port` | Required. The port the service listens on. Testkit passes it to the process as `PORT`. |
| `start` | The command that starts the service |
| `entry` | The service's entry file, such as `src/index.ts` |
| `build` | How Testkit builds the service. Set `null` when the service needs no build. |
| `cwd` | The service's directory, relative to the repo root |
| `env` | `{ values, databases }`. See [`env`](#env). |
| `toolchain` | A name from [`toolchains`](#toolchains), or an inline `toolchain.node(...)` |
| `shutdownTimeoutMs` | How long the service gets to exit before Testkit kills it. The default is 1000 ms. |

`app.node` also accepts `baseUrl`, `buildInputs`, `outDir`, `readyPath`, `readyTimeoutMs`, `readyUrl`, `runtime`, and `tsconfig`, and every field under [Service fields](#service-fields). The types in `@gfdelarue/testkit/config` define each one.

### `app.next(...)`

Takes the same options as `app.node`, except `entry`, `outDir`, `readyPath`, and `tsconfig`. It adds `mode`, which is `"dev"` or `"start"`, and `api`.

```ts
frontend: app.next({
	cwd: "frontend",
	mode: "start",
	port: 3000,
	dependsOn: ["api"],
	env: {
		values: { NEXT_PUBLIC_API_URL: "{baseUrl:api}" },
	},
}),
```

## Service fields

### `env`

Environment values for the service process and for every test that targets the service. `app.node` and `app.next` take `{ values, databases }`.

Values can use [template tokens](#template-tokens).

`databases` expands one service's database into prefixed entries.

```ts
env: {
	databases: { api: { prefix: "API", service: "api" } },
},
```

That block sets `API_DATABASE_HOST`, `API_DATABASE_PORT`, `API_DATABASE_NAME`, `API_DATABASE_USER`, `API_DATABASE_PASSWORD`, and `API_DATABASE_SSL`. An entry in `values` with the same name wins.

A test reads the values from its engine's context.

| Engine | Where the values are |
| --- | --- |
| `k6` | `__ENV` |
| `node-vitest` | `context.env`, read-only |
| `playwright` | `readPlaywrightContext()?.env` |

### `envFile` and `envFiles`

Env files to load into the same environment as `env`: `envFile` takes one path, and `envFiles` takes a list.

### `database`

`database.postgres(...)` gives the service a local Postgres in Docker. The service process receives it as `DATABASE_URL`. [Test the database](https://turnstone.cloud/docs/database-tests.md) has a working example.

| Option | Meaning |
| --- | --- |
| `template.inputs` | Files that rebuild the template when they change |
| `template.migrate` | Steps that build the schema |
| `template.seed` | Steps that add data |
| `template.verify` | Steps that must pass before Testkit caches the template |
| `binding` | `"shared"` reuses one database. `"per-runtime"` clones the template for each runtime instance. |
| `runtimeConnections` | The connection budget of one runtime instance |
| `maxConnections`, `reservedConnections` | The capacity of the local Postgres, which the scheduler uses to admit tasks |
| `roles` | Roles to create, each as `{ name, login: false }` |
| `sourceSchema` | A source database to replay the schema from, such as `database.schema.fromEnv("PRODUCTION_DATABASE_URL")` |
| `image` | The Postgres image |

A template stage is an ordered list of steps.

| Step | Example |
| --- | --- |
| A command | `{ kind: "command", run: "npm run db:seed" }` |
| An SQL file | `{ kind: "sql-file", path: "db/schema.sql" }` |
| A module export | `{ kind: "module", target: "src/testkit/verify-seed.ts#verifySeed" }` |

Docker also runs `psql` and `pg_dump`. To use another client image, set `TESTKIT_POSTGRES_CLIENT_IMAGE`.

### `databaseFrom`

The name of another service. This service receives that service's `DATABASE_URL`.

### `dependsOn`

The services that must start first.

### `runtime`

| Field | Meaning |
| --- | --- |
| `instances` | How many copies of the service Testkit keeps running |
| `maxConcurrentTasks` | The most test files that one instance serves at once. Omit it to leave `taskConcurrency` as the only limit. |
| `toolchain` | A name from [`toolchains`](#toolchains), or an inline `toolchain.node(...)` |
| `prepare.steps` | Steps that run once per instance before the service starts. Each step takes `cwd` and `inputs`. |
| `prepare.inputs` | Files that rerun the steps when they change |

```ts
runtime: {
	instances: 1,
	maxConcurrentTasks: 4,
	prepare: {
		inputs: ["frontend/src", "frontend/package.json"],
		steps: [{ kind: "command", run: "npm run build", cwd: "frontend" }],
	},
},
```

### `skip`

Skip rules for the service. Every rule needs a `reason`.

```ts
services: {
	api: app.node({
		port: 4392,
		skip: {
			files: [
				{
					path: "src/checkout/__testkit__/payout.contract.testkit.ts",
					reason: "The payout sandbox is down until Friday",
				},
			],
			suites: [
				{ selector: "ui:billing", reason: "Billing UI is a placeholder" },
			],
		},
	}),
},
```

| Field | Meaning |
| --- | --- |
| `files[].path` | An exact path, relative to the repo root |
| `suites[].selector` | A suite selector, such as `ui:billing` |
| `reason` | Required. The run prints it and the run record keeps it. |

To skip one file, prefer [`defineFile({ skip })`](https://turnstone.cloud/docs/test-files.md#skip-a-file-with-a-reason) in the file itself. `testkit run --ignore-skip-rules` runs skipped files anyway.

### `requirements`

Lock rules for the service. Files that need the same lock do not run at the same time.

```ts
requirements: {
	suites: [{ selector: "ui:billing", locks: ["billing-ledger"] }],
	files: [
		{
			path: "src/jobs/__testkit__/worker.int.testkit.ts",
			locks: ["global-worker-loop"],
		},
	],
},
```

To lock one file, prefer [`defineFile({ locks })`](https://turnstone.cloud/docs/test-files.md#keep-two-files-apart-with-a-lock).

### `discovery`

`discovery.roots` assigns directories to the service. Testkit normally gives a test file to the service with the deepest matching `cwd`, so set `roots` only when two services share a directory.

## Template tokens

Strings in a service can hold these tokens.

| Token | Value |
| --- | --- |
| `{service}` | The service's name |
| `{runtime}` | The runtime instance |
| `{port}`, `{port:name}` | The port of this service, or of the service `name` |
| `{baseUrl}`, `{baseUrl:name}` | The base URL |
| `{readyUrl}`, `{readyUrl:name}` | The readiness URL |
| `{stateDir}`, `{prepareDir}`, `{lease}`, `{leaseDir}` | Testkit's directories and lease for the instance |

These tokens work only in `env` values.

| Token | Value |
| --- | --- |
| `{dbUrl}` | The connection URL |
| `{dbHost}`, `{dbPort}` | The host and the host port |
| `{dbName}`, `{dbUser}`, `{dbPassword}` | The database name, user, and password |
| `{dbSsl}` | `1` when the connection uses SSL, and `0` when it does not |

A bare database token reads the service's own database. `{dbUrl:api}` reads the database of the service `api`, which must be in `dependsOn`.

Testkit rejects a database token in `start`, `cwd`, `runtime.prepare`, and `database.template` strings. In those places, read `DATABASE_URL` when the command runs.

## Environment limits

Each engine reserves some names and limits how much environment a test can receive.

| Engine | Reserved prefixes | Reserved names | Limit |
| --- | --- | --- | --- |
| `k6` | `K6_`, `TESTKIT_` | `BASE_URL`, `GODEBUG` | 128 entries and 16,384 environment-block units |
| `node-vitest` | `TESTKIT_`, `VITEST_` | `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`, `UPDATE_SNAPSHOT` | 512 entries and 4 MiB |
| `playwright` | `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, `PW_TEST_` | `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR` | 512 entries and 2 MiB |

Every engine also reserves `APPDATA`, `ComSpec`, `HOME`, `LANG`, `LC_ALL`, `LOCALAPPDATA`, `PATH`, `PATHEXT`, `SystemRoot`, `TEMP`, `TMP`, `TMPDIR`, `USERPROFILE`, `XDG_CACHE_HOME`, `XDG_CONFIG_HOME`, and `XDG_DATA_HOME`.

Testkit compares names without regard to case. It rejects an empty name, a name that contains `=`, and two names that differ only in case.

---

# Run a campaign

A campaign gives your coding agent a target. The agent works through the CLI, and each bug it proves lands in GitHub or Gitea as an issue that names a failing test.

## What a campaign is

A campaign gives a coding agent a target and the instructions to file what it finds. Three commands run it.

| Command | What it does |
| --- | --- |
| `testkit campaign start` | Starts the campaign and prints the agent's brief |
| `testkit campaign status` | Lists the bugs that the campaign has recorded |
| `testkit campaign finish` | Ends the campaign and writes the summary for the pull request that adds the tests |

During the campaign, your agent writes Testkit tests. It files each bug it proves with `testkit bug record` and `testkit bug file`. [What a bug report states](https://turnstone.cloud/docs/bug-reports.md) describes the bug record and the issue.

Campaigns, bug filing, and CI reports need no Turnstone account. They use the CLI and your repo on GitHub or Gitea.

## Before you start

You need these four things.

- Testkit 0.4.1 or later, with one test passing. [Get started](https://turnstone.cloud/quickstart) covers both.
- A clone whose `origin` remote is the repo on GitHub or Gitea. Testkit files issues in that repo. To file them in another repo, set [`bugs.repository`](https://turnstone.cloud/docs/configuration.md#bugs).
- A token for that repo that can create issues and labels.
- A coding agent that can run commands in the clone.

To check the version, run `npx testkit --version`. It prints a line that starts with `@gfdelarue/testkit/` and the installed version.

### Set the token

Testkit reads the token from the environment.

| Host | Variable |
| --- | --- |
| GitHub | `GITHUB_TOKEN` or `GH_TOKEN` |
| Gitea | `GITEA_TOKEN` |

With no token set, Testkit asks `git credential fill` for one.

Set the variable in the shell that your agent works in. Keep the token out of the repo and out of every bug record.

Testkit reaches GitHub Enterprise Server only from a GitHub Actions job on that server, so `testkit bug file` works there only in a job.

## Name your product's areas

An area is a named part of your product and the paths that hold its code. Add `areas` to `testkit.config.ts` at the repo root. If the file does not exist, create it.

```ts title=testkit.config.ts
import { defineConfig } from "@gfdelarue/testkit/config";

export default defineConfig({
	areas: {
		checkout: { name: "Checkout", paths: ["src/checkout/**"] },
		billing: { name: "Billing", paths: ["src/billing/**", "src/invoices/**"] },
	},
});
```

Each key is an area id. An area id uses lowercase letters, digits, and single hyphens. Each path is a glob from the repo root. The same file holds the rest of your config, such as the services that your tests start.

Testkit puts an `area:<id>` label on each issue it files, and Turnstone reads the bug's area from that label. Without `areas`, Testkit files bugs with no area.

[`areas`](https://turnstone.cloud/docs/configuration.md#areas) gives the rules for ids, names, and paths.

## Pick a target

The target is one line of at most 200 characters that tells the agent what to hunt, such as `Refund flows in checkout`.

Keep the target narrow. A narrow target gets the agent reading code sooner and keeps its reports about one part of the product.

Pick code where the right behaviour is written down: an API reference, a spec, a ticket, or a contract with a customer. Every bug record names that source in its `authority` field. Without one, a report is an opinion.

## Start the campaign

Run the command from the repo root.

```sh
npx testkit campaign start --target "Refund flows in checkout" --id refunds-2026-10
```

`--id` sets the campaign id. Without it, Testkit makes the id from the target and the date, such as `refund-flows-in-checkout-2026-10-14`. The campaign id goes into the facts block of each issue that the campaign files.

A checkout holds one active campaign. If a campaign is active, `campaign start` stops and tells you to run `testkit campaign finish` first.

## Give the brief to your agent

`campaign start` prints the agent's brief. Give that output to the agent.

```text title=The start of the brief
# Testkit campaign refunds-2026-10

Target: Refund flows in checkout

## Area map

- `checkout`, Checkout: 1 file, 54.2% of source bytes
- `billing`, Billing: 2 files, 41.9% of source bytes
- Unclaimed: 1 file, 3.9% of source bytes

Unclaimed source files:

- `src/server.js`
```

The areas, the files, and the shares in this brief come from a small example repo. The area map lists each area's share of the repo's source, the source files that no area claims, and the files that two areas claim.

The brief asks the agent to write or refresh `areas` before it hunts, so that every source file belongs to exactly one area. Read the change it makes to `testkit.config.ts`.

The brief then has these parts.

| Part | What it holds |
| --- | --- |
| How to work | The rules of the hunt. A failing test is a bug only when an authority says what the product should do. |
| For each confirmed bug | The steps that file a bug |
| Bug record | Every field of the bug record, with an example |
| Severities, Cause, Impact, Escape | The values that a bug record can use |
| How to end | The command that finishes the campaign |

## What your agent does for each bug

For each bug that the agent confirms, it does these steps.

1. It writes a Testkit test that fails because of the bug.
2. It writes the bug record as a JSON file. `npx testkit bug guide` prints the fields and the cause, impact, and escape lists.
3. It runs `testkit bug record`. The command checks the record, stores a draft under `.testkit/bugs/`, and prints the draft id.
4. It commits the test and pushes the commit.
5. It runs `testkit bug file` with the draft id.

```console $ npx testkit bug record bug.json
Recorded a-refund-pays-out-twice-d1c8d076
Draft: .testkit/bugs/a-refund-pays-out-twice-d1c8d076.json
Next: commit and push the test, then run `testkit bug file a-refund-pays-out-twice-d1c8d076`.
```

`testkit bug file` runs the test at the checked-out commit and files the issue only when the test fails. Replace `DRAFT_ID` with the draft id that `testkit bug record` prints.

```console $ npx testkit bug file DRAFT_ID
Filed #12: https://github.com/acme/shop/issues/12
Area: checkout, 14.0% of source bytes
Introduced by: 9b7a4c3d8e6f
```

The repo, the issue number, and the commit in this output are examples.

The issue carries the `testkit` label and a `severity:<level>` label. With `areas` set, it also carries an `area:<id>` label. The issue body names the regression test, the commit that introduced the faulty lines, and the command that reproduces the failure.

If `testkit bug file` stops without filing, [What `testkit bug file` checks](https://turnstone.cloud/docs/bug-reports.md#what-testkit-bug-file-checks) lists each cause and what to do.

## Check what comes back

Every bug arrives as an issue that names a failing test in the repo.

1. Run the command under **Reproduce** in the issue. The test must fail on your machine for the reason the issue gives.
2. Open the source that the issue cites under **Authority**. It must say what the issue claims.
3. Read the root cause. It must name code, not a symptom.

Close a report that fails any of the three. [What a bug report states](https://turnstone.cloud/docs/bug-reports.md) lists the sections to expect.

## See what the campaign recorded

`campaign status` lists the bugs recorded so far.

```console $ npx testkit campaign status
Active campaign: refunds-2026-10
  Target: Refund flows in checkout
  Started: 2026-10-14T00:26:02.987Z
  Start commit: ac58e0f35137c177346285256a5559520a083f2f
Recorded bugs: 1
  high: A refund pays out twice
    Test: src/checkout/__testkit__/refund.contract.testkit.ts
    Issue: not filed, draft a-refund-pays-out-twice-d1c8d076
Area map: 3.9% of source bytes unclaimed, 0 double-claimed files.
```

The bug in this output has a draft and no issue, so `testkit bug file` has not filed it yet. The last line gives the share of the source that no area claims and the number of files that two areas claim.

## Finish the campaign

When the agent finishes, write the summary.

```sh
npx testkit campaign finish
```

`campaign finish` ends the campaign, writes a summary to `.testkit/campaigns/<id>/summary.md`, and prints it. The summary is for the pull request that adds the tests. It holds these parts.

- The target, the start and finish times, and the range of commits.
- The bugs that the campaign filed.
- The bugs that the campaign recorded and did not file, each with its draft id.
- The Testkit test files that the campaign added or changed.

## Keep the test failing until the fix lands

The pull request that adds the tests stays red while a bug is open, and that is correct. You can merge it in two ways.

- Push the fix to the same branch, watch the test pass, and merge both together.
- Set [`bugs.knownFailures`](https://turnstone.cloud/docs/bug-reports.md#known-failures) to `"allow"`. A run then passes when an open issue with the `testkit` label covers each failure. A new failure still fails the run.

Do not skip the test to get a green check. Do not rewrite it to assert what the product does today. Either change hides a bug that is still there.

## Stop a run

An agent can leave a run going. `testkit run status` shows active runs, and `testkit stop` ends every run that Testkit owns in the repo.

```sh
npx testkit run status
npx testkit stop --reason "campaign paused"
```

---

# What a bug report states

Testkit files each bug as an issue in GitHub or Gitea. The issue has five sections, names the test that fails, and ends with a facts block.

## From a bug record to an issue

A bug report starts as a bug record. The bug record is a JSON file that your coding agent writes during a [campaign](https://turnstone.cloud/docs/campaigns.md), or that you write yourself. Three commands turn it into an issue.

| Command | What it does |
| --- | --- |
| `testkit bug guide` | Prints the bug record format, the severities, and the cause, impact, and escape lists |
| `testkit bug record <file>` | Checks a bug record, stores it as a draft under `.testkit/bugs/`, and prints the draft id |
| `testkit bug file <id>` | Runs the draft's test and files the bug as an issue |

The commands work with or without an active campaign. Bug filing needs no Turnstone account.

## The five sections

Every issue that Testkit files has five sections.

| Section | What it answers | Where it comes from |
| --- | --- | --- |
| Expected behaviour | What should the product do? | `expected` in the bug record |
| Authority | Where is that written down? | `authority` in the bug record |
| Observed behaviour | What does the product do instead? | `observed` in the bug record |
| Root cause | Which code makes it happen? | `rootCause` in the bug record |
| Reproduce | Which command shows it? | Testkit writes the command from `test.file` |

A report with all five can be checked by someone who was not there. Every field of the bug record is required, so no report lacks one.

## Write the bug record

```sh
npx testkit bug guide
```

The command prints the fields of a bug record, the severities, and the lists of cause, impact, and escape ids.

| Field | Value |
| --- | --- |
| `title` | One line of at most 200 characters that names the broken behaviour. It becomes the issue title. |
| `severity` | `critical`, `high`, `medium`, or `low` |
| `expected` | What the product should do |
| `authority` | The source that says so: documentation, a specification, an API contract, or the code's own stated intent. Name it and quote the part that applies. |
| `observed` | What the product does instead |
| `rootCause` | Why the code behaves this way |
| `test.file` | The Testkit test file that fails because of the bug |
| `test.title` | The failure's title, in one line of at most 200 characters |
| `fault.path` | The source file that holds the mistake |
| `fault.lines` | The first and last faulty line as `[start, end]`, counted from 1. One faulty line is `[41, 41]`. |
| `cause` | One cause id |
| `impact` | One impact id |
| `confidence.cause` | `high` or `low`. How sure the writer is of the cause. |
| `confidence.impact` | `high` or `low`. How sure the writer is of the impact. |
| `escape` | One escape id |

```json title=bug.json
{
	"title": "A refund pays out twice",
	"severity": "high",
	"expected": "The second POST /orders/{id}/refunds for the same order returns 409 and pays nothing.",
	"authority": "`docs/api/refunds.md`, \"A refund is idempotent per order.\"",
	"observed": "The second request returns 201 and creates a second payout.",
	"rootCause": "`src/checkout/refunds.ts` retries the payout call without an idempotency key.",
	"test": {
		"file": "src/checkout/__testkit__/refund.contract.testkit.ts",
		"title": "a refund pays out once"
	},
	"fault": { "path": "src/checkout/refunds.ts", "lines": [41, 58] },
	"cause": "mishandled-error",
	"impact": "financial-error",
	"confidence": { "cause": "high", "impact": "high" },
	"escape": "no-test"
}
```

These rules apply to the fields.

- Write `test.file` and `fault.path` relative to the product directory, the directory that holds `testkit.config.ts`.
- `test.title` must be the failure's title exactly as Testkit reports it. In a Vitest or Playwright file, that is the names of the `describe` blocks around the test and the test's own name, joined by single spaces. In a k6 file, that is the label of the failed assertion.
- A k6 script that fails as a whole has no failure title. Write in `test.title` what the script tests. Testkit then files one issue that covers the whole file.
- Keep credentials, tokens, and passwords out of the record.

### Severities

| Severity | Meaning |
| --- | --- |
| `critical` | The product is unsafe to run or to trust. |
| `high` | A core behaviour is broken with no acceptable workaround. |
| `medium` | A behaviour is broken and a workaround exists, or the reach is limited. |
| `low` | A minor defect with little consequence. |

### Cause, impact, and escape

Each of these fields takes one id from a list. `testkit bug guide` prints each id's definition and the rule that separates it from its neighbours.

| Field | Question | Ids |
| --- | --- | --- |
| `cause` | What kind of mistake is in the code? | `missing-validation`, `mishandled-error`, `race-condition`, `unbounded-or-inefficient-work`, `incorrect-query-or-selection`, `missing-access-control`, `null-or-default-mishandling`, `stale-state`, `interface-mismatch`, `omitted-step`, `incorrect-logic`, `other` |
| `impact` | What is at stake for the business? | `unauthorized-access`, `financial-error`, `audit-or-compliance-gap`, `data-loss-or-corruption`, `workflow-breakdown`, `availability-or-performance`, `blocked-operation`, `wrong-information-shown`, `other` |
| `escape` | Why did the existing tests not catch the bug? | `no-test`, `untested-input`, `untested-interleaving`, `wrong-assertion`, `test-double`, `other` |

A bug gets one id for each field. The impact list is in precedence order, and the first id that applies wins. Use `other` when none fits.

## The test that comes with it

The bug record names one test. The test's name states the expected behaviour, so the name is true once the product is fixed.

```ts title=src/checkout/__testkit__/refund.contract.testkit.ts
import { defineFile } from "@gfdelarue/testkit";
import { contractScenario, expect } from "@gfdelarue/testkit/node-vitest/contract";

export const testkit = defineFile({ engine: "node-vitest" });

contractScenario("a refund pays out once", async ({ http }) => {
	const first = await http().post("/orders/8841/refunds");
	const second = await http().post("/orders/8841/refunds");
	expect(first.status).toBe(201);
	expect(second.status).toBe(409);
});
```

Against the broken product, the test fails on the line that states the expected behaviour.

```console $ npx testkit run --file src/checkout/__testkit__/refund.contract.testkit.ts
[1/1] FAIL api contract src/checkout/__testkit__/refund.contract.testkit.ts 0s
    AssertionError: expected 201 to be 409 // Object.is equality at
    /repo/src/checkout/__testkit__/refund.contract.testkit.ts:10:24
```

## The issue Testkit files

The issue's title is the `title` of the bug record.

### Labels

| Label | Meaning |
| --- | --- |
| `testkit` | Filed by Testkit with a regression test. Every issue that Testkit files carries it. |
| `severity:<level>` | The bug's severity, such as `severity:high` |
| `area:<id>` | The product area that holds the fault. The label's description is the area's name. |

Testkit creates a label that does not exist yet and never changes one that does. An issue carries an `area:<id>` label when [`areas`](https://turnstone.cloud/docs/configuration.md#areas) is set.

Severity and area are labels because people re-triage both. To change a bug's severity or area, replace the label. An issue with two `severity:` labels counts as having none, and the same holds for `area:` labels.

### Body

The body has the five sections, three lines that locate the bug, and the facts block.

````text title=The body of the issue for bug.json
## Expected behaviour

The second POST /orders/{id}/refunds for the same order returns 409 and pays nothing.

## Authority

`docs/api/refunds.md`, "A refund is idempotent per order."

## Observed behaviour

The second request returns 201 and creates a second payout.

## Root cause

`src/checkout/refunds.ts` retries the payout call without an idempotency key.

## Reproduce

```sh
npx testkit run --file src/checkout/__testkit__/refund.contract.testkit.ts
```

Test: `src/checkout/__testkit__/refund.contract.testkit.ts`, "a refund pays out once".
Found at `4f0c2d1e9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d`.
Fault: `src/checkout/refunds.ts`, lines 41 to 58.

<details>
<summary>Testkit facts</summary>

<!-- testkit-facts:start -->

```json
{
  "v": 1,
  "cause": "mishandled-error",
  "impact": "financial-error",
  "confidence": { "cause": "high", "impact": "high" },
  "escape": "no-test",
  "caughtBy": "campaign",
  "campaign": "refunds-2026-10",
  "testType": "contract",
  "test": {
    "file": "src/checkout/__testkit__/refund.contract.testkit.ts",
    "title": "a refund pays out once"
  },
  "foundAt": "4f0c2d1e9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d",
  "fault": { "path": "src/checkout/refunds.ts", "lines": [41, 58] },
  "area": { "id": "checkout", "name": "Checkout", "share": 0.14 },
  "introducedBy": "9b7a4c3d8e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b",
  "introducedMethod": "blame"
}
```

<!-- testkit-facts:end -->

</details>
````

The file names, the commits, and the campaign in this body are examples.

The reproduce command runs from the repo root. When `testkit.config.ts` is in a subdirectory, the command carries `--dir <directory>`, and the `--file` path is relative to that directory. The `Test:` and `Fault:` lines always give paths from the repo root. A fault on one line reads `line 41`.

### Facts block

The facts block is one JSON object between two marker comments. It sits in a collapsed section named **Testkit facts**, and tools read it. [Facts](https://turnstone.cloud/docs/facts.md) lists what Turnstone reads from it.

| Field | Written | Meaning |
| --- | --- | --- |
| `v` | Always | The format version, `1` |
| `cause` | Always | The kind of mistake in the code |
| `impact` | Always | What is at stake for the business |
| `confidence.cause` | Always | The writer's confidence in `cause`, `high` or `low` |
| `confidence.impact` | Always | The writer's confidence in `impact`, `high` or `low` |
| `escape` | Always | Why the existing tests did not catch the bug |
| `caughtBy` | Always | `campaign` when a campaign was active when the bug was recorded, and `manual` otherwise |
| `campaign` | With `caughtBy: "campaign"` | The id of the campaign that caught the bug |
| `testType` | Always | The type of the regression test, from its file suffix |
| `test.file` | Always | The regression test file |
| `test.title` | When the issue covers one test | The failure's title |
| `foundAt` | Always | The commit the test failed at |
| `fault.path` | Always | The source file that holds the mistake |
| `fault.lines` | Always | The first and last faulty line at `foundAt`, counted from 1 |
| `area.id` | When an area claims `fault.path` | The area's key in `areas` |
| `area.name` | With `area.id` | The area's name when the bug was filed |
| `area.share` | With `area.id` | The area's share of the repo's source bytes at `foundAt`, from 0 to 1 |
| `introducedBy` | When the blame resolves | The newest commit among the fault lines at `foundAt` |
| `introducedMethod` | With `introducedBy` | How Testkit found `introducedBy`. The value is `blame`. |

All paths in the block use `/` and are relative to the repo root. Commits are full commit ids.

The package ships the full format, with the rules for a tool that reads it, as `node_modules/@gfdelarue/testkit/docs/bug-records.md`.

## What `testkit bug file` checks

`testkit bug file` runs the test at the checked-out commit and files the issue only when the test fails. It files nothing in these cases.

| `testkit bug file` stops when | What to do |
| --- | --- |
| Tracked files have uncommitted changes | Commit or discard the changes. Then run the command again. |
| No remote branch contains the commit | Push the commit. Then run the command again. |
| An open issue already covers the test | Read that issue. Testkit keeps one open bug for each test. |
| `areas` is set, and no area claims the fault's file or two areas do | The message names the file. Change `areas` in `testkit.config.ts` so that one area claims the file. |
| `test.title` is not one of the failure titles that the file reports | The message lists the titles that a bug record can use. |

The token needs permission to create issues and labels. [Set the token](https://turnstone.cloud/docs/campaigns.md#set-the-token) lists the variables that Testkit reads it from.

### Secrets

Testkit redacts the secrets it recognises in the title, in every section, and in `test.title`. It does so when it records the bug and again when it files the issue. Testkit cannot recognise every secret, so keep credentials, tokens, and passwords out of the record.

Testkit does not rewrite a path or an area's name. It stops in these cases.

- `testkit bug record` rejects a record whose `test.file` or `fault.path` holds text that Testkit redacts.
- `testkit bug file` stops when the area's name or the reproduce command holds such text.
- Testkit files no bug under a failure title that holds such text. Rename the test, or give the assertion a label.

## Known failures

A known failure is a failing test that an open issue with the `testkit` label covers. `bugs.knownFailures` in `testkit.config.ts` decides what a known failure does to a run.

| Value | What a run does |
| --- | --- |
| `"fail"` | Every failing test fails the run. This is the default. |
| `"allow"` | A run whose failures are all known passes. One new failure fails the run. |

```ts title=testkit.config.ts
export default defineConfig({
	bugs: { knownFailures: "allow" },
});
```

Use `"allow"` to keep CI passing while a filed bug waits for its fix. These rules apply.

- An issue covers a failure when its facts block gives the failing file as `test.file` and the failure's title as `test.title`. An issue with no `test.title` covers the whole file.
- Closing an issue makes its test count again.
- A failure that names no test is always new. A crash, a timeout, and a file that does not load are such failures.
- When Testkit cannot read the issues, every failure is new.
- A run that fails outside its tests fails under both values.

With `"allow"`, the run reads the open issues, so `testkit run` needs a token. [Pass a run whose failures are all filed](https://turnstone.cloud/docs/ci.md#pass-a-run-whose-failures-are-all-filed) shows the CI step.

## Where the evidence lives

The issue points to evidence where it already is. Do not copy it somewhere new.

| Evidence | Where it lives |
| --- | --- |
| The failing test | Your repo, at the commit on the issue's `Found at` line |
| The command that shows the failure | The issue's **Reproduce** section |
| The faulty lines | Your repo, at the path and lines on the issue's `Fault:` line |
| The full run | The CI job's uploaded `.testkit/results` |

[Read a run's results](https://turnstone.cloud/docs/results.md) shows how to find a file's output in a run.

Turnstone holds none of this evidence. [What Turnstone keeps](https://turnstone.cloud/docs/cloud.md) lists the facts that Turnstone stores.

## What a report never does

- It never skips the failing test.
- It never asserts the broken behaviour so that the test passes.
- It never reports a failure without a cause. A failing test with no root cause is a lead, and the agent keeps working on it.

---

# What Turnstone keeps

Turnstone is an optional, read-only dashboard. It keeps facts about bugs and charts them. Your reports, tests, and evidence stay in GitHub or Gitea.

## Turnstone is the optional part

Testkit has four parts. Your repo holds the tests. The CLI runs them. GitHub or Gitea holds the issues, the checks, and the pull requests. Turnstone is the fourth part, and the other three work without it.

The CLI in Testkit 0.4.1 has no Turnstone command. It does not sign in to Turnstone, it does not ask for a Turnstone key, and it does not upload a run to Turnstone. Turnstone gets its facts from GitHub or Gitea, where your team already works.

Campaigns, bug filing, and CI reports need no Turnstone account. `testkit bug file` and `testkit ci report` write to GitHub or Gitea, with a token for that host.

## What Turnstone keeps

Turnstone keeps facts about each bug.

| Fact | Example |
| --- | --- |
| Issue link | The URL of the issue in your forge |
| Severity | How bad the bug is |
| Area | The part of the product it is in |
| Commits | The commits linked to the bug, such as the one that introduced it |
| Timestamps | When it was introduced, found, and fixed |

[Facts](https://turnstone.cloud/docs/facts.md) lists every fact and the place Turnstone reads it from.

## What Turnstone never keeps

- The bug report. The text stays in the issue.
- The tests. They stay in your repo.
- The evidence. Run output, logs, and artifacts stay in `.testkit/results` and in your CI.

Turnstone charts a bug without storing its report. That is deliberate. The parts of a bug report that could embarrass you or leak something are the parts Turnstone does not keep.

## If the dashboard shows no bugs

If the dashboard shows no bugs after a sync, check two things.

1. Check that the issue carries the `testkit` label.
2. Check that the project reads the repo that holds the issue.

[Sync](https://turnstone.cloud/docs/sync.md) covers both.

## If Turnstone lost everything

You would lose your charts. Your issues, tests, pull requests, and CI history would be where they are now, because Turnstone never held them.

## What the charts are for

A list of bugs tells you what broke. The facts Turnstone keeps let you ask where bugs come from, why, and how to have fewer of them.

## What it costs

The CLI, the runner, campaigns, and everything they send to GitHub or Gitea are free. Turnstone has a free tier, so you can connect early and build up history.

To see your own repo, [sign in](https://turnstone.cloud/dashboard) and connect it on [GitHub](https://turnstone.cloud/docs/connect-github.md) or [Gitea](https://turnstone.cloud/docs/connect-gitea.md).

---

# Connect a GitHub repository

Connect a GitHub repository to a Turnstone project, and see the access that Turnstone asks for.

A Turnstone project reads one repository. Follow this page to connect a repository on GitHub to a new project.

## Before you connect

Check these conditions first:

- You are an owner or an admin of your Turnstone organisation. A member or a viewer cannot create a project.
- The Turnstone GitHub App is installed on the repository.
- Your GitHub account can read the repository.

## Connect the repository

1. Open [the Turnstone dashboard](https://turnstone.cloud/dashboard) and sign in.
2. Select **New project**.
3. Enter a project name.
4. Choose **GitHub** as the provider.
5. Enter the repository as owner and name, for example `acme/checkout`. Do not enter the full URL.
6. Select **Connect repository**. GitHub asks you to authorise the Turnstone GitHub App.
7. Approve the request. GitHub returns you to the dashboard, and you should see the message "Repository connected."

If Turnstone does not connect the repository, check the conditions in [Before you connect](#before-you-connect). Then start again from step 2.

Turnstone syncs the repository when you connect it. [Sync](https://turnstone.cloud/docs/sync.md) describes when Turnstone reads the repository again.

## Access

Turnstone uses GitHub access at two points.

When you select **Connect repository**, GitHub asks you to authorise the App for your account. Turnstone uses that authorisation once. It checks that your GitHub account can read the repository and that the App is installed on it. Turnstone does not store your GitHub token.

After that, Turnstone reads through the App's installation. Turnstone stores the id of the installation. A sync reads the repository's issues, commits, and pull requests.

Turnstone never writes to GitHub. It creates no issue, no comment, no check, and no commit status. The Testkit CLI posts the commit status and the pull request comment from your CI job, with your CI job's token.

To end Turnstone's access, remove the repository from the App's installation in your GitHub settings.

---

# Connect a Gitea repository

Add your Gitea host to Turnstone, connect a repository, and see the access that Turnstone asks for.

Turnstone connects to a Gitea host through an OAuth2 application that you create on that host. An owner of your Turnstone organisation adds the host once. After that, owners and admins can connect its repositories.

## Add the host

To add a host, you must be an owner of your Turnstone organisation. You also need a Gitea account that can create an OAuth2 application.

1. In Gitea, create an OAuth2 application. Set its redirect URI to `https://turnstone.cloud/auth/gitea/callback`.
2. Copy the client ID and the client secret that Gitea shows.
3. In Turnstone, open your organisation's **Settings** and select **Providers**.
4. Select **Add Gitea host** to open **Add a Gitea host**. Enter a display name.
5. Enter the host URL, for example `https://gitea.example.com`. The URL must use HTTPS.
6. Enter the OAuth2 client ID and the OAuth2 client secret.
7. If the host's certificate comes from a private or self-signed certificate authority, paste the authority's PEM certificates into **CA bundle**.
8. Select **Add Gitea host**. You should see the message "Gitea host added" and the host under **Gitea hosts**.

Turnstone stores the client secret encrypted and never shows it again.

## Connect the repository

To connect a repository, you must be an owner or an admin of your Turnstone organisation. Your Gitea account must be able to read the repository.

1. Open [the Turnstone dashboard](https://turnstone.cloud/dashboard) and select **New project**.
2. Enter a project name.
3. Choose **Gitea** as the provider.
4. Enter the host name without `https://`, for example `gitea.example.com`.
5. Enter the repository as owner and name, for example `acme/checkout`.
6. Select **Connect repository**. Gitea asks you to authorise the OAuth2 application.
7. Approve the request. Gitea returns you to the dashboard, and you should see the message "Repository connected."

Turnstone syncs the repository when you connect it. [Sync](https://turnstone.cloud/docs/sync.md) describes when Turnstone reads the repository again.

## Access

Turnstone asks Gitea for two scopes, `read:repository` and `read:issue`. With them, Turnstone checks that your Gitea account can read the repository, and each sync reads the repository's issues, commits, and pull requests.

Turnstone stores an access token and a refresh token for the host, both encrypted. The tokens are read-only, and each sync uses them. They act with the Gitea account of the person who connected the repository.

Turnstone never writes to Gitea. It creates no issue, no comment, and no commit status. The Testkit CLI posts the commit status and the pull request comment from your CI job, with your CI job's token.

To end Turnstone's access, revoke the OAuth2 application's grant in your Gitea account settings. An owner can also select **Remove host** under **Settings**, then **Providers**. Turnstone does not remove a host that a project still uses. Removing the host deletes its stored client secret and tokens.

---

# How Turnstone syncs a repository

When Turnstone reads your issues, what it reads, and what it does with an issue that has no facts.

A sync is one read of a connected repository. Turnstone reads the issues that Testkit filed and takes the facts from each issue. The dashboard's charts come from those facts. A sync writes nothing to GitHub or Gitea.

## When Turnstone syncs

Turnstone syncs a repository at these times:

- When you connect the repository to a project. This sync reads every issue that carries the `testkit` label.
- Every 5 minutes after that. This sync reads the issues that changed since the last sync.
- Once a day. This sync reads every issue again.
- When you select **Sync now**. This sync also reads every issue.

Use **Sync now** when you filed a bug or changed a label and you want the dashboard to show the change before the next scheduled sync. Owners and admins see the button.

1. Open the project.
2. In the **Issue sync** panel, select **Sync now**.

The same panel is on the project's connection settings page, under **Settings**, then **Connection**. It shows when the last sync ran and whether that sync read every issue. If a sync fails, the panel gives the reason.

## What a sync reads

Turnstone reads only the issues that carry the `testkit` label. Testkit puts that label on every issue it files. Turnstone does not read the other issues in the repository.

From each of those issues, Turnstone reads the state, the dates, the labels, and the facts block in the body. Turnstone also reads the commit that closed the issue and the commit that introduced the bug, with that commit's pull request. [Facts](https://turnstone.cloud/docs/facts.md) lists each fact and its source.

Turnstone keeps the facts. It does not store the issue's title or its text.

## Separate issues repository

Turnstone reads issues from the connected repository unless you name another one. Name another one when Testkit files your bugs in a different repository from the one that holds the code. In Testkit, `bugs.repository` in `testkit.config.ts` sets that repository.

If your `testkit.config.ts` sets `bugs.repository`, enter the same repository in the issues repository field. The field is on the project's connection settings page, and it is optional. Turnstone then reads the issues from that repository.

Turnstone needs read access to the issues repository too. On GitHub, install the Turnstone GitHub App on it.

When you change the issues repository, Turnstone deletes the facts it stored for the project. It then reads every issue from the new repository.

## Missing or invalid facts

Testkit writes a facts block into the body of each issue it files. An issue with the `testkit` label can still reach Turnstone in one of these two states:

- **No facts block.** The body has no facts block. For example, someone added the `testkit` label to an issue by hand.
- **Invalid facts block.** The body has a facts block that Turnstone cannot read. For example, an edit broke the JSON, or the block has a format version that Turnstone does not know.

Turnstone keeps the issue in both cases and records which case applies. The bug keeps the facts that do not come from the block. Those facts are its issue number and link, its state, its dates, its severity, and its area id. Turnstone takes no fact from the block, so the bug has no cause, no impact, and no introducing commit.

To repair an invalid block, edit the issue body so that the block follows the [bug record format](https://turnstone.cloud/docs/facts.md#bug-record-format). A later sync reads the repaired block.

---

# The facts Turnstone keeps

Each fact that Turnstone keeps about a bug, where Turnstone reads it from, and what Turnstone never stores.

Turnstone keeps a fixed set of facts about each bug. This page lists each fact and the place that Turnstone reads it from.

## Issues Turnstone reads

Turnstone reads only the issues that carry the `testkit` label. Each of those issues is one bug.

## Facts and sources

| Fact | Source |
| --- | --- |
| Issue number and link | The provider. Turnstone keys a bug by its issue. |
| Open or closed | The issue's state. |
| Caught date | The issue's creation time. |
| Fixed date and fixing commit | The issue's close time and the commit that closed it. An issue closed with no commit counts as closed, not as fixed. |
| Severity | The `severity:<level>` label. |
| Area | The `area:<id>` label. The label wins over the facts block. The area's name and its share of the code come from the facts block. |
| Cause, impact, how it was caught, test type, campaign, the commit it was found at, the introducing commit | The facts block in the issue body. |
| Introduced date, the change, and its kind | Turnstone derives them from the introducing commit. The kind comes from the pull request's title prefix, such as `feat:` or `fix:`, or from its labels. |

The severity levels are `critical`, `high`, `medium`, and `low`.

Turnstone keeps four more values from the facts block. Two of them are the finder's confidence in the cause and in the impact, each `high` or `low`. The third is the reason that the existing tests did not catch the bug. The fourth is the method that Testkit used to find the introducing commit.

People change severity and area by changing a label on the issue. A later sync reads the new label.

## What Turnstone never stores

Turnstone stores no title, no issue text, no file path, no author, no test, and no evidence.

The facts block also names the test's file, the test's title, and the file that holds the fault. Turnstone does not keep those fields.

The issue stays with GitHub or Gitea, and the regression test stays in your repository. If Turnstone lost its data, you would lose your charts and nothing else.

Turnstone never writes to GitHub or Gitea.

## Bug record format

The bug record format is the contract between Testkit and Turnstone. It defines the labels, the sections of the issue body, and every field of the facts block. The format is at version 1.

The [Testkit package](https://www.npmjs.com/package/@gfdelarue/testkit) ships the format as a document. After you install Testkit, read this file:

```text
node_modules/@gfdelarue/testkit/docs/bug-records.md
```

`npx testkit bug guide` prints the fields of a bug record and the lists of cause, impact, and escape ids.