# Gate merges in CI

Run Testkit on every pull request, block the merge when a test fails, and report each run to GitHub or Gitea as a commit status and a pull request comment.

## Add the script

```json title=package.json
{
	"scripts": {
		"testkit": "testkit run"
	}
}
```

## Add the workflow

```yaml title=.github/workflows/testkit.yml
name: Testkit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  testkit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true
```

Three lines need a reason.

- `setup playwright --with-deps` installs Chromium for browser tests. If the repo has no `ui` files, delete the step.
- `if: always()` uploads the results when the run fails. A failed run is the one you want to read.
- `include-hidden-files: true` is required because `.testkit` starts with a dot. Without it, `actions/upload-artifact@v4` uploads nothing.

`ubuntu-latest` has Docker, which [database tests](https://turnstone.cloud/docs/database-tests.md) need.

## What fails the gate

`npm run testkit` exits `0` only when at least one file completed and every required file passed. Each of these makes it exit `1`.

- A test fails.
- A [setup module](https://turnstone.cloud/docs/test-files.md#run-node-code-before-a-file) throws. The summary adds a `Runtime errors` line.
- A file is skipped, even when every other file passes.
- A file is interrupted or does not run.
- The selection matches no file.

```console $ npx testkit run e2e
SKIP api no matching files (types=e2e suites=all files=all)

Summary
Result: FAILED
Passed: 0
Failed: 0
Skipped: 0
Not run: 0
Files: 0
Duration: 0s
Run ID: 1791891434722-1572097-fbd9051c
```

If a job's selection is empty on purpose, add `--allow-empty` to that job. Do not add it to the default command. A gate that passes with no tests is not a gate.

```sh
npm run testkit -- e2e --allow-empty
```

One setting changes the first case. With [`bugs.knownFailures`](https://turnstone.cloud/docs/bug-reports.md#known-failures) set to `"allow"`, a failing test does not fail the run while an open issue covers it.

## Require the check

A failing job does not block a merge until the forge requires it.

- **GitHub.** In the repo's settings, add a branch protection rule or a ruleset for `main`. Turn on **Require status checks to pass**, and select the `testkit` job.
- **Gitea.** In the repo's settings, open **Branches** and edit the protection rule for `main`. Turn on **Enable Status Check**, and select the check.

Gitea Actions uses the same workflow syntax. Put the file in `.gitea/workflows/`, and check that your Gitea version supports each action the workflow uses.

## Report the run to GitHub or Gitea

`testkit ci report` reads the last run in the checkout and posts it in three places.

| Place | What Testkit posts |
| --- | --- |
| The commit | One commit status |
| The job | The step summary |
| The pull request | One comment, which Testkit updates in place |

The report needs no Turnstone account. In GitHub Actions, add the `permissions` block and the report step to the job that runs your tests.

```yaml title=.github/workflows/testkit.yml
jobs:
  testkit:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      issues: read
      pull-requests: write
      statuses: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - run: npx testkit ci report
        if: always()
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true
```

Four parts need a reason.

- The `npm run testkit` step decides whether the job fails. `ci report` exits `0` when it posts the report, and that includes the report of a failed run.
- `if: always()` runs the report step after a failed run.
- The `env` lines pass the job's token to the report. GitHub Actions passes the job's token to a `run` step only when the step sets it. Gitea Actions accepts the same lines.
- `permissions` lets the job's token read the code and the open issues, post the commit status, and write the pull request comment.

After a failed run on a pull request, the report step prints lines like these.

```console $ npx testkit ci report
Status testkit on 4f0c2d1e9b7a: failure, 1 failed: 1 new, 0 known
Pull request comment: created https://github.com/acme/shop/pull/7#issuecomment-31
Step summary: written
```

The commit, the repo, and the comment in this output are examples.

### Commit status

`testkit ci report` posts one commit status on the checked-out commit. When CI checks out a pull request's merge commit, the status goes on the pull request's head commit instead.

| Part | Value |
| --- | --- |
| Context | `testkit`, or the value of `--context` |
| State | `success` when the run passed, or when `bugs.knownFailures` is `"allow"` and every failure is known. `failure` otherwise. |
| Target URL | The CI run's URL, when the CI provider gives one |
| Description | One of the three forms below, in at most 140 characters |

| Run | Description |
| --- | --- |
| Every file passed | `142 files passed` |
| Tests failed | `3 failed: 1 new, 2 known` |
| The run failed outside its tests | `run failed: <reason>` |

The counts are failures. A failure is known when an open issue with the `testkit` label covers it. [Known failures](https://turnstone.cloud/docs/bug-reports.md#known-failures) gives the rule.

### Pull request comment

`testkit ci report` keeps one comment on the run's pull request and updates it in place. The comment names each failure by its file and its title.

The comment's first line is `<!-- testkit-ci-report -->`. Testkit finds its comment by that line.

### Pass the token

Testkit reads the token from the environment.

| Host | Variable |
| --- | --- |
| GitHub | `GITHUB_TOKEN` or `GH_TOKEN` |
| Gitea | `GITEA_TOKEN` first. In a Gitea Actions job, Testkit reads the job's `GITHUB_TOKEN` second. |

With no token set, Testkit asks `git credential fill` for one. The token that `actions/checkout` keeps in the checkout does not answer that request, so set the variable on the step.

### Pull requests from forks

GitHub gives a pull request from a fork a read-only token. `ci report` then writes the step summary and fails when it posts the status. If the repo takes pull requests from forks, add `continue-on-error: true` to the step.

```yaml
- run: npx testkit ci report
  if: always()
  continue-on-error: true
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

### No results for the commit

When the checkout holds no results for its commit, `ci report` posts a `failure` status with this description.

```text
run failed: Testkit found no results for this commit
```

If the commit status reads that way, run `testkit run` before the report in the same checkout.

### Report more than one run

A job that runs Testkit more than once passes `--context <name>` to each report, so each run keeps its own status and its own comment. The default context is `testkit`.

The jobs in [Split a slow gate](#split-a-slow-gate) report on the same commit and the same pull request. Give each of those jobs its own context too.

```yaml
- run: npx testkit ci report --context testkit-${{ matrix.type }}
  if: always()
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

With a context other than `testkit`, the comment's first line is `<!-- testkit-ci-report:<context> -->`.

## Pass a run whose failures are all filed

Every failing test fails the run by default. To keep CI passing while a filed bug waits for its fix, set [`bugs.knownFailures`](https://turnstone.cloud/docs/bug-reports.md#known-failures) to `"allow"`. A run then passes when an open issue with the `testkit` label covers each failure. A new failure still fails the run.

With `"allow"`, the run reads the open issues. Pass the job's token to the `npm run testkit` step as well.

```yaml
- run: npm run testkit
  env:
    GITHUB_TOKEN: ${{ github.token }}
```

The job also needs the `issues: read` permission.

## Split a slow gate

Run one test type per job. Each job passes its type to the script.

```yaml title=.github/workflows/testkit.yml
jobs:
  testkit:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        type: [contract, int, ui, dal]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
        if: matrix.type == 'ui'
      - run: npm run testkit -- ${{ matrix.type }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results-${{ matrix.type }}
          path: .testkit/results
          include-hidden-files: true
```

List only the types the repo has. A type with no files is an empty selection, and it fails its job.

If these jobs run `testkit ci report`, give each job its own context. [Report more than one run](#report-more-than-one-run) shows the step.

To run more than one file at a time inside a job, raise [`execution.taskConcurrency`](https://turnstone.cloud/docs/configuration.md#execution).

## Read a failed job

The job log holds the same lines as a local run. The `testkit-results` artifact holds everything else: service logs, engine output, and each file's artifacts. [Read a run's results](https://turnstone.cloud/docs/results.md) shows how to find a file in it.