Gate merges in CI

Add the script

{
	"scripts": {
		"testkit": "testkit run"
	}
}

Add the workflow

name: Testkit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  testkit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true

Three lines need a reason.

• setup playwright --with-deps installs Chromium for browser tests. If the repo has no ui files, delete the step.
• if: always() uploads the results when the run fails. A failed run is the one you want to read.
• include-hidden-files: true is required because .testkit starts with a dot. Without it, actions/upload-artifact@v4 uploads nothing.

ubuntu-latest has Docker, which database tests need.

What fails the gate

npm run testkit exits 0 only when at least one file completed and every required file passed. Each of these makes it exit 1.

• A test fails.
• A setup module throws. The summary adds a Runtime errors line.
• A file is skipped, even when every other file passes.
• A file is interrupted or does not run.
• The selection matches no file.

SKIP api no matching files (types=e2e suites=all files=all)

Summary
Result: FAILED
Passed: 0
Failed: 0
Skipped: 0
Not run: 0
Files: 0
Duration: 0s
Run ID: 1791891434722-1572097-fbd9051c

If a job's selection is empty on purpose, add --allow-empty to that job. Do not add it to the default command. A gate that passes with no tests is not a gate.

$npm run testkit -- e2e --allow-empty

One setting changes the first case. With bugs.knownFailures set to "allow", a failing test does not fail the run while an open issue covers it.

Require the check

A failing job does not block a merge until the forge requires it.

• GitHub. In the repo's settings, add a branch protection rule or a ruleset for main. Turn on Require status checks to pass, and select the testkit job.
• Gitea. In the repo's settings, open Branches and edit the protection rule for main. Turn on Enable Status Check, and select the check.

Gitea Actions uses the same workflow syntax. Put the file in .gitea/workflows/, and check that your Gitea version supports each action the workflow uses.

Report the run to GitHub or Gitea

testkit ci report reads the last run in the checkout and posts it in three places.

The report needs no Turnstone account. In GitHub Actions, add the permissions block and the report step to the job that runs your tests.

jobs:
  testkit:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      issues: read
      pull-requests: write
      statuses: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
      - run: npm run testkit
      - run: npx testkit ci report
        if: always()
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results
          path: .testkit/results
          include-hidden-files: true

Four parts need a reason.

• The npm run testkit step decides whether the job fails. ci report exits 0 when it posts the report, and that includes the report of a failed run.
• if: always() runs the report step after a failed run.
• The env lines pass the job's token to the report. GitHub Actions passes the job's token to a run step only when the step sets it. Gitea Actions accepts the same lines.
• permissions lets the job's token read the code and the open issues, post the commit status, and write the pull request comment.

After a failed run on a pull request, the report step prints lines like these.

Status testkit on 4f0c2d1e9b7a: failure, 1 failed: 1 new, 0 known
Pull request comment: created https://github.com/acme/shop/pull/7#issuecomment-31
Step summary: written

The commit, the repo, and the comment in this output are examples.

Commit status

testkit ci report posts one commit status on the checked-out commit. When CI checks out a pull request's merge commit, the status goes on the pull request's head commit instead.

The counts are failures. A failure is known when an open issue with the testkit label covers it. Known failures gives the rule.

Pull request comment

testkit ci report keeps one comment on the run's pull request and updates it in place. The comment names each failure by its file and its title.

The comment's first line is <!-- testkit-ci-report -->. Testkit finds its comment by that line.

Pass the token

Testkit reads the token from the environment.

With no token set, Testkit asks git credential fill for one. The token that actions/checkout keeps in the checkout does not answer that request, so set the variable on the step.

Pull requests from forks

GitHub gives a pull request from a fork a read-only token. ci report then writes the step summary and fails when it posts the status. If the repo takes pull requests from forks, add continue-on-error: true to the step.

- run: npx testkit ci report
  if: always()
  continue-on-error: true
  env:
    GITHUB_TOKEN: ${{ github.token }}

No results for the commit

When the checkout holds no results for its commit, ci report posts a failure status with this description.

run failed: Testkit found no results for this commit

If the commit status reads that way, run testkit run before the report in the same checkout.

Report more than one run

A job that runs Testkit more than once passes --context <name> to each report, so each run keeps its own status and its own comment. The default context is testkit.

The jobs in Split a slow gate report on the same commit and the same pull request. Give each of those jobs its own context too.

- run: npx testkit ci report --context testkit-${{ matrix.type }}
  if: always()
  env:
    GITHUB_TOKEN: ${{ github.token }}

With a context other than testkit, the comment's first line is <!-- testkit-ci-report:<context> -->.

Pass a run whose failures are all filed

Every failing test fails the run by default. To keep CI passing while a filed bug waits for its fix, set bugs.knownFailures to "allow". A run then passes when an open issue with the testkit label covers each failure. A new failure still fails the run.

With "allow", the run reads the open issues. Pass the job's token to the npm run testkit step as well.

- run: npm run testkit
  env:
    GITHUB_TOKEN: ${{ github.token }}

The job also needs the issues: read permission.

Split a slow gate

Run one test type per job. Each job passes its type to the script.

jobs:
  testkit:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        type: [contract, int, ui, dal]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx testkit setup playwright --with-deps
        if: matrix.type == 'ui'
      - run: npm run testkit -- ${{ matrix.type }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: testkit-results-${{ matrix.type }}
          path: .testkit/results
          include-hidden-files: true

List only the types the repo has. A type with no files is an empty selection, and it fails its job.

If these jobs run testkit ci report, give each job its own context. Report more than one run shows the step.

To run more than one file at a time inside a job, raise execution.taskConcurrency.

Read a failed job

The job log holds the same lines as a local run. The testkit-results artifact holds everything else: service logs, engine output, and each file's artifacts. Read a run's results shows how to find a file in it.