Flaky has a cause.

testkit shake reruns your branch's new tests under the conditions that make tests flaky, compares each with a control, and gives you the command that brings the failure back.

npm install --save-dev @gfdelarue/testkitnpx testkit shake --since origin/main

Get startedThe CLI is free.

ConditionWhat changesRunsFailedVerdict
control Nothing changes Every run passed 0 of 5 steady
shuffle Vitest files and their tests run in a random order Every run passed 0 of 5 no effect
copies Each file runs beside two copies of itself Every run passed 0 of 5 no effect
timezone-east TZ is set 14 hours ahead of UTC Every run failed 5 of 5 reproduces p 0.004
timezone-west TZ is set 11 hours behind UTC Every run passed 0 of 5 no effect
load Every CPU core is busy Every run passed 0 of 5 no effect
One new test under testkit shake, five runs per condition. The test dates an invoice issued at noon UTC on 1 March. Fourteen hours ahead of UTC, it is already 2 March.

Get a command that reproduces it.

When a time zone or a shuffle seed is the cause, the report names the test and gives a testkit run command that makes it fail. Testkit runs that command once more to check.

.testkit/shake/<id>/shake.md
## timezone-east: reproduces - billing/invoice.test.ts › dates an invoice issued at noon UTC on 1 March failed in 5 of 5 runs (95% CI 57–100%), and in 0 of 5 control runs (95% CI 0–43%) Reproduce: npx testkit run --file billing/invoice.test.ts --timezone Pacific/Kiritimati --retries 0 Testkit ran the first command once more after the shake, and the failure came back.

File it with the condition attached.

testkit bug record keeps the time zone, the shuffle seed, and the retry setting. testkit bug file reruns the test with them and puts them in the issue's reproduce command.

An example issue that testkit bug file writes

#214 Invoices take their date from the server's time zone

  • testkit
  • severity:medium
  • area:billing
Root cause
issueDate() reads the local date, so a server more than 12 hours ahead of UTC dates a noon invoice the next day.
Reproduce
npx testkit run --file billing/invoice.test.ts --timezone Pacific/Kiritimati --retries 0

Blame needs evidence.

A condition gets the blame only when the test fails more often under it than in the control. Testkit checks the gap with Fisher's exact test and corrects for the number of tests, so one test failing by chance among many doesn't count. When chance could explain the gap, the verdict is inconclusive.

A shake turns off the retries your Vitest or Playwright config sets. A test that flakes for another reason, such as a slow third-party API, fails in the control too, and the control's verdict is flaky. For a test that fails only now and then, use --runs 10 or more.

Now shake yours.

npm install --save-dev @gfdelarue/testkitnpx testkit shake --since origin/main

Get startedThe CLI is free. Turnstone has a free tier.