Flaky Test Triage skill for Codex
Name the cause, prove the fix, retire the retry. Codex reproduces an intermittent failure in a rerun loop before touching code, classifies it (test pollution, async race, clock, randomness, parallel resource clash, leak) and fixes the nondeterminism itself instead of adding sleeps or retries. It proves the fix with a statistically meaningful number of clean runs and keeps quarantined tests visible with owners and due dates.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- flaky-test-triageInstalls to ~/.agents/skills/flaky-test-triage/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install flaky-test-triageThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project flaky-test-triageInstalls to .agents/skills/flaky-test-triage/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/flaky-test-triage.zipunzip -o flaky-test-triage.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) flaky-test-triagePowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $flaky-test-triage, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Diagnoses and fixes flaky tests across pytest, Jest, Vitest, Go, JUnit, RSpec and Playwright - measures the failure rate by rerunning, classifies the cause (order dependence, async timing, shared state, time zones, randomness, network, resource leaks), fixes the root cause instead of adding sleeps or retries, and runs a quarantine process with owners. Use when a test passes locally but fails in CI, fails intermittently, only fails in parallel or in a certain order, or when CI retries are hiding a growing list of unreliable tests.
Use it when
- A test passes locally and fails intermittently in CI
- Failures depend on test order, worker count, time of day or time zone
- CI retries mask a growing set of unreliable tests
- You need evidence a flake is really fixed before closing the ticket
Use something else when
- The flake is inside a Playwright spec and you want framework-specific fixes: use playwright-e2e
- The test fails every time; that is a regression: use systematic-debugging
How to check the result
The test passes the number of repeated runs that matches its old failure rate (for example 150 clean runs for a 2% flake) at CI parallelism, and still fails when the code it covers is broken.
The mistake it prevents
Adding a sleep, a longer timeout or a blanket retry, which turns a visible race into an invisible one that returns on slower runners.
A flaky test is fixed when the cause is named, the fix removes the nondeterminism (not the symptom), and the test survives enough repeated runs to make the old failure rate statistically implausible. A retry that turns red into green is a measurement, not a fix.
Inspect first
- Read
AGENTS.mdfor the test command, CI config and any existing quarantine convention. - Collect evidence before editing: the failing CI log, the error message and stack, which job, runner OS, shard and worker count, and how often it fails (CI history or the flaky-test report from Playwright, BuildPulse, Datadog, Trunk, Buildkite Test Engine).
- Diff the failing environment against local: parallelism, test order, time zone, locale, CPU count, container vs host, database version, network access.
- Read the test and every fixture or
beforeEachit uses. Most flakes live in setup.
Measure the failure rate
Reproduce before fixing; a fix you cannot reproduce is a guess. Rerun the single test in a tight loop with the same parallelism CI uses:
python3 scripts/rerun.py -n 50 --stop-after 3 -- pytest tests/test_orders.py::test_refund -q -p no:cacheprovider
python3 scripts/rerun.py -n 100 -- go test ./internal/queue -run '^TestDrain$' -count=1 -raceThe script records exit codes and durations, saves output of failing runs to .flaky-logs/,
and exits 1 if any run failed. Framework-native loops are often faster:
| Runner | Repeat | Randomise order / parallel |
|---|---|---|
| pytest | pip install pytest-repeat; pytest --count=50 -x tests/x.py::test_y | pytest-randomly with -p randomly --randomly-seed=1234; pytest -n 8 (xdist) |
| Jest | `for i in $(seq 50); do npx jest x.test.ts | |
| Vitest | vitest run x.test.ts --retry=0 in a loop | --sequence.shuffle --sequence.seed=1234 |
| Go | go test -run '^TestY$' -count=200 -race ./pkg | -shuffle=on (prints the seed), -parallel 16 |
| JUnit 5 | @RepeatedTest(100) temporarily | junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$Random |
| RSpec | loop with --seed | --order random --seed 1234; rspec --bisect --seed 1234 finds the polluting spec |
| Playwright | npx playwright test x.spec.ts --repeat-each=30 --workers=4 | --fully-parallel; open the trace of a failing run |
How many clean runs prove a fix? With zero failures in n runs, the true failure rate is below about 3/n at 95% confidence (rule of three):
| Old failure rate | Clean runs needed |
|---|---|
| 10% | 30 |
| 5% | 60 |
| 2% | 150 |
| 1% | 300 |
If the failure only reproduces in CI, add a temporary CI job that loops the test, and stress
the machine locally (stress-ng --cpu 8 or run the suite twice in parallel) to widen races.
Classify the cause
| Symptom | Likely cause | Confirm by |
|---|---|---|
| Fails only in the full suite, passes alone | Test pollution: shared global, singleton, module cache, DB rows, env vars, mocks not restored | Bisect order (rspec --bisect, pytest -p randomly with failing seed, run halves) |
| Fails only in parallel | Shared resource: same DB, port, temp file, fixed username | Run with 1 worker; grep for hard-coded ports, paths and unique keys |
| Fails on slow CI, passes locally | Race with async work, fixed sleeps, too-short timeout, assertion before render | Add CPU stress locally; look for sleep, setTimeout, missing await |
| Fails at certain times or dates | Clock: midnight, month end, DST, leap day, now() read twice | Freeze time at the failing instant |
| Fails in CI time zone or locale only | TZ, locale-dependent formatting or sorting | TZ=UTC vs TZ=America/Los_Angeles; LANG=C |
| Different data each run | Unseeded randomness, Faker, UUID ordering, hash or map iteration order, unordered SQL | Log the seed; check ORDER BY and set/dict comparisons |
| Timeouts, connection refused | Real network, service not ready, container health not awaited | Block network; check readiness waits |
| Gets worse as the suite grows | Resource leak: open handles, connections, goroutines, file descriptors | Jest --detectOpenHandles, goleak, pool stats |
| Browser test, element not found or wrong element | Locator ambiguity, animation, navigation not awaited | Playwright trace, Cypress video/screenshots |
Fix patterns
- Async timing. Wait for the condition, not for time. Replace
sleep(2)with a polling assertion (expect.poll, Playwright web-first assertions,Eventuallyin Go testify, Awaitility in Java,wait_forhelpers). Await every promise; enable@typescript-eslint/no-floating-promisesin tests. - Time. Inject a clock or freeze it (
vi.useFakeTimers().setSystemTime(...),jest.useFakeTimers(),freezegun/time-machine,page.clock). Compute “now” once. - Ordering. Assert on sorted values or use order-insensitive matchers; add
ORDER BYin the query under test if order matters to users. - Randomness. Seed generators per test and print the seed on failure.
- Isolation. Each test creates its own data with unique keys (sequence or UUID), uses
a per-worker database or schema, binds port 0, writes to a per-test temp dir, and
restores mocks and env (
vi.restoreAllMocks(),monkeypatch,t.Setenv). - External services. Mock at the HTTP boundary or run a container with a real readiness check; never call shared staging APIs from unit or integration tests.
- Leaks. Close servers, pools and browsers in teardown; fail the suite on leaked handles.
If the test reveals a real race in production code, the fix belongs in the product, not the test. Say so explicitly; that is the most valuable outcome of a flaky-test investigation.
Quarantine policy
Quarantine keeps the main branch trustworthy while a fix is pending. It is a ticket, not a graveyard.
- Tag the test (
@pytest.mark.flaky_quarantine,test.fixme,t.Skip("quarantine: #123"),@Tag("quarantine")) and link an issue with an owner and a due date - Keep running quarantined tests in a separate non-blocking CI job so data keeps coming
- Review the list weekly; anything older than the agreed limit (e.g. 30 days) is fixed or deleted with the owner’s agreement
- Never quarantine tests covering payments, auth or data deletion without the owner’s explicit sign-off
Global retries (retries: 2, --reruns 2) are acceptable only if CI reports retried-then-
passed tests as flaky (Playwright does by default; --fail-on-flaky-tests makes them fail
on 1.52+), so they stay visible.
Verification
- Reproduce the failure and record the rate before the fix.
- Apply the fix; rerun with the clean-run count from the table at CI parallelism, plus a shuffled-order run of the whole file or package.
- Confirm the fix did not weaken the test: temporarily break the code it covers and check the test still fails.
- Remove temporary
@RepeatedTest, loops and debug logging before committing.
Output template
## Flaky test: <test id>
**Observed:** failed 7/50 local runs at 8 workers (14%); 11 CI failures in 14 days
**Cause:** <category> - <one-sentence mechanism, e.g. test_refund reads orders created by
test_bulk_import because both use customer id 42 in the shared DB>
**Evidence:** <seed, log excerpt path, trace, bisect result>
**Fix:** <what changed and why it removes the nondeterminism>
**Proof:** 0/150 failures after fix (rate < 2% at 95%); mutation check: test still fails when <X> is broken
**Product bug?:** yes/no - <details>Gotchas
- Raising a timeout hides a race until the next slower runner.
--runInBandor-p 1fixing the failure is a diagnosis, not a fix; CI time will double.- Deleting or skipping the test removes coverage; get the owner’s agreement first.
- A flake that disappears after a dependency bump has usually moved, not vanished.
What the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Flaky Test Triage"
short_description: "Name the cause, prove the fix, retire the retry"
default_prompt: "$flaky-test-triage test_refund fails about once in ten CI runs, figure out why and fix it"scripts/rerun.py
#!/usr/bin/env python3
"""Run a test command repeatedly and measure how often it fails.
Usage:
python3 rerun.py -n 30 -- pytest tests/test_orders.py::test_refund -x -q
python3 rerun.py -n 50 --stop-after 3 --logs .flaky-logs -- go test ./pkg/queue -run TestDrain -count=1
Each run's exit code and duration are recorded; output of failing runs is saved to
--logs (default .flaky-logs/) so the failures can be compared. Exits 0 when every run
passed, 1 when at least one failed (flaky or broken), 2 on usage errors.
"""
from __future__ import annotations
import argparse
import os
import statistics
import subprocess
import sys
import time
from pathlib import Path
def main() -> int:
parser = argparse.ArgumentParser(description="Rerun a test command and report its failure rate.")
parser.add_argument("-n", "--runs", type=int, default=20, help="number of runs (default 20)")
parser.add_argument("--stop-after", type=int, default=0, help="stop after this many failures (0 = never)")
parser.add_argument("--timeout", type=float, default=0, help="per-run timeout in seconds (0 = none)")
parser.add_argument("--logs", default=".flaky-logs", help="directory for failing-run output")
parser.add_argument("command", nargs=argparse.REMAINDER, help="test command after --")
args = parser.parse_args()
cmd = args.command[1:] if args.command[:1] == ["--"] else args.command
if not cmd or args.runs < 1:
parser.print_usage(sys.stderr)
print("error: give a command after -- and a run count >= 1", file=sys.stderr)
return 2
logs = Path(args.logs)
results: list[tuple[int, int, float]] = [] # (run, exit code, seconds)
for i in range(1, args.runs + 1):
start = time.monotonic()
try:
proc = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
timeout=args.timeout or None,
env={**os.environ, "FLAKY_RERUN_INDEX": str(i)},
)
code, out = proc.returncode, proc.stdout
except subprocess.TimeoutExpired as exc:
code, out = 124, (exc.stdout or b"") + b"\n[rerun.py] TIMEOUT\n"
except FileNotFoundError:
print(f"error: command not found: {cmd[0]}", file=sys.stderr)
return 2
elapsed = time.monotonic() - start
results.append((i, code, elapsed))
mark = "PASS" if code == 0 else f"FAIL({code})"
print(f"run {i:>3}/{args.runs} {mark:<9} {elapsed:7.2f}s", flush=True)
if code != 0:
logs.mkdir(parents=True, exist_ok=True)
(logs / f"run-{i:03d}.log").write_bytes(out)
failures = sum(1 for _, c, _ in results if c != 0)
if args.stop_after and failures >= args.stop_after:
print(f"stopping: reached {failures} failures")
break
total = len(results)
failed = [r for r in results if r[1] != 0]
durations = [r[2] for r in results]
print("\nsummary")
print(f" command : {' '.join(cmd)}")
print(f" runs : {total}")
print(f" failures : {len(failed)} ({100 * len(failed) / total:.1f}%)")
print(f" duration : median {statistics.median(durations):.2f}s, max {max(durations):.2f}s")
if failed:
pass_d = [d for _, c, d in results if c == 0]
fail_d = [d for _, c, d in failed]
if pass_d:
print(f" pass med : {statistics.median(pass_d):.2f}s fail med: {statistics.median(fail_d):.2f}s")
print(f" fail runs : {', '.join(str(r[0]) for r in failed)}")
print(f" logs : {logs}/run-NNN.log")
verdict = "BROKEN (fails every run)" if len(failed) == total else "FLAKY"
else:
verdict = "STABLE in this sample (not proof; see sample-size table in SKILL.md)"
print(f" verdict : {verdict}")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.