Test Strategy skill for Codex

A test strategy is good when every important risk in the system has a named layer that catches it, each layer is the cheapest one able to catch that risk, the whole suite gives a trusted signal within the team’s CI budget, and the plan to get there is incremental. Pyramid, trophy and honeycomb are outcomes of that analysis, not starting points.

Inspect first

1. Read AGENTS.md, README, CI workflows and the test directories. Note languages, frameworks and how tests are run and split.
2. Inventory the current suite with numbers, not impressions:

# counts by type (adapt globs to the repo's conventions)
rg --files -g '*.test.*' -g '*_test.go' -g 'test_*.py' -g '*Test.java' | wc -l
rg --files -g '*.spec.ts' -g 'e2e/**' -g 'cypress/**' | wc -l
gh run list --workflow ci.yml --limit 50 --json conclusion,createdAt,updatedAt   # duration and failure rate; needs network

3. Collect escaped defects: the last 20-30 production bugs or incidents. For each, ask which layer should have caught it. This is the strongest evidence you will get.
4. Map the architecture: UI-heavy frontend, API service, data pipeline, library, event-driven system, mobile app; and the external dependencies.
5. Ask the user for constraints: CI time budget, release cadence, regulated areas, team familiarity with tools.

Risk map first

List the system’s risks, then assign each to the cheapest layer that can catch it reliably.

Risk	Cheapest reliable layer
Type errors, null handling, unused code, unsafe patterns	Static: type checker, linter, SAST
Business rules, calculations, parsing, state machines	Unit (plus property-based for wide input spaces)
SQL, transactions, migrations, serialisation, framework wiring	Integration with real DB/broker
Breaking another team’s API or being broken by it	Contract tests
Auth, routing, validation at the HTTP edge	In-process API/integration tests
Critical user journeys across UI and backend	A few E2E tests
Layout and styling regressions	Visual regression on components/pages
Accessibility defects	axe scans + manual pass
Performance and capacity	Benchmarks, load tests with thresholds
Unknown unknowns, UX problems	Exploratory testing sessions
Resilience to dependency failure	Fault injection, chaos experiments

Choose the shape

System	Shape that usually fits	Why
Library, SDK, algorithm-heavy core	Pyramid: mostly unit + property-based	Logic is the product; few boundaries
CRUD API / backend service	Honeycomb: mostly integration with real DB, fewer pure units	Bugs live in queries and wiring, not in trivial functions
React/Vue/Svelte frontend	Trophy: static + component/integration tests via Testing Library, thin E2E	Component behaviour is where users feel bugs
Microservices	Integration per service + contract tests between them; very few cross-service E2E	Cross-service E2E environments are slow and flaky
Data pipelines	Data-quality checks + transformation unit tests + small end-to-end run on fixture data	Data shape is the main risk
Mobile app	Unit + UI component tests, a small device E2E smoke suite	Device E2E is costly to run and maintain

Escape hatch: if escaped defects cluster in one layer the shape says is minor, the evidence wins over the shape.

CI budget

Set targets with the team, then design to them:

Stage	Contents	Target
Pre-commit / local	Format, lint, type check, affected unit tests	< 1 min
PR required checks	Static, all unit, integration, contract, smoke E2E	< 10-15 min wall clock
Merge to main	Full E2E, visual, a11y	< 30 min
Nightly / scheduled	Long fuzzing, load tests, full cross-browser, quarantine job	Unbounded, but reviewed daily

Use sharding and test selection (see test-impact-analysis) before cutting tests that catch real defects.

Diagnose the current suite

Symptom	Likely imbalance	Move
Unit tests green, production breaks at DB or API edges	Too many mock-heavy units, too few integration	Replace mock-the-repository tests with real-DB integration tests
CI takes 45+ minutes, mostly E2E	E2E covering rules that lower layers could	Push permutations down; keep one E2E per critical journey
Refactors break hundreds of tests	Tests coupled to implementation (mocks of internals, snapshots)	Test through public interfaces
Frequent flaky failures	E2E and timing-dependent tests	flaky-test-triage; reduce E2E count
Cross-team releases break each other	No contract tests	Add consumer-driven contracts
Bugs found only by users in UX	No exploratory testing	Time-boxed charters before releases (test-plan-writing)

Workflow

• Inventory the suite and CI timings; collect escaped defects
• Build the risk map for this system; mark which risks have no layer today
• Choose the target shape and justify it with the defect data
• Define per-layer rules: what belongs there, what does not, tooling, ownership
• Define CI stages and time budgets
• Write a staged plan: three to five steps, each shippable on its own, highest-risk gap first
• Define how success is measured: escaped defects per release, CI time, flake rate, time to diagnose a failure

Do not propose rewriting the suite. Propose the next three changes that move the measurements, and say what to stop doing.

Output template

# Test strategy: <system>
**Context:** <architecture, team size, release cadence, CI budget>
**Evidence:** <suite inventory, CI duration p50/p90, flake rate, escaped defects by layer>

## Risks and the layer that owns each
| Risk | Layer | Tooling | Today |
|---|---|---|---|

## Target shape
<shape + why, with the defect evidence>

## Layer rules
- Unit: <what belongs, what does not>
- Integration: <real deps, isolation strategy>
- Contract / E2E / manual: <scope and limits, e.g. "max 15 E2E journeys">

## CI stages and budgets
## Plan
1. <change> - expected effect - owner - done when
## Stop doing
## Measures of success

Verification

• Every risk in the map has an owning layer, or is explicitly accepted with a reason.
• Each of the sampled escaped defects maps to a layer in the target strategy that would catch it.
• The plan’s first step is achievable in about a sprint and measurable.

Gotchas

• Copying the test pyramid onto a thin CRUD service produces hundreds of mock-only unit tests that never catch a real bug.
• Coverage targets are not a strategy; they measure execution, not risk.
• “Everything through E2E” feels thorough and produces a slow, flaky suite nobody trusts.
• Strategies without CI budgets drift: each new test is reasonable, the sum is an hour.