Systematic Debugging skill for Codex

Done means three things are true: you can state the causal chain from trigger to symptom in two sentences, a check that failed before your change passes after it, and the diff contains nothing that the explanation does not require. Anything less is a guess that happened to coincide with the symptom going quiet.

Hard rules

• No production edits before a reproduction exists, or before you have written down why one is impossible and what you will use instead.
• One variable per experiment. If an experiment proves nothing, undo it before the next.
• Never “fix” by widening a catch, adding a sleep, adding a retry, or special-casing the failing input. Those hide the mechanism; they do not remove it.
• Two failed fixes in a row means your model of the bug is wrong. Stop editing, go back to observation, and collect new evidence.
• Interactive debuggers (pdb, node inspect, dlv without a script) block a non-interactive Codex shell. Use logging, assertions, env-var diagnostics and scripted runs instead.

Before the first command

1. Read AGENTS.md for the test command, how to run a single test, and any services that must be running. Use that command, not one you guess.
2. Write the bug as a contract: “Given X, calling Y should produce A; it produces B.” If you cannot fill in A, ask the user what correct looks like.
3. Collect the facts that change the search: exact error text, versions in use, last known good commit or release, and whether it fails everywhere or only somewhere.

git log --oneline -20 -- <suspect-path>      # recent changes near the failure
git diff --stat <last-good>..HEAD            # size of the haystack since it worked
git diff <last-good>..HEAD -- package-lock.json poetry.lock go.sum Cargo.lock | head -80

Classify, then pick the first probe

Bug shape	First probe
Exception with a stack trace	Open the deepest frame in project code; print the inputs that reached it
Wrong value, no error	Trace the value backwards to where it was first computed or parsed
Worked at a known commit	Hand off to bisect: git bisect run <check> (see git-bisect)
Only fails in CI / one machine	Diff the environments before reading code (table below)
Hang or timeout	Get a thread/goroutine dump while it is stuck
Intermittent	Measure the failure rate first; see flaky-test-triage for test-only flakes
Fix did not stick	Re-run the original reproduction; the first fix probably addressed a sibling path

Environment diff checklist

When one environment fails and another passes, compare these before opening source:

• Runtime and tool versions (node -v, python3 -VV, go version, java -version).
• Resolved dependency versions, not ranges (npm ls <pkg>, pip show <pkg>, go list -m all).
• Environment variables and config precedence (which .env file, which profile).
• Locale, timezone (TZ), filesystem case sensitivity, line endings, CPU architecture.
• Data: row counts, encodings, a record that exists in one place only.
• Network: the Codex sandbox often has network disabled, so a call that silently falls back or times out locally may behave differently in CI. Check for that first.

The loop

Run this loop until the explanation is complete. Keep a short ledger in the conversation so the user can follow the reasoning:

H1  price arrives as string from CSV importer      test: log typeof at applyDiscount   -> CONFIRMED
H2  discount applied twice on retry                 test: count calls per request        -> ruled out (1 call)
H3  rounding in toFixed                             not tested; H1 explains the output

1. Reproduce with the cheapest harness

Prefer, in order: a failing test in the existing suite, a one-line command, a tiny script in a scratch directory. Narrow the runner to the single case so each iteration is seconds:

pytest tests/test_cart.py::test_bundle_discount -x -q
npx vitest run src/cart.test.ts -t "bundle discount"
go test ./internal/cart -run 'TestBundleDiscount$' -count=1
cargo test bundle_discount -- --nocapture

-count=1 stops Go from serving a cached pass. For intermittent failures, loop and count:

fails=0; for i in $(seq 1 40); do go test ./internal/cart -run TestBundle -count=1 >/dev/null 2>&1 || fails=$((fails+1)); done; echo "fails=$fails/40"

2. Shrink

Remove everything the failure does not need: fewer records, shorter input, fewer steps, one service instead of three. Each removal that keeps the failure is progress; each one that makes it pass tells you that piece matters. A failure that fits in ten lines usually explains itself. For bugs destined for an upstream issue, minimal-repro-cases covers reduction in depth.

3. Explain

Instrument the boundary that separates your hypotheses, not everything at once.

print(f"DBG-7 applyDiscount price={price!r} type={type(price).__name__}", file=sys.stderr)

console.error("DBG-7", { price, type: typeof price, stack: new Error().stack?.split("\n")[2] });

Tag every probe with a unique prefix (DBG-7) so grep -rn "DBG-" src/ finds them all for removal. Built-in diagnostics that need no code change:

Runtime	Diagnostic
Python	python -X dev -X faulthandler script.py; PYTHONASYNCIODEBUG=1 for asyncio
Node	node --enable-source-maps --trace-warnings --stack-trace-limit=50 app.js
Go	GOTRACEBACK=all; go test -race for data races; kill -QUIT <pid> dumps goroutines
Rust	RUST_BACKTRACE=1 (or full)
JVM	jcmd <pid> Thread.print for a stuck process

Read errors literally. The message, the first project frame, and any Caused by: or During handling of the above exception section carry more signal than framework frames.

4. Change one thing

Make the fix at the origin of the bad state, the place where the wrong value is created or the wrong decision is made, not where it finally explodes. Keep the diff to what the explanation demands. Refactors you noticed along the way go in a follow-up note.

Then hunt for siblings: the same mistake usually exists elsewhere.

rg -n "parseFloat\(row\[" src/      # same parsing pattern in other importers

5. Prove

• The reproduction passes.
• A regression test exists that fails without the fix. Prove it: git stash push -- <fixed-file>, run the test and watch it fail, git stash pop, run it and watch it pass.
• The surrounding suite, type checker and linter pass (commands from AGENTS.md).
• For intermittent bugs, the loop count from step 1 now shows zero failures over a sample large enough to matter (if it failed 1 in 10, run at least 50).
• All DBG- probes are gone: rg -n "DBG-" . returns nothing.

If the regression test passes without the fix, it does not test the bug. Rewrite it.

Hangs and slowness

• Capture where it is stuck before guessing why. Python: py-spy dump --pid <pid> (needs pip install py-spy, which may need network approval) or faulthandler.dump_traceback_later(30). Node: run with --inspect only if the user can attach; otherwise add an unref’d timer that logs process.getActiveResourcesInfo() to see which handles keep the event loop alive.
• A test that never finishes in the sandbox is often waiting on network. Check for DNS or HTTP calls with a short timeout before profiling anything.
• Real performance regressions belong to the profiling skills (node-profiling, python-profiling, go-pprof), not to this loop.

When you cannot reproduce

Say so plainly. Then:

1. List what evidence would discriminate between the remaining hypotheses (a log line, a request ID, a data sample, a version string).
2. Propose diagnostics that capture it on the next occurrence, guarded by a flag or log level, and ask before adding anything that ships to production.
3. If a speculative fix is still wanted, label it speculative in the commit message and in the report, and state what observation would confirm or refute it.

For live systems, hand off to production-debugging; for a written incident analysis, hand off to root-cause-analysis.

Report

### Bug: <one-line contract, e.g. "bundle discount yields NaN total">
**Mechanism:** <trigger> -> <intermediate bad state> -> <symptom>. Introduced in <sha or "unknown">.
**Evidence:** <the probe or test output that confirmed it; hypotheses ruled out and how>
**Change:** <file:line, what changed and why there rather than at the crash site>
**Proof:** <regression test name>, fails on the parent commit, passes now; <suite> green.
**Siblings:** <other places with the same pattern, fixed or listed>
**Open questions:** <anything unverified, or "none">