Production Debugging skill for Codex

Done well means: users are protected first (mitigate before diagnosing when impact is ongoing), evidence is gathered with the least intrusive tool that answers the question, nothing in production changes without explicit approval, and every action taken is written down with a timestamp.

Ground rules

• Read-only by default. Querying metrics, logs, traces and kubectl get/describe/logs is fine. Restarts, scaling, config changes, flag flips, kubectl exec into a production container, attaching debuggers, heap dumps and database writes each require the user’s explicit go-ahead for that specific action.
• Never paste secrets, tokens or customer data from production into chat, tickets or commits. Summarise or redact.
• Prefer mitigation over cleverness during user impact: rollback, failover, disable the feature flag, shed load. Diagnose afterwards with the evidence you preserved.
• Keep a running log (time, command, result) in the incident channel or a scratch file; it becomes the postmortem timeline.
• In a Codex sandbox, production access usually needs network and credentials that may be unavailable; say so and hand the user exact commands to run instead of guessing.

Workflow

• 1. State the symptom precisely: what, since when, who is affected, how measured
• 2. Check what changed: deploys, config, flags, traffic, dependencies, certificates, cron
• 3. Decide: is impact ongoing and is there a safe mitigation? If yes, propose it now
• 4. Narrow with telemetry: which service, endpoint, instance, region, tenant
• 5. Inspect a live process only if telemetry cannot answer, with the least intrusive tool
• 6. Form one hypothesis, predict what you should see, test it
• 7. Fix or mitigate with approval; verify with the same signal that showed the problem
• 8. Preserve evidence and timeline for the postmortem

Narrowing with telemetry

# what changed
git log --since="6 hours ago" --oneline origin/main
kubectl rollout history deploy/checkout -n prod
kubectl get events -n prod --sort-by=.lastTimestamp | tail -30

# is it one instance or all?
kubectl top pods -n prod -l app=checkout
kubectl get pods -n prod -l app=checkout -o wide       # same node? same zone?

Questions that split the space fastest:

Question	If yes
Started exactly at a deploy?	Roll back first; diff the release
Only some instances/nodes/zones?	Instance-level: noisy neighbour, bad node, config drift; cordon or replace
Only some tenants/endpoints?	Data-dependent: specific input, large account, hot key
Correlates with traffic?	Capacity: saturation, pool exhaustion, rate limits
Correlates with a dependency’s latency?	Downstream: check its dashboards and status, timeouts and retries
Periodic?	Cron, batch jobs, GC cycles, cache expiry, certificate or token refresh

Live inspection, least intrusive first

Need	Tool	Overhead and risk
What is every thread doing (JVM)	jcmd <pid> Thread.print	Brief safepoint pause; safe
CPU hot spots (JVM)	async-profiler: asprof -d 30 -e cpu -f /tmp/cpu.html <pid>	Low; needs the binary in the container
What is Python doing	py-spy dump --pid <pid>; py-spy record -d 30 -o /tmp/p.svg --pid <pid>	Low; needs SYS_PTRACE capability
Go goroutines / CPU / heap	curl -s localhost:6060/debug/pprof/goroutine?debug=2; go tool pprof http://.../profile?seconds=30	Low, if pprof endpoint exists on an internal port
Node.js	--cpu-prof on a canary, or kill -USR1 <pid> to open the inspector (local port only)	Inspector pauses on breakpoints; never on all replicas
Open connections, sockets	ss -tanp, ss -s, lsof -p <pid> | wc -l	Safe
Syscalls	strace -f -p <pid> -c for a few seconds	High overhead; seconds only, one instance
Kernel-level latency	bpftrace/perf	Low but needs privileges; platform team approval
Heap dump	jcmd <pid> GC.heap_dump /tmp/h.hprof, gcore	Pauses the process for seconds to minutes and writes a file the size of the heap, which contains customer data

On Kubernetes, use an ephemeral debug container instead of installing tools in the app image (requires approval; it attaches to a live pod):

kubectl debug -n prod -it pod/checkout-7d9f -c debugger \
  --image=nicolaka/netshoot --target=app --profile=general -- bash

--target shares the app container’s process namespace so you can see its PIDs. Tools needing ptrace (py-spy, gdb) may need --profile=sysadmin, which the cluster policy may forbid; ask the platform owner rather than working around it.

Take evidence from one instance, ideally one already removed from the load balancer (mark unready, or cordon and let it drain), so a pause does not hurt users.

Dynamic levers

• Raise log level for one service, one instance or one tenant at runtime if the app supports it (Spring Boot actuator loggers endpoint, a debug header honoured only from internal networks, a flag). Set an expiry and put it back.
• Feature flags: disabling a new code path is often the fastest safe mitigation; confirm the flag’s blast radius first.
• Traffic shifting: send a slice of traffic to a canary with extra instrumentation.

Hypothesis discipline

Write each hypothesis as: “If X is the cause, then Y should be true; I will check Z.” One change at a time. If two changes are made together and the problem goes away, you do not know which fixed it, and the postmortem will be guesswork. For deeper method, see $systematic-debugging.

Verification

• The signal that showed the problem (error rate, latency percentile, failing synthetic) returns to baseline and stays there for at least one full traffic cycle you care about (often 30-60 minutes, or the next batch run).
• Temporary levers are reverted: log levels, debug containers (kubectl get pod -o yaml shows ephemeral containers until the pod is replaced), cordons, flags.
• Evidence files (profiles, dumps) are moved to approved storage or deleted; heap dumps contain customer data and need the same handling as a database copy.

Output

## Production debug log: <symptom>
- Impact: <who, what, since>
- Changes in window: <deploys, flags, config>
- Mitigation: <action, approved by, time, effect>
- Evidence: <queries, profiles, dumps (where stored)>
- Hypotheses tested: <hypothesis -> result>
- Root cause (or current best explanation): <...>
- Follow-ups: <fix, test, alert, runbook>
- Temporary changes reverted: <list with times>

Gotchas

• Restarting “to see if it helps” destroys the evidence (memory state, stuck threads); grab a thread dump or profile from one instance first if it is safe to wait a minute.
• Debug logging at full volume in production can itself cause an outage (disk, log pipeline, cost); scope it narrowly.
• kubectl exec and debug containers bypass the normal deploy path and may violate compliance rules; check policy and record what was done.
• Time zones: align everything to UTC before correlating events from different tools.