Root Cause Analysis skill for Codex

A good RCA answers three questions with evidence, not opinion: why did the failure happen, why did our defences not stop it, and why did it take as long as it did to detect and recover. It ends in a small set of actions that each remove or weaken a specific link in the causal chain, with an owner and a way to verify it worked.

Scope and boundaries

• This skill analyses causes. Finding the bug in code is systematic-debugging; running a live incident is incident-command; publishing the blameless narrative document for the wider org is postmortem-writing. An RCA often feeds that postmortem.
• Analyse systems and decisions, never individuals. “Engineer forgot X” is never a root cause; the question is why the system made forgetting X easy and silent.
• If the facts are not yet known, say so. An RCA built on assumptions is worse than an RCA marked “draft, pending log export”.

Workflow

- [ ] 1. Define the problem statement and impact
- [ ] 2. Gather evidence and build the timeline
- [ ] 3. Map the causal chain (five whys, multiple branches)
- [ ] 4. Check the chain with a fault tree
- [ ] 5. Classify causes: trigger, root, contributing
- [ ] 6. Derive actions and rank them
- [ ] 7. Write the RCA document and review it with the user

1. Problem statement

One paragraph, measurable, no causes yet:

Between 14:02 and 14:51 UTC on 2026-03-11, 18% of checkout requests returned HTTP 500. About 2,300 orders failed; no data was lost. Detected by a customer support ticket at 14:20.

Pin down: start and end times, blast radius (users, requests, money, data), how it was detected, and who declared it resolved.

2. Evidence and timeline

Collect primary sources before interviewing memory: deploy logs, git log, CI runs, alerts, dashboards, chat transcripts, tickets. In a repo, the useful commands are:

git log --since="2026-03-10" --until="2026-03-12" --format='%h %ad %an %s' --date=iso
git log -S "MAX_POOL_SIZE" --oneline        # when a constant appeared or vanished
gh run list --workflow deploy.yml --limit 20  # needs network and gh auth

Build a UTC timeline. Mark each entry with its source so readers can audit it:

13:48  PR #4412 merged (pool size 50 -> 10)                    [git]
13:55  Deploy v2026.03.11-2 to prod                             [deploy log]
14:02  p99 checkout latency 300ms -> 9s                         [dashboard]
14:20  First customer ticket                                     [support]
14:31  On-call paged manually                                    [chat]
14:44  Rollback started                                          [deploy log]
14:51  Error rate back to baseline                               [dashboard]

The gaps are findings: 18 minutes from impact to first signal, 11 more to a page.

3. Five whys, done properly

Plain five whys fails in two ways: it follows one chain, and it stops at a person. Do it like this:

• Start from the problem statement, not from your favourite theory.
• Every “because” must cite evidence from step 2. If it cannot, mark it (unverified).
• Branch whenever more than one condition was needed. Most failures need several.
• Run three separate chains: occurrence (why it happened), escape (why review, tests, staging and canaries missed it), and response (why detection and recovery took as long as they did).
• Stop when the next “why” leaves your organisation’s control, or when the answer points to a decision, process or design you can change.

Occurrence
  Why 500s?            DB connection acquisition timed out.               [app logs]
  Why timeouts?        Pool size 10 under peak 40 concurrent requests.    [config, metrics]
  Why 10?              PR #4412 copied the value from the worker config.  [git]
  Why not noticed?     -> escape chain
  Why possible?        Pool size is an untyped env var with no per-service default doc.
Escape
  Why not caught in staging?  Staging peak is 3 concurrent requests.      [metrics]
  Why no load check?          No pre-prod load test for config-only changes.
Response
  Why 18 min to detect?  Alert on 5xx rate requires 15 min sustained window.
  Why 11 min to page?    Alert routes to a channel, not the pager.

4. Fault tree check

Draw the top event and the conditions beneath it with gates. An AND gate means every child was required, so removing any one child prevents the event; those are your cheapest intervention points. An OR gate means any child alone suffices, so you must address all of them or move higher up.

                 [Checkout 500s for 49 min]
                            AND
        +-------------------+--------------------+
 [Pool exhausted]   [Not caught pre-prod]   [Slow detection+recovery]
       AND                  AND                      OR
  +----+-----+        +-----+------+          +------+-------+
[size=10][peak 40] [no load test][staging  [15-min alert  [alert not
                                   idle]     window]        paged]

If the tree shows an OR gate you did not cover with an action, the RCA is incomplete.

5. Classify

Class	Definition	Example
Trigger	The event that started it; often blameless and unavoidable	The deploy at 13:55
Root cause	A condition that, if removed, prevents this and similar failures	Pool size is unvalidated config with no safe default
Contributing	Made it worse, more likely, or slower to fix	Staging traffic too low; slow alert
Non-cause	Investigated and ruled out; list it to stop rumours	The DB version upgrade on 03-09

Expect one or two root causes and several contributors. Ten “root causes” means you have not separated them.

6. Actions

Each action must name the causal link it breaks. Rank by effect on recurrence, not by ease. Use these categories and aim for at least one in each of the first three:

Category	Question	Example
Prevent	Can this not happen again?	Validate pool size at startup; fail fast below 20
Detect	Would we see it in minutes?	Page on 5xx rate > 2% for 3 min
Mitigate	Is the blast radius smaller?	Canary 5% for 10 min before full rollout
Process	Does the next person have the knowledge?	Config ownership in CODEOWNERS

Every action gets: owner (a role or person the user names), due date, tracking link, and a verification (“Load test at 50 concurrent shows no acquisition timeouts”). Reject vague actions: “be more careful”, “improve monitoring”, “add more tests”.

Hard rules

• Blameless language: describe what the system allowed, not what a person failed to do.
• Distinguish facts, inferences and unknowns in the text. Use (unverified) liberally.
• Do not fold the fix into the RCA as if it settles everything; the fix addresses the trigger path, actions address the class of failure.
• Do not post the RCA to a channel, wiki or ticket without the user’s explicit go-ahead.

Verification

Before handing the document over, check:

• Every timeline entry has a source.
• Every “because” in the chains cites evidence or is marked unverified.
• Every root and contributing cause maps to at least one action, or an explicit “accepted risk” line with a reason.
• Every OR gate in the fault tree is covered.
• No sentence names an individual as the cause.

Output template

# RCA: <short title> (<date>)
Status: draft | reviewed | closed        Author: <name>        Severity: <level>

## Summary
<3-4 sentences: what happened, impact, root cause in plain words, headline action>

## Impact
- Duration: <start-end UTC>   Users/requests affected: <n>   Data loss: <yes/no>
- Detection: <how and after how long>

## Timeline (UTC)
<table or list with [source] tags>

## Causal analysis
### Occurrence chain
### Escape chain
### Response chain
### Fault tree
<diagram>

## Causes
- Trigger: ...
- Root cause(s): ...
- Contributing: ...
- Ruled out: ...

## Actions
| # | Action | Breaks link | Category | Owner | Due | Verify by |
|---|---|---|---|---|---|---|

## What went well
<detection or recovery steps worth keeping>

## Open questions