Postmortem Writing skill for Codex

A good postmortem lets someone who was not there understand what users experienced, why the system allowed it, and what will be different next time. Every claim traces to evidence, no sentence assigns fault to a person, and every action item has an owner, a date and a way to tell it is done.

Before writing anything

1. Collect the raw material and keep it read-only: incident channel export, pager/alert history, deploy and config-change logs, relevant graphs (with timestamps), support ticket counts, status page updates, and any notes the incident commander kept.
2. Read the team’s existing template. Search the repo or docs folder first (rg -il "postmortem|post-incident|incident review" docs/) and check AGENTS.md for where documents live. Use their headings if they exist; the template below is the fallback.
3. Confirm with the user: severity level, the audience (internal engineering, whole company, or customers), and whether anything is under legal or security embargo. A customer-facing summary is a separate, shorter document derived from this one.

The live response itself (roles, comms cadence, paging) belongs to $incident-command. Deep causal analysis techniques (five whys, fault trees) belong to $root-cause-analysis; this skill turns their output into the written record.

Workflow

- [ ] 1. Build the timeline from evidence, in UTC, one event per line
- [ ] 2. Mark the four anchor times: start of impact, detection, mitigation, resolution
- [ ] 3. Quantify impact in user terms
- [ ] 4. Write the summary last-but-one (after the timeline is stable)
- [ ] 5. Identify contributing factors, not a single root cause
- [ ] 6. Record what went well and where the response was slowed
- [ ] 7. Derive action items from factors; assign owner, date, ticket
- [ ] 8. Run the blame and vagueness pass
- [ ] 9. List open questions instead of guessing

1. Timeline

• Normalise every timestamp to UTC with the date (2026-03-14 09:42 UTC). Chat exports are often in the exporter’s local time; check the offset against an alert timestamp.
• One line per event. Source each line: (alert), (deploy log), (#inc-482).
• Include the quiet gaps. “09:58-10:31 no action; responder investigating DB metrics” is more useful than skipping to the fix, because gaps reveal detection and tooling problems.
• Describe actions, not people’s states of mind: “on-call rolled back to v4.12.3”, not “on-call panicked and rolled back”.

Anchor	Definition	Derived metric
Impact start	First moment users were affected (from metrics, not from the first alert)	-
Detected	First alert or report that a human saw	Time to detect = detected - impact start
Mitigated	User impact stopped or reduced to acceptable	Time to mitigate = mitigated - detected
Resolved	System back to normal state, temporary fixes accounted for	Duration = resolved - impact start

If impact start can only be estimated, say so and give the evidence (“error rate first exceeds baseline at 09:31 on the API dashboard”).

2. Impact

State impact the way an affected customer or the business would describe it:

• Who: which users, regions, plans, tenants. Counts or percentages, not “some users”.
• What: which actions failed or degraded (checkout returned 502, emails delayed ~40 min).
• How much: requests failed, orders lost or delayed, SLO budget burned, revenue estimate if finance supplied one. Mark estimates as estimates.
• Data: was any data lost, corrupted, or exposed? If exposure is possible, stop and tell the user this needs the security or privacy process before the doc circulates.

3. Contributing factors

Complex failures rarely have one root cause. List the conditions that together made the incident possible, grouped so actions map cleanly:

Group	Prompt question	Example factor
Trigger	What change or event started it?	Config push raised pool size above DB max_connections
Latent condition	What was already true that let the trigger hurt?	No validation of pool size against DB limits
Detection	Why did we not notice sooner?	Alert on 5xx rate only fires after 10 min sustained
Response	What slowed mitigation?	Rollback runbook referenced a removed script
Blast radius	Why was impact as wide as it was?	All regions received the config simultaneously

For each factor, ask “why was this possible?” until you reach something a team can change (a check, a default, a process, a tool), then stop. Stopping at “human error” is a sign the analysis is incomplete: ask what made the error easy and its consequence large.

4. Action items

Every action traces to a factor. Every factor has at least one action, or an explicit note that the team accepts the risk.

Field	Rule
Action	Verb-first, specific, verifiable: “Add CI check rejecting pool_size > 0.8 x max_connections”
Type	prevent / detect / mitigate / process
Owner	A named person (not a team); team in brackets is fine
Due	A date. Large items get a date for the design or first milestone
Ticket	Link or ID; create the ticket only after the user confirms
Priority	P1 (before next similar change), P2 (this quarter), P3 (backlog, revisit)

Reject vague actions and rewrite them:

Vague	Specific
Improve monitoring	Alert when DB connection usage > 85% for 2 min, routed to the DB on-call
Be more careful with configs	Require a second approver for prod config changes touching connection settings
Update the runbook	Replace step 4 of the rollback runbook with deployctl rollback --to-previous; test in staging
Add more tests	Add an integration test that boots the service with max pool size against a DB capped at 100 connections

Keep the list short. Five completed actions beat fifteen that rot. Push “nice to have” ideas into a separate “considered, not doing” list with the reason.

5. Blameless language pass

Search the draft for these and rewrite each hit:

rg -n -i "should have|failed to|forgot|careless|mistake|human error|didn't notice|obviously|simply|just" postmortem.md

Instead of	Write
“Alex failed to check the dashboard”	“The dashboard was not part of the deploy checklist”
“Human error caused the outage”	“The CLI accepted a production target without confirmation”
“Should have rolled back sooner”	“Rollback started 34 min after detection; the runbook pointed to a removed script”

Names may appear in the timeline as roles (“on-call engineer”, “IC”) unless the team’s culture uses names; follow the existing documents. Never name a person in the summary, factors or actions-as-blame.

Template

# Postmortem: <short user-facing description> (<YYYY-MM-DD>)

**Status:** draft | in review | final   **Severity:** SEV-2
**Authors:** <names>   **Incident lead:** <role or name>   **Review meeting:** <date>

## Summary
<3-5 sentences: what users experienced, for how long, the trigger, how it was fixed,
and the single most important follow-up.>

## Impact
- Duration: 09:31-10:47 UTC (76 min); time to detect 11 min; time to mitigate 52 min
- Users affected: ~18% of checkout attempts in EU and US (41,200 failed requests)
- Data: no loss or exposure (verified by <check>)
- SLO: 63% of the monthly checkout availability budget consumed

## Timeline (UTC)
| Time | Event | Source |
|---|---|---|

## Contributing factors
### Trigger
### Latent conditions
### Detection
### Response

## What went well
## Where we got lucky
## Action items
| # | Action | Type | Owner | Due | Ticket | Factor |
|---|---|---|---|---|---|---|

## Considered, not doing
## Open questions
## Appendix: graphs, queries, links

“Where we got lucky” is mandatory: it records near-misses (the incident happened at low traffic, an engineer happened to be online) that would make the next one worse.

Gotchas

• Writing the summary first anchors the narrative before the timeline is checked. Write it after the timeline and factors are stable.
• Chat exports mix time zones and edited messages. Prefer system timestamps (alerts, deploy logs) when they conflict with chat, and note the conflict.
• “Root cause: bad deploy” explains nothing. Name what about the deploy and what allowed it.
• Do not speculate to fill gaps. Put unknowns in Open questions with the person who can answer them.
• Customer-facing text must not include internal hostnames, employee names, vendor blame, or unreleased remediation dates. Draft it separately and have the user approve it.
• Publishing the postmortem, posting it to a channel, or creating tickets is outward-facing: show the user the final text and get a clear yes first.

Verification

• Every timeline row has a source; every number in Impact traces to a query or graph.
• The four anchor times are present and the derived durations add up.
• Each contributing factor maps to at least one action or an explicit accepted-risk note.
• The blame search above returns no unaddressed hits.
• A reader outside the team can answer: what broke for users, why, and what changes now. Ask the user to have one such reader skim it if the incident is SEV-1.