Runbooks skill for Codex

A runbook is good when a tired engineer who has never touched the service can follow it from alert to resolution without improvising, every command runs as written, and every risky step says so before it is taken.

A runbook is a procedure, not an explainer. Background goes in a linked architecture doc; the runbook carries only what the responder needs, in the order they need it.

Inspect first

Gather from the system, not from memory:

cat AGENTS.md 2>/dev/null | head -60
rg -n "runbook_url|runbook" --glob '*.{yml,yaml,json,tf,jsonnet}'   # alert rules and existing links
ls docs/runbooks runbooks ops 2>/dev/null
rg -n "alert:|expr:" --glob '*rules*.y*ml' | head -40                # Prometheus rule names and queries

Collect for each runbook: the alert name and query (what exactly fired), dashboards, log queries, the service owner and escalation path, deploy and rollback commands, and the most recent incident notes that touched this alert.

One runbook per…

Trigger	Runbook type	Title pattern
A specific alert	Alert response	HighCheckoutErrorRate (exact alert name)
A planned or recurring task	Procedure	“Rotate the Postgres TLS certificate”
A failure mode spanning alerts	Scenario	“Primary database unavailable”

Name alert runbooks exactly after the alert so search from the pager finds them, and put the link in the alert itself:

# Prometheus alerting rule
- alert: HighCheckoutErrorRate
  expr: sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{service="checkout"}[5m])) > 0.05
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "Checkout 5xx above 5% for 10 minutes"
    runbook_url: "https://runbooks.example.internal/checkout/high-error-rate"

Workflow

- [ ] 1. Identify the trigger (alert rule or task) and copy its exact definition
- [ ] 2. Collect dashboards, log queries, owners, deploy/rollback commands
- [ ] 3. Read the last 2-3 incidents for this alert; list what responders actually did
- [ ] 4. Draft using the template below; every step = action + expected result + decision
- [ ] 5. Mark each step SAFE (read-only) or RISKY (changes state); add rollback for each risky step
- [ ] 6. Dry-run every read-only command in staging or with --dry-run where it exists
- [ ] 7. Have someone who did not write it walk through it (tabletop or game day)
- [ ] 8. Link it from the alert (runbook_url) and set a review date

Codex writes and tests runbooks; it does not execute state-changing steps against production. Running any RISKY step (restart, failover, scale, delete, restore, traffic shift) on a real environment requires the user’s explicit confirmation for that specific step, even during a drill.

Step writing rules

• Action, expected result, decision. Each step says what to run, what healthy and unhealthy output look like, and where to go next for each.
• Copy-paste commands with variables declared once at the top (export NS=checkout-prod). No “run the usual command”.
• Absolute specifics: cluster, namespace, region, dashboard URL, log query text.
• Read-only before write. Triage steps first, mitigation second, fixes last.
• Time boxes. “If error rate has not dropped within 10 minutes of rollback, escalate.”
• Mark risk inline at the start of the step, not in a footnote.
• Mitigate before diagnosing. If a rollback or feature-flag flip stops the bleeding, it comes before root-cause investigation.
• Number the steps so responders can say “I’m on step 6” in the incident channel.

Template

# HighCheckoutErrorRate

**Severity:** page | **Owner:** payments-oncall | **Last tested:** 2026-09-14 (game day) | **Review by:** 2027-03-14

## What this alert means
Checkout returns 5xx for more than 5% of requests over 10 minutes. Customers cannot pay.
Alert definition: `monitoring/rules/checkout.yml`.

## Impact check (2 minutes)
1. SAFE. Open the [checkout dashboard](https://grafana.example.internal/d/checkout). Is the error rate still above 5%?
   - No, recovered: note it in the channel, go to **Resolve**.
   - Yes: continue.
2. SAFE. Was there a deploy in the last 60 minutes?
   ```bash
   export NS=checkout-prod
   kubectl -n "$NS" rollout history deploy/checkout | tail -3
   ```
   - Recent deploy: go to **Mitigate: roll back**.
   - No deploy: go to step 3.
3. SAFE. Check the payment provider status page and our provider latency panel.
   - Provider degraded: go to **Mitigate: provider outage**.

## Mitigate: roll back
4. RISKY (changes production). Announce in #inc-checkout, then:
   ```bash
   kubectl -n "$NS" rollout undo deploy/checkout
   kubectl -n "$NS" rollout status deploy/checkout --timeout=5m
   ```
   Expected: `deployment "checkout" successfully rolled out`.
   Rollback of this step: `kubectl -n "$NS" rollout undo deploy/checkout` again restores the newer revision.
5. Wait 10 minutes. Error rate below 1%? Go to **Resolve**. Otherwise **Escalate**.

## Escalate
- After 20 minutes without mitigation, page `payments-secondary`; for provider issues, open a priority ticket with the provider (account ID in the vault entry "psp-support").

## Resolve
- Error rate below 1% for 15 minutes; alert cleared.
- Post a summary in the channel; open a postmortem if customer impact exceeded 5 minutes.

Reviewing an existing runbook

Check	Failure looks like
Commands run	Renamed deployments, removed CLIs, old cluster names
Links resolve	Dashboards moved, wiki pages deleted
Alert still exists and matches	Threshold changed, alert renamed, runbook orphaned
Owner and escalation current	Team reorganised, rota renamed
Decision points complete	A step ends with “investigate further” and no branch
Last tested date	Never, or older than 6 months

# list every runbook link in alert rules, then confirm each one resolves
rg -o --no-filename 'runbook_url: *"?[^"]+' monitoring/ | sed 's/runbook_url: *"\{0,1\}//' | sort -u

Gotchas

• Runbooks that explain the architecture for three screens before the first command. Put “what this alert means” in two lines and link the explainer.
• Steps that require access the on-call engineer does not have. Note required roles at the top and test with an on-call account, not an admin one.
• Commands that depend on the author’s shell aliases, kube context or local scripts. Set the context explicitly (--context, -n, --region).
• Secrets pasted into runbooks. Reference the vault path or secret name instead.
• “Restart the service” as the only step. That hides the problem; include the diagnostics to capture first (logs, heap dump, kubectl describe) when they are cheap.
• Runbooks that never get updated after incidents. Make “update the runbook” a standard postmortem action item.

Verification

• Every SAFE command executed in staging (or the read-only equivalent) and its expected output pasted from that run.
• Every RISKY step has a rollback and a confirmation note; none were run against production without the user’s explicit approval.
• A second person followed it in a tabletop or game day without asking the author.
• The alert rule contains the runbook_url, and the URL resolves.

Report

**Runbooks written:** 3 (HighCheckoutErrorRate, CheckoutLatencyP99, rotate-checkout-tls)
**Linked from alerts:** yes, runbook_url added in monitoring/rules/checkout.yml
**Tested:** read-only steps run against staging; rollback steps not executed (production)
**Gaps found:** escalation contact for the payment provider unknown; ask payments lead
**Review date:** 2027-03-14