Incident Command skill for Codex
One commander, clear roles, updates on the clock. Codex helps run an outage the way experienced incident commanders do: declare early with a severity, separate the commander from the hands-on debuggers, mitigate before chasing root cause, and keep a timestamped decision timeline. It drafts internal, status page, executive and handover updates from bundled templates on a fixed cadence, and sends nothing until the commander approves the text.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- incident-commandInstalls to ~/.agents/skills/incident-command/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install incident-commandThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project incident-commandInstalls to .agents/skills/incident-command/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/incident-command.zipunzip -o incident-command.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) incident-commandPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $incident-command, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Runs an operational incident from declaration to stand-down - severity levels, the incident commander, operations, communications and scribe roles, a fixed update cadence, mitigation-first decision making, a live timeline, handovers between commanders, and the hand-off into a blameless review - with ready-to-use internal and customer update templates. Use when an outage or major degradation is happening now, when asked to act as or support the incident commander, to draft status updates, to set up severity definitions and an incident process, or to run a game day.
Use it when
- An outage or major degradation is in progress
- Someone needs to act as incident commander, scribe or comms lead
- Your team has no severity definitions or incident roles
- You are planning a game day to practise incident response
Use something else when
- The incident is a suspected security breach: use incident-response
- The incident is over and you are writing the review: use postmortem-writing
- You need hands-on diagnosis of the failing system: use production-debugging
How to check the result
The timeline records severity, roles, every production change with its approver and each update time, all updates met the promised cadence, and the postmortem has an owner and due date.
The mistake it prevents
The incident commander debugging alongside everyone else, so nobody owns decisions, communications slip and changes collide.
Done well means: an incident is declared early, one person is clearly in charge, the people fixing are shielded from the people asking, users and stakeholders get updates on a promised cadence, mitigation beats root-cause hunting, and a complete timeline exists when the incident closes.
Severity levels
Agree these before an incident; adapt the numbers to the business:
| Sev | Definition | Response | Update cadence |
|---|---|---|---|
| SEV1 | Critical journey down or data loss/exposure for many customers | Page IC and on-call immediately, all hands as needed, exec notified | Internal every 15-30 min, status page every 30 min |
| SEV2 | Major degradation, a key feature down, or a subset of customers fully blocked | Page on-call, IC assigned | Internal every 30 min, status page if customer-visible |
| SEV3 | Minor impact with a workaround, or risk of escalation | Owning team during working hours | Ticket updates |
Rules: declare at the higher severity when unsure and downgrade later; declaring costs
little, late declaration costs a lot. Security incidents (suspected breach, data
exposure) follow the security process in parallel; see $incident-response.
Roles
| Role | Does | Does not |
|---|---|---|
| Incident commander (IC) | Owns decisions, priorities, roles, cadence; asks “what is the current mitigation plan?” | Debug or type commands |
| Operations lead | Directs the technical investigation and mitigation, assigns workstreams | Talk to stakeholders |
| Communications lead | Drafts and sends internal updates and status page posts after IC approval | Make technical decisions |
| Scribe | Keeps the timeline: time (UTC), who, what, decision | Filter or editorialise |
| Subject matter experts | Investigate assigned questions, report back to ops lead | Make silent production changes |
In a small team one person may hold IC plus comms, but the IC should never also be the main hands-on debugger for SEV1. Codex can act as scribe and comms drafter; any production action or outward message still needs the human IC’s approval.
First 15 minutes
- Declare: open the incident channel (
#inc-<date>-<short-name>), set severity, name the IC - State known impact in one sentence: who, what, since when, how measured
- Assign ops lead, comms lead and scribe (or note who doubles)
- Ask: what changed recently (deploys, flags, config, provider status)?
- Choose the first mitigation to try, with a time box
- Post the first internal update; decide whether to post on the status page
- Set a timer for the next update
Running the incident
The IC loop, every 10-15 minutes:
- Status: what do we know, what changed since the last check?
- Plan: what is each workstream doing, and by when do they report back?
- Decision: continue, try the next mitigation, escalate, or bring in more people?
- Comms: is an update due? Approve the draft.
Decision principles:
- Mitigate first: rollback, failover, feature flag off, scale up, rate limit, block bad traffic. Understanding why can wait until users are fine.
- Prefer reversible actions. Before an irreversible one (data deletion, restoring a backup over live data, mass customer messaging), the IC states it, hears objections, and records the decision and who approved it.
- One change at a time per system, announced in the channel before it is made.
- Time-box hypotheses: “If this has not shown results by 14:40 we try the rollback.”
- Bring in help early; escalation is cheap compared to a long outage.
- Watch for fatigue: SEV1s longer than about 3-4 hours need planned handovers.
Communication
Internal updates go to the incident channel and a broader announcements channel; customer updates go to the status page and, for SEV1, possibly account managers. Every update states impact, what is being done, and when the next update comes. Templates for internal updates, status page phases, executive summaries and handovers are in references/comms-templates.md; read it when drafting any update.
Status page posts, emails to customers and messages to executives are outward-facing. Draft them, show them to the IC (the user), and send or publish only after explicit approval of the exact text. Never speculate about root cause or blame a vendor in customer-facing text.
Timeline
Scribe format, UTC, appended live:
14:02 Alert CheckoutHighErrorRate fired (5xx 12%)
14:05 @maria declared SEV2, IC @maria, ops @dev, comms @li, scribe @sam
14:09 Deploy 2026.03.14-2 at 13:55 identified as candidate
14:12 DECISION: roll back checkout to 2026.03.14-1 (approved by IC)
14:19 Rollback complete; 5xx falling
14:31 5xx at baseline for 10 min; status page moved to Monitoring
15:01 Resolved; postmortem owner @dev, due 2026-03-21Record decisions, hypotheses ruled out, commands run against production, and customer communications. These become the postmortem’s backbone.
Handover
When the IC changes (shift end, fatigue, escalation), do it explicitly in the channel: current impact, current hypothesis and mitigation, open workstreams with owners, next update time, pending decisions. The incoming IC states “I am IC now”. Use the handover template in the references file.
Stand-down
Resolve when the user impact has ended and stayed ended for a period you agreed on (often 30-60 minutes), not when the fix is merged. Then:
- Final internal update and status page resolution (approved text)
- Revert temporary mitigations or ticket them with owners
- Assign a postmortem owner and date (within 5 business days for SEV1/SEV2)
- Preserve evidence: dashboards links with fixed time ranges, logs, timeline
- Thank people; check whether anyone needs time off after a night incident
Write the review blameless and focused on systems; see $postmortem-writing.
Game days
Practise with a planned scenario in staging (or a controlled production fault with
approval): rotate the IC role, time the first 15-minute checklist, and fix what slowed
the team. See $chaos-engineering for fault injection.
Verification
An incident was run well if: the severity, roles and timeline are recorded; every update went out within the promised cadence; mitigation came before root-cause work; each production change has an approver in the timeline; and the postmortem has an owner and date.
Gotchas
- Twenty people in one channel all debugging: the IC must assign workstreams and move side discussions to threads or breakout channels.
- Status page silence is worse than “still investigating”; customers assume nobody knows.
- Executives joining the main channel pull responders into explanations; give them a separate stakeholder channel and regular summaries.
- Declaring “resolved” the moment graphs improve leads to reopened incidents; wait out the agreed quiet period.
What the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Incident Command"
short_description: "One commander, clear roles, updates on the clock"
default_prompt: "$incident-command We have a SEV1 checkout outage, act as scribe and draft the first status page update"references/comms-templates.md
# Incident communication templates
Use UTC times. Replace every `<...>`. Keep sentences short; say what users experience,
not internal service names. Outward-facing posts need the incident commander's approval
of the exact text before publishing.
## Internal update (incident channel and announcements channel)
```text
[SEV<1|2|3>] <short name> - update <n> at <HH:MM> UTC
Impact: <who is affected and how; numbers if known, e.g. "~15% of checkout attempts in EU fail">
Since: <HH:MM> UTC
Current status: <investigating | mitigation in progress | monitoring | resolved>
What we are doing: <current mitigation or investigation, owner>
What we know: <confirmed facts only>
What we ruled out: <optional>
Decisions needed: <none | decision and from whom>
Next update: <HH:MM> UTC
IC: <name> Ops: <name> Comms: <name>
```
## Status page phases
Investigating (first post, within 15 minutes of confirmed customer impact):
```text
We are investigating reports of <symptom users see, e.g. "errors when completing
checkout"> affecting <scope, e.g. "some customers in Europe">. We will provide an
update by <HH:MM> UTC.
```
Identified:
```text
We have identified the cause of <symptom> and are working on a fix. <Workaround if
any.> <Unaffected parts, e.g. "Existing orders and the API are not affected.">
Next update by <HH:MM> UTC.
```
Monitoring:
```text
A fix has been applied and <metric users notice, e.g. "checkout success rates"> have
returned to normal. We are monitoring to make sure the issue does not return.
<Any action customers need to take, e.g. "Payments that failed between 13:52 and 14:31
UTC were not charged and can be retried.">
```
Resolved:
```text
This incident has been resolved. Between <start> and <end> UTC, <plain description of
impact and scope>. <Customer action if any.> We will share a summary of what happened
and what we are changing <timeframe, e.g. "within five business days">.
```
Do not: name individual employees, blame a vendor before they confirm, speculate about
causes, promise a time to resolution you cannot control, or include internal hostnames.
## Executive summary (stakeholder channel or email)
```text
Subject: [SEV<n>] <short name> - <status> as of <HH:MM> UTC
Customer impact: <one sentence, with scope and duration so far>
Business impact: <orders/revenue/contractual SLA exposure if known; "unknown yet" is fine>
Current state: <mitigated | ongoing | resolved>
Next steps: <one or two lines>
Next update: <HH:MM> UTC or on status change
Contact: <IC name>; please route questions through <stakeholder channel>, not the
incident channel.
```
## Customer account manager brief (SEV1 with named customers)
```text
Customers affected: <list or segment>
What they experienced: <symptom, start and end times>
What to tell them now: <approved wording, consistent with the status page>
What not to say yet: <unconfirmed cause, credits, timelines>
Where updates appear: <status page URL>
Owner for follow-up: <name>
```
## IC handover
```text
Handover from <outgoing IC> to <incoming IC> at <HH:MM> UTC
Impact now: <...>
Mitigations applied: <list, with times>
Current hypothesis and plan: <...>
Workstreams: <owner - task - expected report time>
Pending decisions: <...>
Comms: last update at <HH:MM>, next due <HH:MM>; status page phase <...>
Temporary changes to revert later: <flags, scaling, blocks>
Incoming IC confirms: "I am IC now."
```
## Stand-down message
```text
[SEV<n>] <short name> - RESOLVED at <HH:MM> UTC
Duration of impact: <start> to <end> UTC
Summary: <two sentences>
Temporary changes still in place: <list with owners and tickets, or none>
Postmortem: owner <name>, draft due <date>, review meeting <date>
Thanks to: <people and teams>
```
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.