On-Call Practices skill for Codex

Done well means: a page always reaches a prepared human who can act within minutes, nobody carries an unsustainable load, every shift ends with a handoff that transfers context, and on-call pain feeds back into engineering priorities.

Inspect first

1. Read AGENTS.md, existing runbooks and the paging tool configuration (Terraform for pagerduty_schedule / pagerduty_escalation_policy, incident.io or Grafana IRM config, Alertmanager receivers).
2. Collect facts: team size and time zones, services covered, pages per week for the last quarter, after-hours share, mean time to acknowledge, and who actually got paged.
3. Ask about constraints: labour rules and compensation policy for the region, existing support contracts (customer-facing response times), and whether a NOC or support tier triages before engineering.

Rotation and compensation changes affect people’s lives and contracts; Codex proposes, humans decide. Never apply schedule changes to the paging tool without explicit confirmation of the exact change.

Rotation shape

Team situation	Rotation	Notes
4-5 engineers, one time zone	Weekly primary, previous week’s primary as secondary	Minimum viable; watch load closely
6-8 engineers, one time zone	Weekly primary plus separate secondary	Each person on primary about once every 6-8 weeks
Two regions 6+ hours apart	Follow-the-sun, 12 h shifts each	Removes most night pages; needs strong handoffs
Fewer than 4 people	Business hours only plus best-effort, or share a rotation with another team	A 1-in-3 rotation burns people out
Many small services	Team-of-teams rotation with clear service ownership routing	Avoid a single “platform” person paged for everything

Defaults: hand off mid-week (Tuesday or Wednesday morning) rather than Monday, so the weekend is never the first or last day of a shift, and so handoff happens with everyone at work. Nobody should be on call for more than one week in four for sustained periods.

Escalation policy

Level 1: primary on-call           notify immediately (push + SMS + phone)
Level 2: secondary on-call         if not acknowledged in 5 minutes
Level 3: engineering manager       if not acknowledged in 15 minutes
Repeat policy: 2 times, then notify incident commander channel

• Acknowledgement timeout 5 minutes for paging severity; shorter times create false escalations while someone is unlocking a laptop.
• Low-urgency alerts (tickets) never escalate after hours; they wait for business hours.
• Test the whole chain quarterly with a scheduled test page.

PagerDuty Terraform sketch:

resource "pagerduty_escalation_policy" "checkout" {
  name      = "Checkout"
  num_loops = 2
  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_primary.id
    }
  }
  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.checkout_secondary.id
    }
  }
}

Check vendor status before investing in a tool: Atlassian is retiring Opsgenie in favour of Jira Service Management and Compass, and Grafana OnCall OSS moved to Grafana IRM.

Before a shift

• Paging app installed, notification rules tested (phone call, not just push)
• Laptop, VPN, production access and break-glass credentials work
• Read the last handoff note and open incidents
• Know the runbook index and how to declare an incident
• Know who is secondary and who is the escalation manager
• Personal plans compatible with 5-minute acknowledgement (or arrange a swap)

Handoff template

## On-call handoff: <team> <date> -> <date>
**Outgoing:** <name>  **Incoming:** <name>

### Open issues
- <alert/incident>: status, next step, ticket link

### Pages this shift
| When | Alert | Real? | Action | Follow-up ticket |
|---|---|---|---|---|

### Changes to watch
- Deploys, migrations, feature flags rolled out, upstream maintenance windows

### Noise to fix
- <alert name>: fired N times, no action needed; ticket <link>

### Notes for incoming
- <anything unusual: customer escalations, risky deploys scheduled>

Hold a 15-minute live handoff when there are open issues; the written note is the record, not a replacement for the conversation.

Measuring load

Track monthly and review in the team’s operations meeting:

Metric	Healthy	Act when
Pages per shift (week)	< 2 incidents per 12 h on duty on average	Higher for two consecutive months
After-hours pages	Trending down	Any person woken more than twice a week
Actionable ratio	> 80% of pages needed a human	Below 50%: alert cleanup sprint ($alerting-rules)
Time to acknowledge	p90 under 5 min	Repeated escalations to secondary
Interrupt work hours	Visible in planning	On-call cannot do any planned work

Rule: the primary on-call does no sprint feature work; their job is pages, tickets from alerts, toil reduction and runbook fixes. Plan capacity accordingly.

Onboarding new people

1. Read runbooks and recent postmortems for the services covered.
2. Shadow one full shift (receives pages, does not own them).
3. Reverse shadow one shift (owns pages, experienced engineer as backup).
4. Join the rotation. Pair new members with an experienced secondary for the first shift.

Wellbeing and fairness

• Compensate on-call (time off in lieu or pay) per company policy and local law; flag to the user that HR or legal must confirm the policy, Codex does not decide it.
• After a night incident, the person paged starts late or takes time off the next day; the secondary covers.
• Swaps are easy and recorded in the tool, not agreed only in chat.
• Managers are in the escalation chain and on call for coordination, not as heroes.

Verification

• A test page sent through each schedule reaches the right person by phone within the timeout, and escalation reaches the secondary when not acknowledged.
• Schedules have no gaps: check the paging tool’s coverage view for the next 8 weeks.
• After one month, load metrics are recorded and reviewed in a meeting with actions.

Output

A proposal document: rotation shape and calendar, escalation policy, handoff template, load metrics and thresholds, onboarding steps, open questions for HR/management, and the Terraform or tool configuration diff for review.

Gotchas

• Push notifications alone fail silently at night (do not disturb); require phone or SMS for high urgency and let people test it.
• A secondary who is never paged is not practising; occasionally route lower-urgency work to them.
• Holidays and leave break schedules; set overrides weeks ahead.
• Rotations that include people without production access produce escalations, not fixes.