Alerting Rules skill for Codex

Done well means: every page is urgent, actionable and about something users feel; every alert links to what to do next; related alerts arrive as one notification; and the rules are tested like code.

Inspect first

1. Read AGENTS.md; locate rule files (rg -l "^groups:" --glob '*.y*ml', PrometheusRule manifests), Alertmanager config, and any Grafana-managed alerts.
2. Pull alert history if available: which alerts fired in the last 30 days, how often, and whether anyone acted. Alertmanager has no history store; use the paging tool’s export or ALERTS / ALERTS_FOR_STATE series in Prometheus.
3. Check what SLOs exist. If they do, burn-rate alerts replace most threshold alerts; see $slo-error-budgets.

What deserves a page

Signal	Page?	Example
Users are failing now (errors, latency, unavailability)	Yes	Error ratio > 5% for 5 m on checkout
Error budget burning fast	Yes	14.4x burn over 1 h and 5 m
Certain failure soon, needs a human within hours	Ticket, page only if < a few hours	Disk full in 4 h by linear prediction
Cause-based condition with no user impact yet	Ticket or dashboard	One pod restarting, CPU 90%
Something a machine can fix	Neither; automate	Restart a stuck worker
Informational	Never	Deploy finished

Default to symptoms at the edge users touch (load balancer, API gateway, synthetic check). Cause alerts (CPU, memory, single replica) generate noise without telling you whether anyone is affected.

Rule anatomy

groups:
  - name: checkout-symptoms
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="checkout"}[5m]))
          > 0.05
          and
          sum(rate(http_requests_total{service="checkout"}[5m])) > 1
        for: 5m
        keep_firing_for: 10m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout 5xx ratio {{ $value | humanizePercentage }}"
          description: "More than 5% of checkout requests failed for 5 minutes."
          runbook_url: "https://runbooks.example.com/checkout/high-error-rate"
          dashboard: "https://grafana.example.com/d/checkout"

• for filters blips; 5 m is a common default for pages, longer for tickets. Too long and you page after customers have already complained.
• keep_firing_for (Prometheus 2.42+) stops flapping resolve/fire cycles.
• The minimum-traffic guard (> 1 req/s) stops 1-of-2-requests-failed pages at night.
• Labels route (severity, team); annotations explain. Never put $value in labels; it changes every evaluation and creates a new alert each time.
• Every paging alert has runbook_url. If you cannot write the runbook, it should not page.

Other patterns:

# target or exporter missing entirely (rate() on nothing returns nothing)
absent(up{job="checkout"} == 1)

# disk fills within 4 h (ticket unless very short)
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0

# latency symptom from histogram
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))) > 1.5

Alertmanager routing

route:
  receiver: default-ticket
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: oncall-pager
      continue: false
    - matchers: [severity="ticket"]
      receiver: team-tickets
inhibit_rules:
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity=~"page|ticket"]
    equal: [cluster]
receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-key
  - name: team-tickets
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-url
        channel: "#payments-alerts"
  - name: default-ticket

• group_by decides what arrives as one notification; group by service, not by pod.
• Inhibit symptom alerts when a known parent outage is firing (cluster down, region down).
• Keep secrets in *_file fields, not inline.
• Silences are for planned work and known incidents, always with an expiry and a comment; creating one affects the whole team’s paging, so confirm with the user first.

Noise review workflow

• Export 30 days of pages: alert name, count, time to ack, action taken
• For each alert ask: did a human need to act, now? Was the action in the runbook?
• Delete or demote alerts with no action taken in their last N firings
• Merge duplicates that fire together (same incident, different names)
• Raise for, add traffic guards or keep_firing_for on flappers
• Replace cause alerts with one symptom or burn-rate alert per user journey
• Re-measure pages per on-call shift after two weeks

Target fewer than two paging incidents per on-call shift on average; above that, people stop reading pages.

Tests

# tests/checkout_test.yml
rule_files: [../rules/checkout.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="checkout",code="500"}'
        values: '0+60x20'
      - series: 'http_requests_total{service="checkout",code="200"}'
        values: '0+600x20'
    alert_rule_test:
      - eval_time: 12m
        alertname: CheckoutHighErrorRate
        exp_alerts:
          - exp_labels: { severity: page, team: payments }
            exp_annotations:
              summary: "Checkout 5xx ratio 9.091%"
              description: "More than 5% of checkout requests failed for 5 minutes."
              runbook_url: "https://runbooks.example.com/checkout/high-error-rate"
              dashboard: "https://grafana.example.com/d/checkout"

promtool check rules rules/*.yml
promtool test rules tests/*.yml
amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=page service=checkout

Write a test for the firing case and for one case that must stay silent (low traffic, short blip). If the exact annotation text is awkward to assert, test labels only.

Verification

• All four commands above pass.
• Each new paging alert has a runbook URL that resolves and a dashboard link.
• Backtest the expression over the last 30 days in the Prometheus UI: count how many times it would have fired and compare with known incidents. Many fires with no incident means noise; missed incidents means the threshold or signal is wrong.

Output

For each alert: name, expression, for, severity, route, the user impact it detects, the runbook link, and the backtest result (would-have-fired count over 30 days).

Gotchas

• Alerts on ratios with tiny denominators flap; add a minimum traffic clause.
• absent() returns the labels in its matcher only; alerts based on it lose other labels.
• A rule that errors (bad PromQL at evaluation time) silently never fires; watch prometheus_rule_evaluation_failures_total.
• Alertmanager running as multiple replicas needs clustering; otherwise each replica sends its own copy.
• Paging on every replica’s CPU turns one bad deploy into fifty pages.