Uptime Monitoring skill for Codex

Done well means: you learn about an outage from your monitors before customers tell you, a single flaky probe location never pages anyone, certificates and domains never expire by surprise, and customers see an honest status page that is updated on a predictable cadence.

Inspect first

1. Read AGENTS.md; list the public entry points (marketing site, app, API, auth, webhooks receiver) and their owners.
2. Check existing tooling: blackbox exporter modules, Uptime Kuma, Better Stack, Checkly, Datadog Synthetics, Pingdom, Statuspage, Instatus. Avoid adding a second overlapping tool.
3. Ask what the contract promises (SLA percentage, response-time commitments) because the external check is usually the evidence for it.

What to check

Target	Check	Interval	Assertion
Main site	HTTPS GET /	1 min	200, body contains a stable string, total time < 3 s
API	HTTPS GET a cheap authenticated or public endpoint	1 min	200 and expected JSON field; not /healthz alone
Login / critical journey	Scripted browser or multi-step API check	5-10 min	Completes; see $synthetic-monitoring
TLS	Certificate expiry and chain	1 h	> 21 days to expiry, warn at 21, page at 7
DNS	Resolve apex and key records against public resolvers	5 min	Expected records
Domain registration	WHOIS/RDAP expiry	daily	> 30 days
Cron / batch jobs	Heartbeat (job pings a URL on success)	job period + grace	Ping received
Inbound webhooks / email	Send a canary and check receipt	15 min	Processed within N min

Check what users hit (through CDN and load balancer), not the origin directly; add an origin check separately when you need to tell CDN problems from app problems.

Probe locations and alert quorum

• Run from at least three regions, ideally on a different provider than production.
• Alert only when at least two locations fail for two consecutive runs. One location failing is usually that probe’s network.
• Keep a separate low-urgency alert when a single region fails consistently for 15 minutes; it may be a real regional routing problem.

Prometheus blackbox exporter

# blackbox.yml
modules:
  http_2xx_body:
    prober: http
    timeout: 10s
    http:
      method: GET
      valid_status_codes: [200]
      fail_if_body_not_matches_regexp: ["<title>Acme</title>"]
      preferred_ip_protocol: ip4
  tcp_tls:
    prober: tcp
    tcp: { tls: true }

# prometheus scrape config
- job_name: blackbox-http
  metrics_path: /probe
  params: { module: [http_2xx_body] }
  static_configs:
    - targets: [https://www.example.com/, https://api.example.com/v1/ping]
      labels: { probe_region: eu-west }
  relabel_configs:
    - { source_labels: [__address__], target_label: __param_target }
    - { source_labels: [__param_target], target_label: instance }
    - { target_label: __address__, replacement: blackbox-exporter:9115 }

Alerts:

# down from 2+ regions for 2 minutes
count by (instance) (probe_success{job="blackbox-http"} == 0) >= 2

# certificate expires within 14 days
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14

# slow from outside
avg by (instance) (probe_duration_seconds{job="blackbox-http"}) > 3

Use for: 2m on the down alert. Run one blackbox exporter per region, each scraped by (or remote-writing to) the central Prometheus with a probe_region label.

Tool choice

Situation	Tool
Prometheus already in place, engineering audience	Blackbox exporter in 3 regions
Small team, want UI and simple status page, self-hosted	Uptime Kuma (single container; back up its data volume)
Want zero infrastructure	A hosted uptime service with multi-region probes and on-call integration
Git-based, free, low frequency	Upptime (GitHub Actions, 5-minute minimum, not for paging)

Heartbeats for scheduled jobs

Absence is the signal: the job calls a unique URL only after success, and the monitor alerts when no ping arrives within period plus grace.

#!/bin/sh
set -eu
/app/bin/nightly-export
curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" > /dev/null

Keep the heartbeat URL in a secret; anyone with it can fake success.

Status page

Structure components the way customers think, not the way services are deployed: “Dashboard”, “API”, “Webhooks”, “Login”, “Billing”, with regions if relevant.

Incident update cadence: first post within 15 minutes of confirmed customer impact, then every 30 minutes (or the promised interval) until resolved, even when the update is “still investigating”. Templates:

Investigating - We are seeing elevated errors when loading dashboards for some
customers. We are investigating and will post an update by 14:30 UTC.

Identified - The errors are caused by a failed database failover in eu-west. We are
restoring the primary. API and webhooks are not affected. Next update by 15:00 UTC.

Monitoring - A fix has been applied and error rates are back to normal. We are
monitoring. Some dashboard exports requested between 13:52 and 14:41 UTC may need to
be retried.

Resolved - This incident is resolved. Between 13:52 and 14:41 UTC, about 18% of
dashboard loads in eu-west failed. A summary will follow within 5 business days.

Posting, editing or resolving a public status page incident and notifying subscribers are outward-facing: draft the text, show it to the user, and publish only after explicit confirmation. Never automate status-page posts directly from alerts without a human gate.

Verification

• Point a check at a URL that returns 500 (or block one probe region) and confirm the quorum behaviour: one region failing does not page, two do.
• Confirm the TLS expiry metric reports the real certificate’s date (compare with openssl s_client -connect www.example.com:443 -servername www.example.com </dev/null 2>/dev/null | openssl x509 -noout -enddate).
• Skip one cron run in staging and confirm the heartbeat alert fires after the grace.
• Calculate monthly uptime from the checks and compare it with the SLA wording (planned maintenance excluded or not).

Output

A table of checks (target, type, interval, locations, assertion, alert route), the status page component list, and the incident templates adopted.

Gotchas

• Checking /healthz only proves the process answers; it misses broken routing, auth and CDN errors. Check a real user-facing path.
• Monitors that follow redirects can report 200 from a login or error page; assert body content or final URL.
• Probes from inside the same cloud region share failure modes with production.
• Status pages hosted on the same infrastructure as the product go down with it.