Health Checks skill for Codex

Done well means: a process is restarted only when restarting fixes it, traffic is routed only to instances that can serve it, slow starters are not killed while booting, and a dependency outage degrades the service instead of making the orchestrator kill every replica at once.

Inspect first

1. Read AGENTS.md; find existing endpoints (rg -n "healthz|readyz|livez|/health|HEALTHCHECK|livenessProbe|readinessProbe|startupProbe").
2. Read the Kubernetes manifests or Helm values, ECS task definitions, load balancer target group settings and Dockerfile for current probe config.
3. Measure startup: how long from process start to first successful request (kubectl logs timestamps, or run locally and time it). Note the p99, not the best case.
4. List dependencies and decide which are hard (cannot serve any request without it) versus soft (feature degraded without it).

Semantics

Check	Question	Failure action	Should include dependencies?
Liveness /livez	Is this process wedged beyond self-recovery?	Restart container	No. Only in-process state: event loop responsive, no deadlock
Readiness /readyz	Should this instance receive new traffic now?	Remove from endpoints, no restart	Hard local prerequisites: warm caches loaded, connection pool established. Shared dependencies with care (see below)
Startup /startupz or reuse /livez	Has the app finished booting?	Restart only after the budget expires	No
Deep status /status (internal)	What is the state of each dependency?	None; for humans and dashboards	Yes, with timeouts, never wired to a probe

The cardinal rule: a liveness probe that checks the database turns a database blip into a restart storm across every replica, which makes recovery slower.

Readiness and shared dependencies: if the database is down for everyone, marking all replicas unready removes the whole service from the load balancer and users get connection errors instead of a clean 503 with a retry hint. Prefer: readiness fails only on instance-local problems; the app returns fast 503s for requests needing a down dependency, and alerts fire on the error rate.

Endpoint implementation

• Cheap: no allocation-heavy work, no external calls in liveness, response under 100 ms.
• Unauthenticated but not public: serve on an internal port or block at the ingress.
• Return 200 or 503 with a tiny JSON body; do not log every probe request.
• Readiness flips to 503 as soon as shutdown starts (SIGTERM), before the server stops.

Node (Express) sketch:

let ready = false;
let shuttingDown = false;

app.get('/livez', (_req, res) => res.status(200).json({ status: 'ok' }));
app.get('/readyz', (_req, res) =>
  ready && !shuttingDown ? res.status(200).json({ status: 'ready' }) : res.status(503).json({ status: 'not ready' }),
);

await warmUp();          // migrations checked, pools connected, caches primed
ready = true;

process.on('SIGTERM', () => {
  shuttingDown = true;   // endpoints controller removes the pod
  setTimeout(() => server.close(() => process.exit(0)), 10_000); // let in-flight and late requests finish
});

For gRPC, implement the standard grpc.health.v1.Health service and use the native grpc probe in Kubernetes rather than shipping grpc_health_probe in the image.

Kubernetes probes

startupProbe:
  httpGet: { path: /livez, port: http }
  periodSeconds: 5
  failureThreshold: 30          # 150 s budget; set from measured p99 startup x 1.5
livenessProbe:
  httpGet: { path: /livez, port: http }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3           # ~30 s of consecutive failure before restart
readinessProbe:
  httpGet: { path: /readyz, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
lifecycle:
  preStop:
    sleep: { seconds: 5 }       # Kubernetes 1.30+; older: exec ["sleep","5"] if the image has sleep
terminationGracePeriodSeconds: 30

• With a startup probe, liveness and readiness do not run until it succeeds, so you do not need a large initialDelaySeconds.
• timeoutSeconds defaults to 1 s; a GC pause or a busy event loop exceeds it. Use 2-5 s.
• terminationGracePeriodSeconds must exceed preStop sleep plus the app’s drain time.
• Exec probes fork a process every period; prefer HTTP or gRPC probes.
• Do not reuse the same aggressive settings for liveness and readiness; liveness should be the most tolerant of the three.

Other platforms

Platform	Setting	Advice
AWS ALB / NLB target group	health check path, interval, healthy/unhealthy thresholds	Point at /readyz; set deregistration delay to cover in-flight requests
ECS	container healthCheck (restarts task) plus target group check	Container check = liveness semantics only
Docker / Compose	HEALTHCHECK --interval=10s --timeout=3s --retries=3 CMD ["/app/healthcheck"]	Use a tiny binary or script in the image; curl may not exist in slim images
Cloud Run / App Runner	startup and liveness probes in service config	Same semantics as Kubernetes
Nomad / Consul	service checks	Readiness semantics; Consul removes unhealthy instances from DNS

Workflow

• Classify dependencies as hard-local, shared or soft
• Implement /livez (process only) and /readyz (local readiness + shutdown flag)
• Add optional /status for humans with per-dependency timeouts
• Measure startup p99; set the startup probe budget
• Set probe timeouts and thresholds; add preStop and grace period
• Drop probe requests from access logs and traces

Verification

• Kill the database in a staging environment: pods must stay running (no restarts in kubectl get pods -w), the service returns fast 503s, and it recovers by itself when the database returns.
• Freeze the process (kill -STOP <pid> inside the container in staging) and confirm liveness restarts it after roughly periodSeconds x failureThreshold.
• Run a rolling deploy under load ($load-testing or a simple hey/k6 loop) and confirm zero 5xx at the load balancer during the rollout.
• kubectl describe pod shows no probe failures during normal operation.

Output

Endpoint contract (path, port, what each checks, status codes), probe YAML with the measured startup time behind each number, and the results of the three staging tests.

Gotchas

• A health endpoint behind authentication middleware returns 401, which probes treat as failure.
• Probes hitting the same thread pool as user traffic fail under load, triggering restarts exactly when capacity is needed; give them a cheap path.
• Sidecars (service mesh proxies) need to be ready before the app accepts traffic and must outlive it on shutdown; use native sidecar containers where available.
• Returning 200 from /readyz before migrations or cache warm-up finish causes errors on every deploy.