Chaos Engineering skill for Codex

An experiment is worth running when it tests a specific hypothesis about steady-state behaviour, its blast radius is the smallest that can answer the question, a named person can abort it in seconds, and the result becomes a fix or a confirmed guarantee. Breaking things without a hypothesis is an outage, not an experiment.

Safety gate (do this before anything else)

• Identify the target environment. Local and ephemeral environments: proceed. Shared staging, pre-production or production: stop and get explicit written approval from the system owner in chat, including time window, scope and abort criteria.
• Confirm monitoring exists for the steady-state metrics; no dashboards means no experiment.
• Confirm an abort mechanism that works even if the tooling fails (delete the chaos resource, stop the FIS experiment, remove the tc rule, scale back up).
• Confirm on-call knows, and no incident or release is in progress.
• Never run experiments that can destroy data (deleting volumes, dropping tables, corrupting backups) outside disposable environments.

Codex must not apply faults to shared clusters, cloud accounts or production on its own initiative. Prepare the manifests, runbook and commands; let the approved operator execute or confirm each step.

Inspect first

1. Read AGENTS.md, architecture docs, and the dependency map: which services, databases, caches, queues and third-party APIs each request path touches.
2. Read the resilience code that the experiment will exercise: timeouts, retries with backoff, circuit breakers, bulkheads, fallbacks, health checks, PodDisruptionBudgets, autoscaling, multi-AZ settings.
3. Read SLOs and the dashboards for the relevant SLIs (success rate, latency p99, queue depth, error budget). See slo-error-budgets.
4. Review recent incidents; the best experiments re-create the conditions of past outages.

Write the hypothesis

### Experiment: Redis latency on cart service
Hypothesis: When Redis p99 latency rises to 500 ms for 5 minutes, cart API success rate stays
>= 99.5% and p99 latency stays < 800 ms, because the cache client times out at 100 ms and
falls back to Postgres.
Steady state (measured 30 min before): success 99.95%, p99 210 ms.
Fault: +500 ms latency, 100 ms jitter on traffic to redis-cart; 5 minutes.
Blast radius: staging namespace `cart`, 1 of 3 Redis replicas' clients via Toxiproxy.
Abort if: success < 98% for 60 s, or p99 > 2 s, or any alert pages on-call.
Abort action: `toxiproxy-cli toxic remove -n redis_latency redis-cart`.
Owner / operator / observer: <names>. Window: <date, time, timezone>.

Choose the fault and the tool

Question	Fault	Tool
Do timeouts and fallbacks work for one dependency?	Latency, connection refused, reset, slow close	Toxiproxy (local, CI, staging)
How does a host or container behave on a bad network?	Delay, loss, corruption, bandwidth limit	tc netem (Linux, needs root)
Does the service survive pod loss and rescheduling?	Pod kill, container kill	Chaos Mesh PodChaos, LitmusChaos
Kubernetes network partitions and DNS failures	Partition, delay, DNS errors	Chaos Mesh NetworkChaos, DNSChaos
Resource exhaustion	CPU, memory stress, disk fill	Chaos Mesh StressChaos, stress-ng
AWS instance, AZ or managed-service disruption	Stop instances, throttle APIs, AZ power interruption scenario	AWS Fault Injection Service with stop conditions on CloudWatch alarms
Multi-step failover (region, database primary)	Orchestrated sequence	Game day with a runbook, tool per step

Start with the smallest fault that tests the hypothesis: one dependency, one replica, a few minutes. Increase scope only after a clean result.

Local and CI experiments (no approval needed)

# Toxiproxy in front of Redis for an integration test or docker compose stack
toxiproxy-server &
toxiproxy-cli create -l 0.0.0.0:26379 -u redis:6379 redis-cart
toxiproxy-cli toxic add -t latency -a latency=500 -a jitter=100 -n redis_latency redis-cart
# point the app at :26379, run the scenario, then
toxiproxy-cli toxic remove -n redis_latency redis-cart

Automate the important ones as resilience tests in the integration suite (Toxiproxy has clients for Go, Java, Python, Node and a Testcontainers module), so the guarantee is checked on every change rather than once a year.

# Linux host or privileged container: 200 ms delay, 1% loss on eth0 (needs root)
tc qdisc add dev eth0 root netem delay 200ms 50ms loss 1%
tc qdisc del dev eth0 root      # abort / cleanup

Kubernetes (staging, with approval)

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: cart-to-redis-latency
  namespace: cart
spec:
  action: delay
  mode: all
  selector:
    namespaces: ["cart"]
    labelSelectors: { app: cart-api }
  direction: to
  target:
    mode: all
    selector:
      namespaces: ["cart"]
      labelSelectors: { app: redis-cart }
  delay:
    latency: "500ms"
    jitter: "100ms"
  duration: "5m"

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-one-cart-pod
  namespace: cart
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: ["cart"]
    labelSelectors: { app: cart-api }

Abort: kubectl delete networkchaos cart-to-redis-latency -n cart. Verify with kubectl get networkchaos -A that nothing is left running after the window.

Running the experiment or game day

1. Announce start in the agreed channel (the operator does this; Codex drafts the message).
2. Record steady state for at least 15-30 minutes beforehand.
3. Inject the fault; note the exact start time.
4. Observe the SLIs and the specific mechanism (did the circuit breaker open? did the fallback query run? did autoscaling trigger?). Collect traces and logs.
5. Abort immediately on any abort condition. Aborting is a valid result.
6. Remove the fault; confirm return to steady state and that no residual faults remain.
7. Debrief within a day while details are fresh.

Reading results

Outcome	Meaning	Next step
Hypothesis held, mechanism observed	Guarantee confirmed	Automate as a recurring test; widen scope next time
Hypothesis held, mechanism not observed	Passed by luck (fault did not apply, traffic too low)	Fix the experiment, rerun
Hypothesis failed	Found a weakness before customers did	File fixes with owners; rerun after the fix
Aborted	Blast radius larger than expected	Treat as an incident review; shrink scope

Output template

## Chaos experiment report: <name> (<date>)
Environment: <staging/prod> | Approved by: <owner> | Operator: <name>
Hypothesis: <...>
Result: held | failed | aborted at <time> because <condition>
Timeline: <start, fault injected, observations, removed, steady state restored>
Observations: <SLIs with numbers, mechanism evidence (traces, logs, dashboard links)>
Findings:
1. <weakness> - impact - fix - owner - due
Follow-ups: <automated resilience test added? next experiment?>

Gotchas

• Retries without backoff and jitter turn a latency fault into a retry storm; watch the dependency’s request rate, not only the caller’s errors.
• Health checks that call dependencies make every pod fail readiness together when the dependency slows. Liveness checks must not depend on downstream services.
• A fault that does not apply (wrong selector, wrong port) produces a false “held”. Always confirm the fault is active (latency visible in dependency metrics) before judging.
• Chaos tooling with cluster-wide permissions is itself a risk; scope its RBAC to target namespaces.