Resilience Patterns skill for Codex

Done well means every remote call has a deadline, retries are rare, bounded and only on idempotent work, a failing dependency is isolated so it cannot exhaust the caller’s threads or connections, and the team can show (with a fault injection test) what users see when each dependency is slow or down.

Before changing anything

1. Read AGENTS.md and find the HTTP/gRPC client setup: where clients are constructed, whether a shared wrapper exists, and which resilience library is already a dependency. Extend the existing one; two layers of retry are worse than none.
2. List every outbound call on the hot path: target, protocol, current timeout (often none, or the library default), retry behaviour, and whether the operation is idempotent. Search for constructors, not call sites: rg -n "new HttpClient|http.Client\{|httpx\.(Async)?Client|axios.create|fetch\(" src.
3. Get latency data for each dependency (p50, p99, p99.9) from metrics or traces. Timeouts chosen without it are guesses.
4. Draw the call tree for one user request and note who else retries: the SDK, the service mesh (Envoy/Istio retry policy), the load balancer, the job queue.

Pattern selection

Symptom or risk	Pattern	Default
Calls hang, threads pile up	Timeout / deadline	Set on every call, below the caller’s own deadline
Brief transient errors (connection reset, 503, throttling)	Retry with full-jitter backoff	2 retries max, idempotent operations only
Dependency down for minutes	Circuit breaker	Open at 50% failures over at least 20 calls
One slow dependency starves everything	Bulkhead	Separate pool or semaphore per dependency
Overload, queue growing without bound	Load shedding, bounded queues	Reject early with 429/503 and Retry-After
Non-critical data unavailable	Fallback	Cached or default value, clearly marked degraded
Tail latency on idempotent reads	Hedging	Second request after p95 delay; only with spare capacity

Order inside one call path, outermost first: overall deadline, bulkhead, circuit breaker, retry, per-attempt timeout. The breaker must see each attempt’s failure; the overall deadline must cap the total including backoff.

Timeouts

• Set connect and request timeouts separately. Connect: 100 ms to 1 s inside a region. Request: slightly above the dependency’s p99.9 for that endpoint, never the library default (many HTTP clients default to no timeout or 100 s).
• Propagate a deadline, not a fixed timeout. Each hop gets the remaining budget minus its own work. gRPC does this natively; for HTTP pass a header such as x-request-deadline and convert it to a context or cancellation token.
• A timeout that fires must cancel the work (close the socket, cancel the query), or the dependency keeps working on a request nobody is waiting for.

ctx, cancel := context.WithTimeout(r.Context(), 800*time.Millisecond)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
resp, err := client.Do(req) // client also has Transport timeouts for dial and TLS

const res = await fetch(url, {
  signal: AbortSignal.any([req.signal, AbortSignal.timeout(800)]), // Node 20.3+
});

Retries

Retry only when all hold: the failure is transient (connect error, 502/503/504, 429, gRPC UNAVAILABLE), the operation is idempotent or carries an idempotency key, and the remaining deadline allows another attempt. Never retry 400, 401, 403, 404, 409, 422.

Full-jitter exponential backoff: sleep = random(0, min(cap, base * 2**attempt)), with base 50 to 200 ms and cap 2 to 5 s. Honour Retry-After when present.

Retry at one layer only, usually the one closest to the failing call. Three layers of three attempts produce 27 requests per user action during an incident.

Add a retry budget so retries cannot exceed about 10% of normal traffic; when the budget is spent, fail fast. Envoy and Istio expose retry budgets; in application code, a token bucket shared per dependency does the job.

import httpx
from tenacity import (retry, retry_if_exception_type, stop_after_attempt,
                      stop_after_delay, wait_random_exponential)

TRANSIENT = (httpx.ConnectError, httpx.ReadTimeout, httpx.RemoteProtocolError)

@retry(
    retry=retry_if_exception_type(TRANSIENT),
    stop=stop_after_attempt(3) | stop_after_delay(3),
    wait=wait_random_exponential(multiplier=0.1, max=2),
    reraise=True,
)
def get_price(client: httpx.Client, sku: str) -> dict:
    r = client.get(f"/prices/{sku}", timeout=httpx.Timeout(1.0, connect=0.3))
    if r.status_code in (502, 503, 504):
        raise httpx.ReadTimeout("retryable status", request=r.request)
    r.raise_for_status()
    return r.json()

Circuit breakers

• Closed, open (fail fast), half-open (let a few probes through). Count timeouts and 5xx as failures; do not count 4xx, which are the caller’s fault.
• Require a minimum number of calls before tripping, or a single failure at 3 a.m. opens the breaker for a low-traffic endpoint.
• One breaker per dependency (or per host), not one global breaker.
• When open, return a fallback or a fast, explicit error. Emit a metric and log on every state transition.

CircuitBreakerConfig cfg = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slowCallRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(100)
    .minimumNumberOfCalls(20)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .permittedNumberOfCallsInHalfOpenState(5)
    .build();
CircuitBreaker inventory = CircuitBreakerRegistry.of(cfg).circuitBreaker("inventory");

.NET with Polly v8 via Microsoft.Extensions.Http.Resilience: start from the standard handler (rate limiter, total timeout, retry, breaker, attempt timeout) and tune it rather than hand-assembling a pipeline.

builder.Services.AddHttpClient<InventoryClient>(c => c.BaseAddress = new("https://inventory"))
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout = TimeSpan.FromMilliseconds(800);
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(3);
        o.Retry.MaxRetryAttempts = 2;
    });

Bulkheads and load shedding

• Give each dependency its own connection pool limit or semaphore sized from Little’s law: concurrent calls = request rate x latency. 200 rps at 50 ms needs about 10; cap at 2 to 3 times that.
• Bound every queue. An unbounded executor queue turns overload into memory exhaustion and minutes of latency instead of fast errors.
• Shed load at the edge when queue time exceeds the client’s deadline: work the client has already given up on is pure waste. Respond 503 or 429 with Retry-After.
• Prioritise: health checks and paid or interactive traffic before batch and retries.

Fallbacks

Acceptable: last known good value from cache, a static default, hiding a non-critical widget, queuing the write for later. Not acceptable: silently returning wrong data for money, permissions or inventory. Mark degraded responses (header or field) and count them.

Workflow checklist

- [ ] Inventory outbound calls, current timeouts, retry layers, idempotency
- [ ] Pull p99/p99.9 latency per dependency; set connect + request timeouts
- [ ] Add deadline propagation from inbound request to outbound calls
- [ ] Remove duplicate retry layers; keep one, add jitter and a budget
- [ ] Add a per-dependency breaker and bulkhead; emit state metrics
- [ ] Define the fallback (or explicit error) for each dependency
- [ ] Inject faults and verify behaviour; document the results

Verification

• Unit-test the policy with a fake transport: assert attempt count, that non-idempotent POSTs are not retried, that the breaker opens after the minimum calls, and that total time stays under the overall deadline.
• Fault injection in a test or staging environment: add latency and errors with Toxiproxy, the service mesh’s fault filter, or a stub server. Run each scenario: dependency slow (2x p99), dependency returning 503, dependency black-holing connections.
• Pass criteria: caller p99 stays within its SLO or fails fast; thread and connection pools do not saturate; request volume to the failing dependency does not multiply.
• Never inject faults into production without explicit user approval and an agreed blast radius and abort switch; that is chaos engineering, not this skill.

Gotchas

• Retrying a POST that timed out may duplicate the side effect; the server may have committed. Use idempotency keys or do not retry.
• Timeout shorter than the dependency’s normal p99 creates the outage it was meant to prevent: every slow-but-fine call becomes a retry.
• Client-side breaker per pod: with 50 pods, each needs its own minimum calls before opening, so the dependency still sees a burst. Accept it or use mesh-level outlier detection.
• Database connection pools are bulkheads too; a slow query can drain a shared pool. Set statement timeouts on the database side as well.
• Health checks that call dependencies make every pod unhealthy when one dependency fails, and the orchestrator restarts the whole fleet. Liveness checks only the process.

Output template

## Resilience review: <service>
| Dependency | Timeout (connect/request) | Retries | Breaker | Bulkhead | Fallback | Idempotent |
|---|---|---|---|---|---|---|
| inventory (gRPC) | 200 ms / 800 ms | 2, full jitter, budget 10% | 50%/20 calls/30 s | 32 permits | cached stock, marked stale | yes |

**Changes made:** <files and policies>
**Fault tests run:** <scenario -> observed caller behaviour>
**Open risks:** <dependencies without a safe fallback, retry layers outside this repo>