System Design Review skill for Codex

A useful review finds the two or three things that will actually hurt (the outage, the data loss, the bill) and backs each with a number or a failure sequence. It checks the design against stated requirements, not against an ideal architecture, and leaves the authors with fixes they can schedule.

Gather inputs first

Do not review before you know what “good” means for this system.

1. Read the design doc, diagrams and any linked ADRs. In a repo, read AGENTS.md, then skim service entry points, infra code (terraform/, k8s/, docker-compose.yml) and config to see what is really deployed, not just what the doc claims.
2. Extract or ask for the requirements. If missing, state your assumptions explicitly:

Requirement	Ask for	Example
Load	Peak requests/s, read:write ratio, growth over 12-24 months	1.2k rps peak, 20:1 reads, 3x in a year
Data	Volume now and in 2 years, record size, retention	400 GB, 2 KB rows, keep 7 years
Latency	p50/p99 target per critical path	Checkout p99 < 800 ms
Availability	SLO and what counts as down	99.9% monthly on checkout API
Durability / consistency	Which data must never be lost or double-applied	Payments exactly-once effect
Constraints	Team size, budget, compliance, existing platform	4 engineers, PCI scope, AWS only

3. Draw (or reconstruct) the request path for the top two user journeys, hop by hop. Most findings come from walking these paths with a failure in mind.

Review passes

Run each pass over the whole design. Write findings as you go.

#	Pass	Core questions
1	Requirements fit	Does the design meet each stated number? Is anything over-built for the load?
2	Capacity math	Rough throughput, storage and connection counts per component at peak and at 3x
3	Failure modes	For each dependency: what happens when it is slow, down, or returns garbage?
4	Data	Who owns each dataset? Consistency model? What happens on partial writes?
5	Coupling	Synchronous call chains, shared databases, deploy-together services
6	Security	Trust boundaries, authn/authz per entry point, secrets, PII flow, tenant isolation
7	Operability	Observability, deploy and rollback, migrations, on-call burden, runbooks
8	Cost	Dominant cost drivers and how they scale with load
9	Evolution	Migration path from today, reversibility, what the next 10x needs

For per-pass checklists with specific probes, read references/review-checklist.md when a pass needs depth (for example, a design with queues, multi-region, or regulated data).

Back-of-envelope checks

Do the arithmetic in the review; it is the fastest way to find real problems.

Peak writes: 1.2k rps x 5% writes = 60 writes/s -> one Postgres primary is fine (headroom > 10x)
Fan-out: each order emits 4 events x 3 consumers = 720 msg/s at peak -> trivial for SQS/Kafka
Sync chain: gateway -> orders -> inventory -> pricing -> tax, each p99 150 ms
   -> worst case approaches 600 ms before retries and the app's own work; budget is gone
Fan-out: one request calls 10 shards in parallel -> 1 - 0.99^10 ~ 10% of requests
   wait on at least one shard's p99; the page's p90 becomes the shards' p99
Availability: 4 serial dependencies at 99.9% each -> ~99.6% best case, below a 99.9% SLO
Connections: 40 pods x pool 20 = 800 connections vs SHOW max_connections = 1000
   -> fine today, breaks when autoscaling reaches 50 pods; add a pooler (PgBouncer/RDS Proxy)
Storage: 2 KB x 5M rows/day x 365 x 7 years ~ 25 TB -> plan partitioning and archival now

Rules of thumb worth checking against:

• A serial chain’s availability is the product of its parts. Every synchronous hop added to a critical path lowers the ceiling.
• Retries multiply load: 3 layers each retrying 3 times turn one failure into up to 27 calls.
• A queue absorbs bursts, not sustained overload; check consumer throughput exceeds the producer’s sustained rate with margin.
• Caches hide load until they empty. Ask what the origin sees on a cold start or a cache flush.

Failure walk

For each critical journey, fill a table like this. Missing cells are findings.

Dependency	Timeout	Retry policy	When it is down	When it is slow	Data risk
Payments API	3 s	2 retries, idempotency key	Order held as pending, retried by job	Threads pile up: needs bulkhead	None (idempotent)
Inventory svc	none set	client default	Checkout fails	Checkout hangs: finding	Oversell if reservation not atomic

Severity and findings

Severity	Meaning
critical	Data loss, security breach, or the design cannot meet a stated hard requirement
high	Likely outage or SLO miss under expected load or a common failure
medium	Operational pain, cost growth, or risk under plausible but less common conditions
low	Improvement, simplification, or clarity
question	Information missing to judge

Each finding: what, where, a failure scenario or number, the fix, and the cost of the fix.

**high - Inventory call has no timeout (checkout path, step 3).** When inventory slows
to 10 s (seen during its nightly reindex), checkout threads block and the 200-thread pool
exhausts in ~4 s at peak, taking down all checkout traffic. Fix: 800 ms timeout, circuit
breaker, and a separate pool for inventory calls. Effort: ~1 day.

Also call out over-engineering: a Kafka cluster for 50 msg/s, microservices for a four-person team, multi-region without an SLO that needs it. Unneeded complexity is a finding with an operational cost.

Workflow

- [ ] 1. Collect requirements; write down assumptions where missing
- [ ] 2. Reconstruct the request paths for the top journeys
- [ ] 3. Run the nine passes; do the capacity arithmetic in writing
- [ ] 4. Fill the failure walk table for each critical dependency
- [ ] 5. Rank findings; merge duplicates; drop anything without a scenario or number
- [ ] 6. Identify what you could not assess and what evidence would settle it
- [ ] 7. Deliver the report (template below)

Gotchas

• Reviewing the diagram instead of the system. Check deployed config: timeouts, pool sizes, replica counts and autoscaling limits often differ from the doc.
• “We will use Kafka so it scales” is not a design. Ask for partition key, ordering needs, consumer group sizing and replay behaviour.
• Exactly-once delivery claims: look for idempotent consumers and dedup keys, not broker settings alone ($event-driven-architecture, $outbox-pattern).
• Multi-tenant designs: the noisy neighbour and the cross-tenant data leak are the two findings to look for first.
• Cost is a design property. A per-request Lambda plus NAT gateway egress can dominate the bill at scale; estimate it.
• Do not demand a rewrite. Prefer the smallest change that removes the risk, and give a path for the larger change if one is truly needed.

Verification

• Every critical and high finding has a scenario or a calculation a reader can check.
• Every stated requirement maps to a pass/fail/unknown line in the report.
• Re-read the findings as the design’s author: is each fix actionable without a meeting?

Report template

# Design review: <system> (<doc link>, <date>)

**Verdict:** ready | ready with fixes | needs another iteration
**Top risks:** 1) <one line> 2) <one line> 3) <one line>

## Requirements check
| Requirement | Target | Design meets? | Evidence |
|---|---|---|---|

## Assumptions made
## Findings (ranked)
### critical / high / medium / low / questions
## Capacity and cost notes
## What I could not assess
## Suggested next steps (ordered, with effort)