Caching Architecture skill for Codex

Done well means each cache exists because a measured read cost justifies it, every cached item has a named owner of invalidation and a maximum staleness the business has accepted, the system survives a cold or empty cache without falling over, and hit ratio, latency and staleness are observable.

Before adding or changing a cache

1. Read AGENTS.md and existing cache code: rg -n "redis|memcache|lru_cache|@Cacheable|cache\.(get|set)|Cache-Control" src. Note every layer already in play; a new layer on top of three existing ones makes staleness impossible to reason about.
2. Measure the read path: which queries or calls dominate latency or database load (APM, slow query log, pg_stat_statements). Cache the expensive, frequently read, rarely changed things; caching something cheap adds failure modes for no gain.
3. For each candidate, write down: read rate, write rate, size, how stale it may be, who writes it, and whether it is per-user or shared.

Where to cache

Layer	Good for	Invalidation	Watch out
Browser / HTTP cache	Static assets, public GET responses	Fingerprinted URLs; Cache-Control max-age	Cannot purge clients; use versioned URLs
CDN / edge	Public pages and API responses, images	TTL plus purge API or surrogate keys	Never cache responses that vary by user without the right Vary or cache key
In-process (LRU in memory)	Tiny, very hot, rarely changing data (config, feature flags, reference tables)	Short TTL or pub/sub broadcast	N copies across pods diverge; memory per pod
Distributed (Redis, Valkey, Memcached)	Shared computed results, sessions, rate counters, query results	Delete on write, TTL, versioned keys	Network hop; becomes a critical dependency
Database-side	Materialized views, summary tables, read replicas	Refresh schedule or triggers	Refresh cost; replica lag

Default for application data: cache-aside in a distributed cache with a TTL, deleting the key after a successful database write. Add an in-process layer only for a handful of very hot keys.

Patterns

Pattern	Read	Write	Use when
Cache-aside	App reads cache, on miss loads DB and fills	App writes DB, then deletes key	Default; app controls everything
Read-through	Cache library loads on miss	As cache-aside	Want loading logic centralised
Write-through	From cache	Write cache and DB synchronously	Reads right after writes must hit fresh data
Write-behind	From cache	Write cache, flush to DB asynchronously	Write-heavy counters where some loss is acceptable; rarely right for business data
Refresh-ahead	From cache	Background refresh before expiry	Hot keys with expensive loads and predictable access

Cache-aside done correctly

MISSING = "__none__"  # redis client created with decode_responses=True

def get_product(product_id: str) -> Product | None:
    key = f"product:v3:{product_id}"
    raw = redis.get(key)
    if raw == MISSING:
        return None
    if raw is not None:
        return Product.model_validate_json(raw)
    product = db.load_product(product_id)
    if product is None:
        redis.set(key, MISSING, ex=30)                # negative cache, short TTL
        return None
    ttl = 300 + random.randint(0, 60)                 # jitter spreads expiries
    redis.set(key, product.model_dump_json(), ex=ttl)
    return product

def update_product(product_id: str, changes: dict) -> None:
    db.update_product(product_id, changes)            # commit first
    redis.delete(f"product:v3:{product_id}")          # then invalidate (delete, not set)

Delete after commit, not before, and delete rather than set the new value: concurrent writers setting values race and can leave the older value cached. A remaining race (reader loads old value, writer commits and deletes, reader then sets the old value) is bounded by the TTL; if that window is unacceptable, use a short TTL, a delayed second delete, or version checks on fill.

Invalidation strategies

Strategy	How	Trade-off
TTL only	Expire after N seconds	Simple; staleness up to N
Delete on write	Writer deletes keys it affects	Writer must know every affected key
Event-driven	Consume change events (outbox, CDC) and delete keys	Decoupled; adds pipeline lag
Versioned keys	Bump a namespace version (catalog:v42:*) to invalidate a group	Old keys expire by TTL; needs memory headroom
Tag / surrogate keys (CDN)	Purge every response tagged product-123	CDN-specific support

Always keep a TTL as a backstop, even with explicit invalidation; a missed delete then heals itself.

Stampedes and hot keys

• Stampede (many concurrent misses for one key hitting the database): use single-flight (one loader per key per process, others wait), a short distributed lock around the load, or serve stale while one request refreshes.
• Mass expiry: add TTL jitter (plus or minus 10 to 20%) so keys written together do not expire together.
• Cold start after a cache flush or failover: warm critical keys, and protect the database with concurrency limits on loaders.
• Hot key on one Redis shard: add a small in-process cache in front for that key, or replicate the key under several suffixes and pick one at random on read.
• Negative caching: cache “not found” for a short TTL so lookups of missing ids do not all hit the database.

Consistency rules

• Decide per data type: “may be 5 minutes stale” (catalog), “must reflect the user’s own write immediately” (profile edits), “never cached” (balances, permissions checks, inventory at checkout).
• Read-your-own-writes: after a user’s write, bypass or refresh the cache for that user’s next read, or write-through for that entity.
• Never cache authorization decisions longer than you would tolerate a revoked user keeping access.
• Per-user data in shared layers needs the user or tenant in the key. Missing it leaks one customer’s data to another; review every key format for this.

Redis specifics

• Set maxmemory and an eviction policy. For a pure cache, allkeys-lru or allkeys-lfu; volatile-* policies evict only keys with a TTL and fail writes when none remain. Check INFO stats for evicted_keys and hit/miss counters.
• Avoid big values (over about 100 KB) and unbounded collections; they block the single-threaded command loop. redis-cli --bigkeys finds them.
• Never use KEYS * in production; use SCAN or versioned keys for group invalidation.
• Flushing a shared cache (FLUSHALL, FLUSHDB) is destructive and can cause a stampede: get explicit user approval and a warm-up plan first.

Gotchas

• Caching serialised ORM objects ties the cache to class layout; bump the key version (v3 above) on any schema change or deploys read incompatible data.
• A cache that the system cannot run without is primary storage; plan capacity and failover for it accordingly or make the fallback path survivable.
• Hit ratio alone misleads: 95% hits on cheap keys and 10% on the expensive one is a bad cache. Track hit ratio per key prefix.
• HTTP caches and CDNs ignore your application’s invalidation logic; set Cache-Control deliberately (private, no-store for personalised responses).

Verification

• Tests for the write path: after update, the next read returns the new value (delete happened after commit, key format matches the reader’s).
• Load test with the cache cold, then warm: database load on cold start stays within limits; p99 improves by the expected amount when warm.
• Concurrency test for stampede protection: 100 simultaneous misses produce one load.
• Dashboards: hit ratio per prefix, evictions, memory, cache latency, and a staleness probe for critical data.

Output template

## Caching design: <system>
| Data | Layer | Pattern | Key format | TTL | Invalidation | Max staleness | Per-user? |
|---|---|---|---|---|---|---|---|
| Product detail | Redis | cache-aside | product:v3:{id} | 300 s +/- 60 | delete after commit; CDC backstop | 5 min | no |

**Not cached (and why):** balances, permission checks
**Stampede protection:** single-flight per key, TTL jitter
**Cold-cache behaviour:** <DB load estimate, warm-up>
**Metrics and alerts:** <hit ratio per prefix, evictions, memory>