Prompt Caching and Cost skill for Codex

Done well means: cost per request is measured from usage, prompts are laid out so the cache hits, every call has explicit output and reasoning budgets, cheap models handle the requests they can (proven by evals), and offline work runs at batch prices.

Measure first

Log per request: feature, model, usage.input_tokens, usage.input_tokens_details.cached_tokens, usage.output_tokens, usage.output_tokens_details.reasoning_tokens, latency, prompt version.

def cost_usd(u, price):  # price per 1M tokens, from the pricing page, in config
    cached = u.input_tokens_details.cached_tokens
    return ((u.input_tokens - cached) * price.input
            + cached * price.cached_input
            + u.output_tokens * price.output) / 1_000_000

Reasoning tokens are billed as output tokens. Keep prices in config, not code — they change. Rank features by monthly cost before optimizing anything.

Workflow

• 1. Read AGENTS.md; find the LLM wrapper and where usage is (or is not) logged
• 2. Add usage logging and a cost dashboard query
• 3. Fix prompt layout for caching
• 4. Set budgets: max output, reasoning effort, history length
• 5. Route by difficulty; move offline jobs to Batch
• 6. Prove quality held with evals; report cost delta

3. Prompt layout for caching

OpenAI caches automatically for prompts above a minimum length (1,024 tokens on current models; varies on older ones) and only on an exact prefix match. Anything that differs early invalidates everything after it.

[1] instructions / developer prompt      static, versioned
[2] tool definitions                     static order, static text
[3] output schema (text.format)          static
[4] long reference material              static per tenant/document
[5] conversation history                 append-only
[6] current user message + per-request data (date, user name, retrieved chunks)

Rules:

• No timestamps, request ids, user names or shuffled retrieval results in [1]–[4].
• Keep tools identical across turns; narrow with allowed_tools / tool_choice instead of adding or removing tools. Changing model, tools, parallel_tool_calls, text.format or reasoning.effort breaks reuse from the changed point on.
• Append to history; do not rewrite or re-summarize earlier turns on every request.
• Set prompt_cache_key to a stable value for requests sharing a prefix (e.g. a tenant or document id). On high-traffic prefixes (above roughly 15 requests per minute per prefix-and-key), spread load across a few keys, since overflow lands on machines without the cache.
• Retention: newer models expose a cache TTL option and charge for cache writes; older models use prompt_cache_retention (in_memory or 24h). Check the prompt caching guide for the configured model before setting either; longer retention costs more to write and only pays off for prefixes reused across that window.

Check: cached_tokens / input_tokens per feature. Near zero on a long, repeated prompt means something in the prefix varies — diff two rendered prompts to find it.

4. Budgets

Lever	Setting	Notes
Output length	max_output_tokens + a word limit in the prompt	Treat status: "incomplete" as an error path, not success
Reasoning	reasoning={"effort": "low"} or lower where evals allow	Often the biggest cost on reasoning models
Verbosity	text={"verbosity": "low"} where supported	Shorter answers, same content
History	Keep last N turns + summary, or server-side compaction	Measure the quality impact on long-conversation cases
Retrieval	Fewer, better chunks (rerank to top 5–8)	Cheaper than a larger context
Tool outputs	Summarize or paginate large results	Raw HTML and full tables are expensive
Images	Downscale and use lower detail when fine print does not matter	Image tokens scale with resolution

5. Routing and escalation

def answer(req):
    tier = route(req)  # rules first: intent, length, customer plan; classifier later
    r = call(model=MODELS[tier], req=req)
    if tier == "small" and (r.parsed is None or r.parsed.confidence == "low"):
        r = call(model=MODELS["large"], req=req)   # escalate, log the escalation
    return r

• Start with rule-based routing on signals you already have; add a small classifier only if rules plateau.
• The escalation rate is a metric: if most requests escalate, the small tier costs more than it saves.
• Validate each tier on the eval set for the traffic slice it serves.
• Keep model names in config (MODELS = {"small": ..., "large": ...}) and check the models and pricing pages when choosing tiers.

Offline or non-interactive work (backfills, nightly classification, evals) → Batch API at half price (see openai-batch-api). Identical repeated questions → exact-match response cache keyed on normalized input + prompt version; for similarity-based reuse see semantic-caching.

Gotchas

• Optimizing cost without an eval gate: a 40% saving that drops answer quality 10% is usually a loss.
• Inserting the current date at the top of the system prompt disables caching for the whole prompt.
• Per-user personalization in the first message (name, plan) has the same effect — move it to the end.
• Retry storms double spend; cap retries and honor Retry-After.
• Streaming does not reduce cost; cancelling abandoned streams does.
• Cached tokens still count against the context window and still consume some rate limit; they are cheaper, not free.

Verification

1. Before/after table per feature: requests, cost per request, cached-token ratio, p50/p95 latency.
2. Render two consecutive prompts for the same feature and diff them; the first differing byte should be in section [5] or [6].
3. Eval pass rate unchanged (within noise) for every changed feature — see llm-evals.

Output contract

| Feature | Cost/req before | after | Cached ratio | p95 latency | Eval pass |
|---|---|---|---|---|---|
| support-reply | $0.0142 | $0.0061 | 3% → 71% | 3.4 s → 2.2 s | 91% → 91% |
| ticket-labels | $0.0021 | $0.0009 | n/a (Batch) | n/a | 96% → 95% |

**Changes:** moved date and user name to final message; froze tool order; effort low;
routed short intents to small tier (escalation rate 7%); labels moved to Batch.