codexskills.dev

Semantic Caching skill for Codex

Cut LLM spend without serving wrong answers. Codex builds a two-layer cache, exact match first and embedding similarity second, with the model, prompt version, tools and tenant baked into every key so old entries expire on change and nothing leaks across customers. The similarity threshold comes from labeled query pairs run through a bundled sweep script that reports precision and hit rate, and eligibility rules keep personal, time-relative and refused answers out.

Install

curl -fsSL codexskills.dev/install.sh \  | sh -s -- semantic-caching

Installs to ~/.agents/skills/semantic-caching/ for every project on this machine.

Download semantic-caching.zip

104-line POSIX script. No sudo, no telemetry. Read install.sh

Use it in Codex

Start a prompt with $semantic-caching, or pick it from /skills. You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.

What Codex reads before loading it

Designs and tunes semantic caches for LLM responses - exact-match layer first, embedding similarity lookup second, cache keys scoped by model, prompt version, tools, tenant and user, eligibility rules that keep personalized and time-sensitive answers out, and a similarity threshold chosen from labeled query pairs with a bundled sweep script. Covers Redis with RedisVL, pgvector and in-process stores. Use when LLM costs or latency are dominated by repeated questions, when adding GPTCache or a Redis semantic cache, or when a cache is serving wrong answers to similar-looking queries.

Use it when

  • Many LLM requests are repeats or close paraphrases of stable questions
  • Adding RedisVL, GPTCache or a pgvector-backed response cache
  • A semantic cache is returning answers for questions that only look similar

Use something else when

  • Requests share a long identical prefix and the provider discounts it: use prompt-caching-cost
  • You need a general application cache with TTLs and stampede protection: use redis-caching

How to check the result

Replay a held-out labeled query set through the cache and confirm the false-hit rate is at or below target, and that cross-tenant or changed-prompt lookups never hit.

The mistake it prevents

Copying a 0.9 similarity threshold from a tutorial instead of measuring it, which serves confidently wrong answers for queries that differ by one entity or a negation.

~/.agents/skills/semantic-caching/SKILL.md

--- name: semantic-caching description: Designs and tunes semantic caches for LLM responses - exact-match layer first, embedding similarity lookup second, cache keys scoped by model, prompt… license: MIT ---

A semantic cache is worth shipping only when its false-hit rate is measured and below an agreed target (typically 1-2% of hits), it never crosses tenant or user boundaries, and it is invalidated automatically when the prompt, model or source data changes. Hit rate is the second goal, not the first.

Decide whether to cache at all

SituationDecision
Same exact requests repeat (FAQ bots, templated calls, evals re-runs)Exact-match cache only; no embeddings needed
Paraphrased repeats of stable, non-personal questionsExact + semantic
Answers depend on user data, account state, time, or retrieved documents that changeDo not cache the final answer; cache retrieval or embeddings instead
Provider already discounts repeated prefixesUse provider prompt caching first (see prompt-caching-cost); it is free of false hits
Agents with tool calls and side effectsNever replay cached tool actions; cache only pure read tools’ results

Estimate the upside before building: sample a week of logs, embed the queries, and count how many have a near neighbour that a human would accept as the same question. If fewer than ~10% do, a semantic cache costs more in complexity than it saves.

Architecture

request
  -> normalize (trim, collapse whitespace, lowercase only if case never matters)
  -> eligibility check (personal? time-sensitive? tools with side effects? -> bypass)
  -> exact key lookup  hash(scope + normalized prompt)        -> hit: return
  -> embed prompt, vector search within the same scope, k=1   -> similarity >= T: return
  -> call model
  -> store (only if response is complete, not refused, not an error)

Scope is part of every key and every vector filter:

scope = {
    "model": "gpt-5-mini",                 # model id actually called
    "prompt_version": PROMPT_SHA[:12],     # hash of system prompt + few-shot examples
    "tools": TOOLS_SHA[:12],               # tool definitions change answers
    "tenant": tenant_id,                   # never share across tenants
    "locale": "en",
    "temperature_bucket": "det",           # cache only low-temperature calls
}

When any scope field changes, old entries simply stop matching. That is the main invalidation mechanism; TTL is the backstop.

Reference implementation (Redis + RedisVL)

RedisVL ships a SemanticCache (import path redisvl.extensions.cache.llm in current releases; older versions used redisvl.extensions.llmcache). Its distance_threshold is cosine distance, so similarity 0.94 means distance_threshold=0.06.

from redisvl.extensions.cache.llm import SemanticCache
from redisvl.query.filter import Tag

cache = SemanticCache(
    name="answers_v3",
    redis_url=REDIS_URL,
    distance_threshold=0.06,        # from the threshold sweep, not a guess
    ttl=7 * 24 * 3600,
    filterable_fields=[{"name": "tenant", "type": "tag"},
                       {"name": "prompt_version", "type": "tag"}],
)

flt = (Tag("tenant") == tenant_id) & (Tag("prompt_version") == PROMPT_SHA[:12])
if hits := cache.check(prompt=q, num_results=1, filter_expression=flt):
    return hits[0]["response"]
answer = call_model(q)
cache.store(prompt=q, response=answer,
            filters={"tenant": tenant_id, "prompt_version": PROMPT_SHA[:12]})

With Postgres, store (scope_hash, embedding vector, prompt, response, created_at) and query ORDER BY embedding <=> $1 LIMIT 1 filtered by scope_hash; see pgvector-search for index choice. In-process (dict + numpy) is fine for a single worker and under ~50k entries.

Use the same embedding model for store and lookup, and include its name in the scope. Re-embedding with a new model means a new cache, not a migration.

Choosing the threshold

Do not copy a threshold from a blog post; it depends on the embedding model and domain.

  1. From logs, take 300-1000 queries. For each, find its nearest neighbour among earlier queries with the production embedding model and record the cosine similarity.
  2. Label each pair: would the neighbour’s answer be correct for this query? Be strict: “reset password” vs “reset 2FA” is a 0 even at similarity 0.93.
  3. Run the sweep:
python3 scripts/threshold_sweep.py pairs.csv --min-precision 0.98 --step 0.01

It prints hit rate and precision per threshold and recommends the lowest threshold that meets the precision target. It exits non-zero if none does; then do not enable semantic hits, keep exact-match only.

  1. Re-run the sweep whenever the embedding model, prompt or domain changes.

Eligibility rules (bypass the cache)

  • The prompt contains user-specific data (names, account ids, “my order”) or the retrieved context differs per user.
  • The question is time-relative (“today”, “latest”, “current price”).
  • The response came from a temperature above ~0.3 or was explicitly regenerated by the user.
  • The response is a refusal, an error, truncated (max_output_tokens), or failed validation.
  • Numbers, negations or entity names differ between query and neighbour. A cheap guard: extract numbers and capitalized entities from both; if the sets differ, treat as a miss even above the threshold.

Operations

  • Emit metrics: exact hits, semantic hits, misses, bypasses, mean similarity of hits, lookup latency. Semantic lookup should add under ~20 ms p95; if embedding takes longer than the saved model call on misses, the cache is a net loss.
  • Sample 1% of semantic hits for human or LLM-judge review weekly; track false-hit rate.
  • Provide a purge path per tenant and per prompt version, and honour data deletion requests: cached answers can contain user content.
  • Mark cached responses in logs and traces (cache: semantic, similarity: 0.95) so debugging a wrong answer does not start at the model.

Workflow

  • Read AGENTS.md, find where LLM calls are made and what is already cached
  • Measure repeat potential from logs before building
  • Implement exact-match layer with full scope in the key
  • Add eligibility checks and the bypass path
  • Build labeled pairs and run scripts/threshold_sweep.py
  • Add semantic layer behind a flag with the chosen threshold
  • Add metrics, hit sampling and purge commands
  • Enable for a slice of traffic and compare quality and cost

Verification

  • Unit tests: a different tenant, prompt version or model never hits; time-relative and personal prompts bypass; refusals and truncated responses are not stored.
  • Replay a held-out labeled set through the cache: false-hit rate at or below the target.
  • A/B or shadow mode for a week: cost per conversation down, user feedback and eval scores unchanged.

Gotchas

  • Short queries (“pricing?”, “help”) sit close to many others in embedding space; require a minimum query length for semantic hits.
  • Negations embed almost identically: “can I cancel” vs “can I not cancel” score above 0.95 with many models.
  • Caching the final answer of a RAG system freezes stale documents; include the index version in the scope or cache only retrieval results.
  • A cache warm-up from synthetic questions creates entries no real user asked; it inflates hit metrics without saving money.

What the zip contains

Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.

agents/openai.yamlgenerated from this page

Gives the Codex app a display name, a one-line description and a starter prompt for this skill.

interface:
  display_name: "Semantic Caching"
  short_description: "Cut LLM spend without serving wrong answers"
  default_prompt: "$semantic-caching Add a semantic cache in front of our support bot's OpenAI calls using Redis"
scripts/threshold_sweep.py74 lines, 2.7 KB 
#!/usr/bin/env python3
"""Pick a semantic-cache similarity threshold from labeled query pairs.

Input CSV columns (header required):
  similarity  - cosine similarity between a new query and its nearest cached query
  acceptable  - 1 if serving the cached answer would be correct for the new query, else 0

Usage:
  python3 threshold_sweep.py pairs.csv [--min-precision 0.98] [--step 0.01]

Prints hit rate and precision (share of cache hits that were acceptable) per
threshold, then the lowest threshold whose precision meets the target.
Exits 1 if no threshold meets the target or the input is invalid.
"""
import argparse
import csv
import sys


def load(path):
    rows = []
    with open(path, newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        missing = {"similarity", "acceptable"} - set(reader.fieldnames or [])
        if missing:
            sys.exit(f"error: missing columns: {', '.join(sorted(missing))}")
        for i, r in enumerate(reader, start=2):
            try:
                rows.append((float(r["similarity"]), int(r["acceptable"])))
            except ValueError:
                sys.exit(f"error: line {i}: similarity must be a number, acceptable 0 or 1")
    if not rows:
        sys.exit("error: no rows")
    return rows


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("csv")
    ap.add_argument("--min-precision", type=float, default=0.98)
    ap.add_argument("--step", type=float, default=0.01)
    args = ap.parse_args()

    rows = load(args.csv)
    n = len(rows)
    positives = sum(a for _, a in rows)
    lo = min(s for s, _ in rows)
    print(f"pairs={n} acceptable={positives} ({positives / n:.1%})")
    print(f"{'threshold':>9} {'hits':>6} {'hit_rate':>8} {'precision':>9} {'false_hits':>10}")

    best = None
    t = 1.0
    while t >= lo - 1e-9:
        hits = [a for s, a in rows if s >= t]
        if hits:
            good = sum(hits)
            precision = good / len(hits)
            print(f"{t:9.3f} {len(hits):6d} {len(hits) / n:8.1%} {precision:9.3f} {len(hits) - good:10d}")
            if precision >= args.min_precision:
                best = (t, len(hits) / n, precision)
        t = round(t - args.step, 6)

    if best is None:
        print(f"\nno threshold reaches precision {args.min_precision}; do not enable semantic hits")
        return 1
    # Lowest threshold scanned that still meets the target maximises hit rate.
    print(f"\nrecommended threshold {best[0]:.3f}: hit_rate {best[1]:.1%}, precision {best[2]:.3f}")
    if n < 200:
        print("warning: fewer than 200 labeled pairs; the estimate is noisy")
    return 0


if __name__ == "__main__":
    sys.exit(main())

About this skill

Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.

Related skills

All 53 AI & LLM Engineering skills
esc