LLM Evals skill for Codex
Ship prompt changes on numbers, not vibes. Codex builds eval suites from production samples, past bugs and adversarial inputs, grades with code wherever possible, and only trusts an LLM judge after measuring its agreement with human labels. A bundled standard-library gate compares runs case by case, reports per-tag shifts and fails CI when critical cases regress or the pass rate drops.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- llm-evalsInstalls to ~/.agents/skills/llm-evals/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install llm-evalsThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project llm-evalsInstalls to .agents/skills/llm-evals/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/llm-evals.zipunzip -o llm-evals.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) llm-evalsPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $llm-evals, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Builds evaluation suites for LLM features — case sets drawn from real traffic and failure reports, code-based checks, LLM-as-judge graders with rubrics and calibration against human labels, pass-rate baselines, and CI regression gates that block prompt or model changes that make things worse. Use when shipping or changing a prompt, model, or retrieval step, when asked "is the new model better?", when a feature regressed after a change, or when setting up eval tooling for an LLM product.
Use it when
- Changing a prompt, model, or retrieval step behind an LLM feature
- Comparing two models or prompt versions
- A feature regressed and nobody noticed until users complained
- Building an LLM-as-judge grader
Use something else when
- Evaluating multi-step agent trajectories and tool use: use ai-agent-evaluation
- Measuring retrieval recall and answer faithfulness for RAG: use rag-evaluation
How to check the result
The gate passes when a run is compared with itself, fails when a key instruction is deliberately removed from the prompt, and the judge's agreement with human labels is reported.
The mistake it prevents
Trusting the overall average while one intent or language quietly regresses; always read per-tag pass rates.
Done well means: a versioned case set that reflects real usage and known failures, graders that are deterministic wherever possible and calibrated where they are not, a stored baseline, and a gate that fails CI on regressions — so prompt and model changes are decided by numbers, not by reading three outputs.
Workflow
- 1. Define what “good” means for this feature (dimensions + hard failures)
- 2. Assemble the case set (real, adversarial, regression cases)
- 3. Choose a grader per dimension
- 4. Calibrate any LLM judge against human labels
- 5. Run baseline, store results
- 6. Wire the regression gate into CI
- 7. Feed production failures back into the set
1. Define success
Write one line per dimension, each testable:
correctness : answer matches the policy doc for the user's plan
format : valid JSON matching TicketLabel schema
grounding : every claim traceable to a retrieved passage
safety : never reveals another customer's data (hard fail)
tone : no blame, no promises of refunds (hard fail)Hard-fail dimensions get a critical tag; one regression blocks the release.
2. Case set
| Source | Share | Notes |
|---|---|---|
| Sampled production inputs (PII scrubbed) | ~50% | Stratify by intent, language, length |
| Known failures and bug reports | ~25% | Every fixed bug becomes a case |
| Adversarial and edge cases | ~15% | Empty input, very long input, injection attempts, other languages |
| Synthetic fill for rare intents | ~10% | Review by hand before adding |
Size: 50–100 cases to start; 200–500 to detect a ~5-point change with confidence.
Store as JSONL in the repo (evals/<feature>/cases.jsonl) with id, input, expected
or reference, and tags. Never edit a case’s meaning after it has a baseline — add a
new id instead.
Keep a held-out slice you do not look at while iterating on the prompt, or you will overfit the prompt to the visible cases.
3. Graders — cheapest reliable first
| Dimension type | Grader |
|---|---|
| Exact label, enum, number | Code: exact match / tolerance |
| JSON shape | Code: schema validation |
| Contains required facts / forbidden phrases | Code: regex or keyword lists |
| Retrieval quality | Code: recall@k against gold ids (see rag-evaluation) |
| Open-ended quality, faithfulness, tone | LLM judge with a rubric |
| Pairwise “is B better than A” | LLM judge, both orders, ties allowed |
LLM judge rules:
- Use a rubric with discrete levels and observable criteria, not “rate 1–10”.
- Ask for a short rationale then a verdict field, via structured output
(
{"reasoning": str, "verdict": "pass" | "fail"}). - Give the judge the reference answer or source passages when they exist.
- For pairwise, run both orders and count only consistent wins (position bias is real).
- Use a strong model as judge, pinned to a snapshot; changing the judge resets baselines.
JUDGE = """You grade a support reply.
PASS only if ALL hold:
1. Every factual claim is supported by the SOURCES.
2. It answers the user's actual question.
3. It makes no refund or legal promises.
Otherwise FAIL. Quote the failing sentence in reasoning."""
def judge(case, output) -> dict:
r = client.responses.parse(
model=os.environ["JUDGE_MODEL"],
instructions=JUDGE,
input=f"QUESTION:\n{case['input']}\n\nSOURCES:\n{case['sources']}\n\nREPLY:\n{output}",
text_format=Verdict,
)
return r.output_parsed.model_dump()4. Calibrate the judge
Hand-label 50–100 outputs (two labelers on a subset). Run the judge on the same outputs and compute agreement (accuracy, and Cohen’s kappa if classes are imbalanced). Target ≥ 0.8 agreement with humans before trusting it in a gate. Read every disagreement; tighten the rubric where the judge is wrong, then re-measure on fresh labels.
5. Run and store results
Each run writes one JSONL line per case:
{"id": "refund-017", "pass": true, "score": 1.0, "tags": ["refund", "critical"],
"latency_ms": 1840, "input_tokens": 912, "output_tokens": 133}Record alongside: prompt version, model snapshot, judge snapshot, temperature/reasoning settings, git sha. Store the baseline file in the repo or an artifact bucket.
Run with nondeterminism in mind: for small sets, run each case 3× and score the majority, or report the variance; a 2-case swing on 60 cases is noise.
6. Regression gate
python3 scripts/eval_gate.py evals/support/results.jsonl \
--baseline evals/support/baseline.jsonl --min-pass 0.85 --max-drop 0.02The script (standard library only) prints per-tag pass rates, regressed and improved
case ids, and a sign test on the flips; it exits 1 if the pass rate is below
--min-pass, dropped more than --max-drop, any critical-tagged case regressed, or
baseline cases are missing. Evaluations call the model, so CI needs network and an API
key; run the full suite on PRs that touch prompts, models or retrieval, and a smoke
subset otherwise. Update the baseline only in the PR that intentionally changes
behavior, with the diff in the PR description.
7. Close the loop
Log production inputs, outputs and user feedback (thumbs, escalations, edits). Weekly,
pull failures into cases.jsonl with tags, label them, and re-run the baseline.
Gotchas
- Judging with the same model and prompt that generated the output inflates scores.
- Averages hide regressions: a +3% overall can hide a −20% on one intent. Read per-tag.
- Eval-set leakage: few-shot examples copied from the case set make the prompt look perfect. Keep examples and cases disjoint.
- Changing graders and prompts in the same PR makes deltas uninterpretable.
- Cost and latency are part of quality; gate on p95 latency and tokens per case too.
- Scrub PII before committing production samples; check data-use terms.
Verification
eval_gate.pyreturns 0 on the baseline compared with itself.- Deliberately break the prompt (remove a key instruction) and confirm the gate fails.
- Judge agreement with human labels is reported and ≥ the agreed threshold.
Output contract
**Feature:** support-reply v14 vs v13 (model snapshot unchanged)
**Cases:** 240 (120 prod, 60 regressions, 40 adversarial, 20 synthetic)
**Pass rate:** 87.5% → 90.4% (+2.9), sign test p=0.04; 0 critical regressions
**By tag:** refund 78% → 91%; billing 95% → 94%; non-English 70% → 72%
**Judge:** <snapshot>, agreement with humans 0.86 (n=80)
**Cost/latency:** tokens per case −8%, p95 2.1 s → 2.0 s
**Decision:** ship; baseline updated in this PRWhat the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "LLM Evals"
short_description: "Ship prompt changes on numbers, not vibes"
default_prompt: "$llm-evals Set up an eval suite for our support reply generator before we switch models"scripts/eval_gate.py
#!/usr/bin/env python3
"""Regression gate for LLM eval results (standard library only).
Each results file is JSONL, one line per case:
{"id": "case-017", "pass": true, "score": 0.9, "tags": ["refund", "critical"]}
"score" and "tags" are optional. Cases are paired by "id".
Usage:
python3 eval_gate.py CANDIDATE.jsonl [--baseline BASELINE.jsonl]
[--min-pass 0.85] [--max-drop 0.02] [--critical-tag critical]
Fails (exit 1) when:
- candidate pass rate < --min-pass
- candidate pass rate dropped more than --max-drop versus baseline
- any baseline-passing case carrying --critical-tag now fails
- ids are duplicated, or baseline cases are missing from the candidate run
Prints per-tag pass rates, flipped cases, and an exact two-sided sign test on the
flips so a 1-2 case wobble is not mistaken for a real change.
"""
import argparse
import json
import math
import sys
from collections import defaultdict
def load(path):
rows, errors = {}, []
with open(path, encoding="utf-8") as fh:
for n, line in enumerate(fh, 1):
if not line.strip():
continue
try:
obj = json.loads(line)
except json.JSONDecodeError as e:
errors.append(f"{path}:{n}: invalid JSON ({e.msg})")
continue
cid = obj.get("id")
if cid is None or not isinstance(obj.get("pass"), bool):
errors.append(f"{path}:{n}: needs string 'id' and boolean 'pass'")
continue
if cid in rows:
errors.append(f"{path}:{n}: duplicate id {cid!r}")
rows[cid] = obj
return rows, errors
def rate(rows):
return sum(r["pass"] for r in rows.values()) / len(rows) if rows else 0.0
def sign_test(a, b):
"""Two-sided exact binomial test on a improvements vs b regressions."""
n = a + b
if n == 0:
return 1.0
k = min(a, b)
p = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
return min(1.0, 2 * p)
def by_tag(rows):
acc = defaultdict(lambda: [0, 0])
for r in rows.values():
for t in r.get("tags", []) or ["(untagged)"]:
acc[t][0] += r["pass"]
acc[t][1] += 1
return acc
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("candidate")
ap.add_argument("--baseline")
ap.add_argument("--min-pass", type=float, default=0.0)
ap.add_argument("--max-drop", type=float, default=0.02)
ap.add_argument("--critical-tag", default="critical")
a = ap.parse_args()
cand, errs = load(a.candidate)
base, berrs = load(a.baseline) if a.baseline else ({}, [])
failures = errs + berrs
if not cand:
failures.append("candidate has no cases")
cr = rate(cand)
print(f"candidate: {sum(r['pass'] for r in cand.values())}/{len(cand)} passed ({cr:.1%})")
tags_c = by_tag(cand)
tags_b = by_tag(base)
print("\nper tag (candidate vs baseline):")
for t in sorted(tags_c):
p, n = tags_c[t]
line = f" {t:<24} {p:>4}/{n:<4} {p / n:6.1%}"
if t in tags_b:
bp, bn = tags_b[t]
line += f" was {bp / bn:6.1%}"
print(line)
if cr < a.min_pass:
failures.append(f"pass rate {cr:.1%} below --min-pass {a.min_pass:.1%}")
if base:
br = rate(base)
missing = sorted(set(base) - set(cand))
if missing:
failures.append(f"{len(missing)} baseline case(s) missing from candidate, e.g. {missing[:5]}")
shared = set(base) & set(cand)
regressed = sorted(i for i in shared if base[i]["pass"] and not cand[i]["pass"])
improved = sorted(i for i in shared if not base[i]["pass"] and cand[i]["pass"])
print(f"\nbaseline: {br:.1%} delta: {cr - br:+.1%}")
print(f"flips: {len(improved)} improved, {len(regressed)} regressed, "
f"sign-test p={sign_test(len(improved), len(regressed)):.3f}")
for i in regressed[:30]:
print(f" REGRESSED {i} tags={cand[i].get('tags', [])}")
for i in improved[:30]:
print(f" improved {i}")
if br - cr > a.max_drop:
failures.append(f"pass rate dropped {br - cr:.1%} (> --max-drop {a.max_drop:.1%})")
crit = [i for i in regressed if a.critical_tag in (cand[i].get("tags") or [])]
if crit:
failures.append(f"critical case(s) regressed: {crit}")
print()
for f in failures:
print(f"GATE FAIL: {f}")
print("GATE PASS" if not failures else f"GATE FAILED ({len(failures)} reason(s))")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.