codexskills.dev

RAG Evaluation skill for Codex

Know whether retrieval or generation failed. Codex builds a frozen RAG test set with graded relevance labels, unanswerable and multi-hop questions, then scores retrieval and generation separately so each failure lands on a stage. A bundled script computes hit rate, recall, MRR and nDCG per k and per tag with a recall gate, and LLM judges are trusted only after agreeing with human labels.

Install

curl -fsSL codexskills.dev/install.sh \  | sh -s -- rag-evaluation

Installs to ~/.agents/skills/rag-evaluation/ for every project on this machine.

Download rag-evaluation.zip

104-line POSIX script. No sudo, no telemetry. Read install.sh

Use it in Codex

Start a prompt with $rag-evaluation, or pick it from /skills. You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.

What Codex reads before loading it

Measures RAG quality in two layers — retrieval (hit rate, recall@k, MRR, nDCG against labeled relevant chunks or documents) and generation (answer correctness, faithfulness to retrieved context, citation validity, abstention on unanswerable questions) — with test-set construction, LLM-judge calibration and per-segment reporting. Use when building a RAG test set, comparing chunking, embedding, hybrid or reranking changes, when a RAG assistant gives wrong or unsupported answers, or before shipping a retrieval change.

Use it when

  • Building a labeled question set for a RAG system
  • Comparing chunking, embedding, hybrid or reranker changes
  • A RAG assistant gives wrong or unsupported answers
  • Adding a retrieval quality gate to CI

Use something else when

  • Evaluating a non-retrieval LLM feature: use llm-evals
  • Building the retrieval pipeline itself: use rag-pipelines

How to check the result

The bundled script's recall gate passes on the baseline and fails on a deliberately broken retriever, and a gold-context run establishes the generation ceiling.

The mistake it prevents

Labeling relevance only from the current retriever's results, so the eval can never reveal documents the system fails to find.

~/.agents/skills/rag-evaluation/SKILL.md

--- name: rag-evaluation description: Measures RAG quality in two layers — retrieval (hit rate, recall@k, MRR, nDCG against labeled relevant chunks or documents) and generation (answer… license: MIT ---

Done well means: retrieval and generation are scored separately on the same labeled question set, so every failure is attributed to a stage; metrics are reported at the k actually sent to the model; unanswerable questions are part of the set; and any LLM judge has a measured agreement with human labels.

The two layers

LayerQuestionMetricsGrader
RetrievalDid the right passages reach the prompt?hit@k, recall@k, MRR, nDCG@kCode, against labeled ids
GenerationGiven those passages, is the answer right and supported?correctness, faithfulness, citation validity, abstention accuracyCode where possible, calibrated LLM judge otherwise
End to endDid the user get a correct, supported answer?correctness on full pipelineSame as generation, on live retrieval

Also run generation with gold context (the labeled relevant passages) once: if answers are wrong even with perfect context, the problem is the prompt or model, not retrieval.

Workflow

  • 1. Read AGENTS.md; find the retriever interface and any existing eval data
  • 2. Build the test set (questions, relevant ids, reference answers, tags)
  • 3. Log a run: retrieved ids per question, answer, citations
  • 4. Score retrieval with scripts/retrieval_metrics.py
  • 5. Score generation (code checks + calibrated judge)
  • 6. Report by segment, compare against baseline, decide

2. Test set

{"id": "q-041",
 "question": "Can I get a refund on an annual plan after 20 days?",
 "relevant": {"kb:refund-policy#4": 2, "kb:annual-plans#1": 1},
 "reference_answer": "Yes, annual plans are refundable within 30 days.",
 "answerable": true,
 "tags": ["billing", "policy", "en"]}
  • 50–200 questions. Sources: real user queries from logs (best), support tickets, questions written by domain experts while reading the docs, and a small share of LLM-generated questions reviewed by a human (generated ones tend to copy the document’s wording and overstate recall).
  • Label relevance with grades (2 = answers it, 1 = useful context) by searching the corpus, not only the current retriever’s results — otherwise you only measure what the system already finds.
  • Include 10–15% unanswerable questions ("relevant": {}, "answerable": false) and multi-hop questions needing two passages.
  • Label at document level as well as chunk level, or re-chunking invalidates labels; --doc-level in the script compares by the id part before #.
  • Freeze a version of the set; add, do not edit.

3–4. Retrieval scoring

Write one line per question to run.jsonl: {"id": "q-041", "retrieved": ["kb:refund-policy#4", ...], "relevant": {...}, "tags": [...]}

python3 scripts/retrieval_metrics.py run.jsonl --k 1,5,8,20 --by-tag --gate-recall 8:0.80
python3 scripts/retrieval_metrics.py run.jsonl --doc-level      # after re-chunking

The script (standard library only) prints hit rate, recall, precision and nDCG per k, MRR, per-tag tables, the ids of questions where nothing relevant was retrieved, and exits 1 when the recall gate fails. Questions without relevant ids are excluded from retrieval averages and listed.

Reading the numbers:

  • recall@k at the prompt’s k is the ceiling for answer correctness.
  • High recall@20, low recall@5 → ranking problem: add a reranker or hybrid search.
  • Low recall@20 → candidate problem: extraction, chunking, embeddings, filters.
  • MRR shows whether the best passage is first, which matters when context is truncated.

5. Generation scoring

Code checks first:

  • Citation validity: every cited id is in the retrieved set (target 100%).
  • Abstention: for answerable: false, the answer abstains (structured answerable=false field, not string matching).
  • Format: schema validation if output is structured.

Judge for the rest (structured verdicts, strong pinned judge model, see llm-evals):

FAITHFULNESS: Split the ANSWER into claims. For each claim, is it fully supported by
the CONTEXT? Output {"claims": [{"text", "supported": bool}], "verdict": pass|fail}.
pass only if every claim is supported.

CORRECTNESS: Does the ANSWER agree with the REFERENCE on the facts the question asks
about? Ignore style and extra correct detail. Output {"reasoning", "verdict"}.

Calibrate each judge on 50+ human-labeled answers; require ≥ 0.8 agreement before using it as a gate. Report faithfulness and correctness separately: a faithful answer from wrong passages is wrong, and a correct answer not supported by context is a hallucination that happened to be right.

Libraries such as Ragas or DeepEval implement similar metrics; if the repo uses one, keep it, but still check its judge prompts against human labels and pin the judge model.

6. Report and decide

Compare with the stored baseline on the same frozen set. Read per-tag tables — a reranker can raise average recall while hurting one language or product area. Repeat generation runs (3× for small sets) to see variance before claiming an improvement.

Gotchas

  • Labels built only from the current retriever’s top results bias the eval toward the current system.
  • Measuring recall@20 while sending 5 chunks to the model overstates quality.
  • Re-chunking changes chunk ids; without doc-level labels every metric drops to zero.
  • Judging faithfulness without giving the judge the exact context the generator saw.
  • Synthetic-only test sets overestimate quality; real queries are messier.
  • Skipping unanswerable questions rewards systems that always answer.

Verification

  • Script gate passes on the baseline run; a deliberately broken retriever (random ids) fails it.
  • Gold-context run establishes the generation ceiling.
  • Judge agreement with human labels reported per judge.

Output contract

**Test set:** rag-eval v3 — 140 questions (18 unanswerable, 22 multi-hop), 4 tags
**Change:** hybrid BM25+vector, rerank 40 → 8 (baseline: vector top-8)
| Metric | Baseline | Candidate |
|---|---|---|
| recall@8 | 0.74 | 0.86 |
| MRR | 0.58 | 0.71 |
| faithfulness | 0.89 | 0.93 |
| correctness | 0.77 | 0.84 |
| abstention accuracy | 0.67 | 0.83 |
| citation validity | 100% | 100% |
**Segments:** non-English recall@8 0.70 → 0.72 (flat; follow-up)
**Judge agreement:** faithfulness 0.88, correctness 0.84 (n=60)

What the zip contains

Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.

agents/openai.yamlgenerated from this page

Gives the Codex app a display name, a one-line description and a starter prompt for this skill.

interface:
  display_name: "RAG Evaluation"
  short_description: "Know whether retrieval or generation failed"
  default_prompt: "$rag-evaluation Build a test set and measure whether our RAG retriever finds the right docs"
scripts/retrieval_metrics.py143 lines, 5.1 KB 
#!/usr/bin/env python3
"""Retrieval metrics from a JSONL run file (standard library only).

Each line is one query:
  {"id": "q17", "retrieved": ["doc3#2", "doc9#1", ...],      # ranked, best first
   "relevant": ["doc3#2", "doc4#0"],                           # or {"doc3#2": 2, "doc4#0": 1}
   "tags": ["billing"]}                                        # optional

Usage:
  python3 retrieval_metrics.py RUN.jsonl [--k 1,5,10,20] [--by-tag]
                               [--gate-recall 10:0.80] [--doc-level]

Prints hit rate, recall, precision and nDCG at each k, plus MRR. --doc-level
compares document ids only (the part before '#'), useful when chunk ids changed
between index versions. --gate-recall K:MIN exits 1 if recall@K is below MIN.
Queries with no relevant ids are reported and excluded from averages.
"""
import argparse
import json
import math
import sys
from collections import defaultdict


def norm(i, doc_level):
    return i.split("#", 1)[0] if doc_level else i


def load(path, doc_level):
    rows, empty = [], []
    with open(path, encoding="utf-8") as fh:
        for n, line in enumerate(fh, 1):
            if not line.strip():
                continue
            try:
                obj = json.loads(line)
            except json.JSONDecodeError as e:
                sys.exit(f"{path}:{n}: invalid JSON ({e.msg})")
            rel = obj.get("relevant") or {}
            if isinstance(rel, list):
                rel = {r: 1 for r in rel}
            grades = {}
            for k, g in rel.items():
                key = norm(k, doc_level)
                grades[key] = max(grades.get(key, 0), g)
            seen, ranked = set(), []
            for r in obj.get("retrieved", []):
                key = norm(r, doc_level)
                if key not in seen:          # dedupe after doc-level collapse
                    seen.add(key)
                    ranked.append(key)
            row = {"id": obj.get("id", f"line{n}"), "ranked": ranked, "grades": grades,
                   "tags": obj.get("tags") or []}
            (rows if grades else empty).append(row)
    return rows, empty


def metrics(row, ks):
    ranked, grades = row["ranked"], row["grades"]
    rel = {k for k, g in grades.items() if g > 0}
    out = {}
    rr = 0.0
    for i, d in enumerate(ranked, 1):
        if d in rel:
            rr = 1.0 / i
            break
    out["mrr"] = rr
    ideal = sorted(grades.values(), reverse=True)
    for k in ks:
        top = ranked[:k]
        hits = sum(1 for d in top if d in rel)
        out[f"hit@{k}"] = 1.0 if hits else 0.0
        out[f"recall@{k}"] = hits / len(rel)
        out[f"precision@{k}"] = hits / k
        dcg = sum((2 ** grades.get(d, 0) - 1) / math.log2(i + 1) for i, d in enumerate(top, 1))
        idcg = sum((2 ** g - 1) / math.log2(i + 1) for i, g in enumerate(ideal[:k], 1))
        out[f"ndcg@{k}"] = dcg / idcg if idcg else 0.0
    return out


def average(rows, ks):
    acc = defaultdict(float)
    for r in rows:
        for m, v in metrics(r, ks).items():
            acc[m] += v
    return {m: v / len(rows) for m, v in acc.items()} if rows else {}


def table(title, avg, ks, n):
    print(f"\n{title} (n={n})")
    print(f"  {'k':>4} {'hit':>7} {'recall':>7} {'prec':>7} {'ndcg':>7}")
    for k in ks:
        print(f"  {k:>4} {avg[f'hit@{k}']:7.3f} {avg[f'recall@{k}']:7.3f} "
              f"{avg[f'precision@{k}']:7.3f} {avg[f'ndcg@{k}']:7.3f}")
    print(f"  MRR {avg['mrr']:.3f}")


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("run")
    ap.add_argument("--k", default="1,5,10,20")
    ap.add_argument("--by-tag", action="store_true")
    ap.add_argument("--doc-level", action="store_true")
    ap.add_argument("--gate-recall", help="K:MIN, e.g. 10:0.8")
    a = ap.parse_args()
    ks = sorted({int(x) for x in a.k.split(",") if x.strip()})
    if a.gate_recall:
        gk = int(a.gate_recall.split(":")[0])
        if gk not in ks:
            ks.append(gk)
            ks.sort()

    rows, empty = load(a.run, a.doc_level)
    if not rows:
        sys.exit("no queries with relevant ids")
    avg = average(rows, ks)
    table("all queries", avg, ks, len(rows))
    if empty:
        print(f"\n  note: {len(empty)} query(ies) without relevant ids excluded, e.g. "
              f"{[r['id'] for r in empty[:5]]}")
    if a.by_tag:
        groups = defaultdict(list)
        for r in rows:
            for t in r["tags"] or ["(untagged)"]:
                groups[t].append(r)
        for t in sorted(groups):
            table(f"tag {t}", average(groups[t], ks), ks, len(groups[t]))
    worst = sorted(rows, key=lambda r: metrics(r, ks)["mrr"])[:10]
    misses = [r["id"] for r in worst if metrics(r, ks)["mrr"] == 0]
    if misses:
        print(f"\nqueries with no relevant result retrieved: {misses}")

    if a.gate_recall:
        k, minimum = a.gate_recall.split(":")
        val = avg[f"recall@{int(k)}"]
        ok = val >= float(minimum)
        print(f"\nGATE recall@{k} = {val:.3f} (min {float(minimum):.3f}): {'PASS' if ok else 'FAIL'}")
        return 0 if ok else 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

About this skill

Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.

Related skills

Part of AI Engineering.

All 53 AI & LLM Engineering skills
esc
New

Introducing Codex Skills Pro

79 professional skills that take Codex through complete engagements, each with its own scripts, references and a report that verifies the result.

All-AccessOne-time payment, no subscription
$49
Get All-AccessOr browse individual skills from $2.99 →

Secure checkout by Creem