LLM Classification skill for Codex
Labels you can trust, with confidence that means something. Codex starts with the taxonomy, not the prompt: written definitions, precedence rules and an other label, checked against human agreement. Outputs are constrained to the label enum, confidence comes from logprobs or self-consistency and is calibrated on a validation split, and a bundled script reports per-class F1, the confusion matrix and ECE so prompt changes can be gated in CI.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- llm-classificationInstalls to ~/.agents/skills/llm-classification/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install llm-classificationThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project llm-classificationInstalls to .agents/skills/llm-classification/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/llm-classification.zipunzip -o llm-classification.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) llm-classificationPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $llm-classification, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Builds text classifiers on LLMs - label taxonomy and decision rules, zero-shot and few-shot prompts with enum-constrained structured output, confidence from logprobs or self-consistency, calibration with expected calibration error, abstain and human-review thresholds, and evaluation with per-class precision, recall, macro F1 and a confusion matrix via a bundled script. Use when routing tickets, tagging intent or sentiment, triaging content, replacing a keyword or regex classifier with an LLM, or deciding whether to fine-tune or train a small model instead.
Use it when
- Routing, tagging or triaging text into a fixed label set with an LLM
- Replacing regex or keyword rules with a model
- You need per-class quality numbers or a CI gate on a classifier
- Deciding between prompting, fine-tuning or a small trained model
Use something else when
- You need many structured fields out of a document, not one label: use structured-extraction
- The classifier is a safety or abuse filter with policy tiers: use ai-content-moderation
How to check the result
Run scripts/classification_report.py on a frozen held-out test set and check macro F1, the top confusions and ECE against the agreed thresholds.
The mistake it prevents
Reporting accuracy on an imbalanced dataset and tuning the prompt against the test set, which hides a failing minority class and overstates quality.
An LLM classifier is done when labels have written decision rules, outputs are constrained to the label set, quality is reported per class on a labeled test set that was never used to write the prompt, and every prediction carries a confidence that is calibrated well enough to route low-confidence cases to a fallback.
Inspect first
- Read
AGENTS.mdand find the provider client wrapper, existing labels and any labeled data (support exports, moderation queues, spreadsheets). - Ask for or find 200+ labeled examples. Without them you can prototype, but you cannot claim accuracy. If none exist, label a sample first (see synthetic-data-generation only for augmenting rare classes, never as the test set).
- Check volume and latency budget: 10 requests/day and 10M requests/day lead to different designs.
Design the taxonomy before the prompt
- Write each label as a name plus a one-sentence definition plus “includes / excludes” lines. Most classifier errors are taxonomy errors.
- Make labels mutually exclusive, or explicitly switch to multi-label (a list output).
- Add an
otherorunclearlabel. Forcing a choice inflates confident errors. - Order matters for ties: state a precedence rule (“if both billing and bug, choose bug”).
- Two humans label the same 100 items; if they agree under ~85%, fix the definitions before blaming the model.
Prompt and output
Constrain the output with structured outputs so the model cannot invent labels.
from enum import Enum
from pydantic import BaseModel
from openai import OpenAI
class Label(str, Enum):
billing = "billing"
bug = "bug"
feature_request = "feature_request"
other = "other"
class Result(BaseModel):
reason: str # short; generated before the label, it improves borderline cases
label: Label
SYSTEM = """Classify the support message into exactly one label.
billing: charges, refunds, invoices, plan changes. Excludes: bugs in the billing page (bug).
bug: product behaves incorrectly or errors. Includes: billing page errors.
feature_request: asks for new capability. Excludes: complaints about existing behaviour (bug).
other: anything else, or not enough information.
Precedence: bug > billing > feature_request > other."""
client = OpenAI()
resp = client.responses.parse(
model=MODEL,
instructions=SYSTEM,
input=[*FEW_SHOT, {"role": "user", "content": text}],
text_format=Result,
)
result = resp.output_parsedFew-shot selection:
- Start zero-shot. Add examples only for confusions the eval shows.
- Use 1-3 examples per confused pair, chosen near the decision boundary, balanced across labels. Do not copy test items into the prompt.
- For large label sets (50+), retrieve the k most similar labeled examples per input (embedding search) instead of a fixed list.
Keep reason short (one sentence) or drop it for latency-critical paths; measure both.
Confidence
| Method | When | How |
|---|---|---|
| Token logprobs | Non-reasoning models that expose logprobs; single-token labels | Request logprobs with top_logprobs (Chat Completions: logprobs=True, top_logprobs=5); softmax over label tokens’ logprobs |
| Self-consistency | Any model, moderate volume | Sample N=5 at temperature ~0.7; confidence = share of votes for the majority label |
| Verbalized confidence (“0-100”) | Last resort | Poorly calibrated; recalibrate before using as a threshold |
| Small trained head on embeddings | High volume, stable labels | Logistic regression on embeddings gives calibrated probabilities cheaply |
Reasoning models usually do not return logprobs; use self-consistency or a separate
classifier for confidence. Make labels single, distinct first tokens (A, B, C or
short words) when relying on logprobs.
Calibration and abstention
- Report ECE on the test set. Under 0.05 is good; above 0.1 means thresholds will not mean what they say.
- Fix miscalibration with temperature scaling or isotonic regression fitted on a
validation split (scikit-learn
IsotonicRegression), never on the test split. - Choose an abstain threshold from the reliability table: route predictions below it to a human or a stronger model. Report coverage (share auto-labeled) and accuracy on the covered set together.
Evaluation
Produce a CSV with gold,pred,confidence for the held-out test set, then:
python3 scripts/classification_report.py results.csv --min-macro-f1 0.85It prints accuracy, per-class precision/recall/F1, macro F1, the confusion matrix and a calibration table with ECE, and exits 1 if macro F1 is below the gate. Use it in CI to block prompt or model changes that regress.
Read results like this:
- Macro F1, not accuracy, when classes are imbalanced. A 95% majority class makes 95% accuracy meaningless.
- Confusion matrix cells point to taxonomy fixes (definitions, precedence) before prompt tweaks.
- A label with predictions but zero gold support means the model is inventing a category or the gold data lacks it.
Model and cost choices
- Start with a small, fast model. Move up only for classes where it fails.
- Cascade: cheap model first; escalate to a stronger model when confidence is under the threshold. Report the share escalated and blended cost per 1k items.
- Above ~1M items/month with stable labels, compare against fine-tuning (openai-fine-tuning) or a trained classifier on embeddings; LLM zero-shot is often 10-100x the cost at equal quality.
- Batch offline backfills with the Batch API (openai-batch-api).
Workflow
- Read
AGENTS.md; locate labeled data and the client wrapper - Write label definitions, includes/excludes, precedence; measure human agreement
- Split data: dev (prompt iteration), validation (calibration), test (final report)
- Zero-shot baseline with enum structured output; run the report
- Add targeted few-shot examples for the top confusions; rerun on dev only
- Add confidence, calibrate on validation, pick abstain threshold
- Final run on test; record model, prompt hash, macro F1, ECE, coverage, cost
- Add the CI gate
Output template
## Classifier report: <name>
Model: <id> Prompt: <hash> Test set: <n> items (<date range>, <source>)
Macro F1: 0.87 Accuracy: 0.91 ECE: 0.04
Abstain threshold 0.70 -> coverage 88%, accuracy on covered 0.95
Top confusions: bug -> billing (14), feature_request -> other (9)
Cost: $0.42 per 1k items, p95 latency 610 ms
Changes since last run: <what changed and the delta>Gotchas
- Iterating on the prompt while looking at test results leaks the test set into the prompt. Freeze the test set and look at it only for final numbers.
- Label names leak meaning: renaming
othertospam_or_otherchanges predictions. Re-evaluate after any rename. - Input truncation silently drops the decisive sentence in long texts; classify a summary or the first and last N tokens deliberately.
- Distribution shift: re-sample production data monthly and re-run the report.
What the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "LLM Classification"
short_description: "Labels you can trust, with confidence that means something"
default_prompt: "$llm-classification Use an LLM to route incoming support tickets into our five queues and tell me how accurate it is"scripts/classification_report.py
#!/usr/bin/env python3
"""Score an LLM classifier: accuracy, per-class precision/recall/F1, macro F1,
confusion matrix, and expected calibration error (ECE) when confidences exist.
Input CSV (header required):
gold - true label
pred - predicted label
confidence - optional, 0..1 probability the classifier gave to `pred`
Usage:
python3 classification_report.py results.csv [--bins 10] [--min-macro-f1 0.85]
Exits 1 on invalid input, or when --min-macro-f1 is given and not met.
"""
import argparse
import csv
import sys
from collections import Counter, defaultdict
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("csv")
ap.add_argument("--bins", type=int, default=10)
ap.add_argument("--min-macro-f1", type=float)
args = ap.parse_args()
rows = []
with open(args.csv, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
cols = set(reader.fieldnames or [])
if not {"gold", "pred"} <= cols:
print("error: CSV needs gold and pred columns", file=sys.stderr)
return 1
has_conf = "confidence" in cols
for i, r in enumerate(reader, start=2):
conf = None
if has_conf and r["confidence"].strip():
try:
conf = float(r["confidence"])
except ValueError:
print(f"error: line {i}: confidence is not a number", file=sys.stderr)
return 1
if not 0 <= conf <= 1:
print(f"error: line {i}: confidence outside 0..1", file=sys.stderr)
return 1
rows.append((r["gold"].strip(), r["pred"].strip(), conf))
if not rows:
print("error: no rows", file=sys.stderr)
return 1
labels = sorted({g for g, _, _ in rows} | {p for _, p, _ in rows})
confusion = defaultdict(Counter)
for g, p, _ in rows:
confusion[g][p] += 1
n = len(rows)
correct = sum(1 for g, p, _ in rows if g == p)
print(f"n={n} accuracy={correct / n:.3f}\n")
print(f"{'label':<24}{'support':>8}{'precision':>11}{'recall':>8}{'f1':>7}")
f1s = []
for lab in labels:
tp = confusion[lab][lab]
fp = sum(confusion[g][lab] for g in labels if g != lab)
fn = sum(confusion[lab][p] for p in labels if p != lab)
support = tp + fn
prec = tp / (tp + fp) if tp + fp else 0.0
rec = tp / support if support else 0.0
f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
if support:
f1s.append(f1)
flag = "" if support else " (never gold: invented or rare label)"
print(f"{lab[:23]:<24}{support:>8}{prec:>11.3f}{rec:>8.3f}{f1:>7.3f}{flag}")
macro = sum(f1s) / len(f1s) if f1s else 0.0
print(f"\nmacro_f1={macro:.3f}")
print("\nconfusion (rows=gold, cols=pred)")
width = max(6, max(len(l[:12]) for l in labels) + 1)
print(" " * 14 + "".join(f"{l[:12]:>{width}}" for l in labels))
for g in labels:
print(f"{g[:13]:<14}" + "".join(f"{confusion[g][p]:>{width}}" for p in labels))
scored = [(c, g == p) for g, p, c in rows if c is not None]
if scored:
bins = [[] for _ in range(args.bins)]
for c, ok in scored:
bins[min(int(c * args.bins), args.bins - 1)].append((c, ok))
ece = 0.0
print(f"\ncalibration ({len(scored)} rows with confidence)")
print(f"{'bin':>11}{'count':>7}{'avg_conf':>10}{'accuracy':>10}")
for i, b in enumerate(bins):
if not b:
continue
avg_c = sum(c for c, _ in b) / len(b)
acc = sum(ok for _, ok in b) / len(b)
ece += len(b) / len(scored) * abs(avg_c - acc)
print(f"{i / args.bins:>5.1f}-{(i + 1) / args.bins:<5.1f}{len(b):>7}{avg_c:>10.3f}{acc:>10.3f}")
print(f"ECE={ece:.3f} (under 0.05 is well calibrated)")
if args.min_macro_f1 is not None and macro < args.min_macro_f1:
print(f"\nFAIL: macro_f1 {macro:.3f} < {args.min_macro_f1}")
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.