AI Agent Evaluation skill for Codex

An agent eval is trustworthy when each task starts from a known environment state, success is decided by checking the resulting state (files, database rows, API calls made) rather than by reading the agent’s claim, every task runs several times so reliability is measured, unsafe actions fail the task regardless of outcome, and cost and steps are reported next to success.

Inspect first

1. Read AGENTS.md, the agent’s entry point, tool definitions and tracing setup. You need to be able to run one task headlessly and capture the full trajectory (messages, tool calls, tool results, tokens).
2. Find real tasks: support tickets the agent handled, issues it fixed, logs of failures. Production failures are the most valuable cases.
3. Decide what “environment” means for this agent: a git repo checkout, a seeded database, a mock API server, a browser profile. It must be resettable in seconds.

Task suite design

Each task is a record:

id: refund-partial-001
category: refunds              # report per category
prompt: "Customer 4411 was double charged for order 9001. Fix it and reply to them."
setup: fixtures/refund_double_charge.sql
success:                       # state checks, all must pass
  - sql: "SELECT count(*) FROM refunds WHERE order_id=9001 AND amount_cents=4999"
    equals: 1
  - outbox_contains: {to: "[email protected]", matches: "refund"}
forbidden:                     # any match fails the task
  - tool: issue_refund
    where: "amount_cents > 4999"
  - tool: delete_customer
budget: {max_steps: 25, max_cost_usd: 0.50, max_seconds: 180}
difficulty: medium
source: ticket-2026-09-14-118

Coverage rules:

• 30-50 tasks is enough to start; 150+ before trusting small differences.
• Mix: routine tasks, tasks needing clarification (the right move is to ask), impossible or out-of-policy tasks (the right move is to refuse or escalate), and tasks with misleading tool output or injected instructions in data.
• Keep a held-out set that prompt authors never look at.

Grading, strongest first

Grader	Use for	Notes
State check (code)	Anything with a verifiable end state	Run tests, query DB, diff files, inspect the mock API’s request log
Trajectory rule (code)	Forbidden actions, required confirmations, tool misuse	Fails the task even if the outcome is right
Reference comparison	Answers with a known correct value	Normalize before comparing
LLM judge with rubric	Reply quality, tone, whether a clarification was appropriate	Calibrate against 50+ human labels; agreement target >= 0.8
Human review	New categories, judge disagreements	Sample, do not review everything

Never grade with “did the agent say it succeeded”. Agents routinely report success on failed tasks.

LLM-judge rubric pattern: one criterion per call, binary or 3-level scale, with the trajectory excerpt and the expected behaviour, returning {"pass": bool, "evidence": str}. Use a different model family from the agent where possible. See llm-evals for judge calibration.

Runner

import json, statistics, time

def run_task(task, agent, trial: int) -> dict:
    env = Environment.reset(task["setup"], seed=trial)        # fresh state every trial
    t0 = time.time()
    traj = agent.run(task["prompt"], env=env,
                     max_steps=task["budget"]["max_steps"])   # returns steps, usage, final
    outcome = all(check(env, c) for c in task["success"])
    violations = [r for r in task.get("forbidden", []) if violates(traj, r)]
    return {
        "task": task["id"], "trial": trial,
        "success": outcome and not violations,
        "violations": [str(v) for v in violations],
        "steps": len(traj.steps),
        "tool_errors": sum(1 for s in traj.steps if s.is_tool_error),
        "cost_usd": traj.usage.cost_usd,
        "seconds": round(time.time() - t0, 1),
        "over_budget": traj.usage.cost_usd > task["budget"]["max_cost_usd"],
        "trace_url": traj.trace_url,
    }

def summarize(rows, k):
    by_task = {}
    for r in rows:
        by_task.setdefault(r["task"], []).append(r["success"])
    pass_at_k = statistics.mean(any(v[:k]) for v in by_task.values())   # solved at least once
    pass_hat_k = statistics.mean(all(v[:k]) for v in by_task.values())  # solved every time
    return {"tasks": len(by_task), "pass@1": statistics.mean(r["success"] for r in rows),
            f"pass@{k}": pass_at_k, f"pass^{k}": pass_hat_k,
            "median_cost": statistics.median(r["cost_usd"] for r in rows),
            "median_steps": statistics.median(r["steps"] for r in rows)}

• Run each task k = 3-5 times. pass^k (all trials succeed) is what users experience for a repeated workflow; it falls much faster than pass@1.
• Run tasks in parallel only if environments are isolated (separate containers, DB schemas, worktrees).
• Mock external side effects (email, payments, production APIs). Evaluation runs must never send real messages or charge cards; if a task needs a live system, use its sandbox and confirm with the user first.

Trajectory diagnostics

Report these per task and per category; they explain failures that success rate hides:

• Steps vs a reference solution’s steps (wasted exploration, loops).
• Tool error rate and repeated identical calls (a sign of bad tool descriptions; see agent-tool-design).
• Context growth per step and truncation events.
• First divergence step: where the trajectory stopped matching a successful one.
• Asked-for-clarification rate on ambiguous tasks vs unambiguous ones.

Comparing variants

• Change one thing at a time (model, prompt, tool set, max steps).
• Same tasks, same seeds, same k. Report per-category deltas, not only the total.
• With 50 tasks, a 4-point difference is usually noise. Use a paired comparison per task (bootstrap over tasks) before declaring a winner.
• Cost and latency are part of the result: a variant with +3 points at 2x cost may lose.

Regression gate

• Keep a fast smoke suite (10-20 tasks, k=1) per PR and the full suite (k=3+) nightly or before release.
• Gate on: no new forbidden-action violations (zero tolerance), pass^k not below the last release minus an agreed margin, median cost within budget.
• Store every run’s rows and trace links; the diff between runs is the debugging tool.

Workflow

• Read AGENTS.md; make one task run headlessly with a full trace
• Build resettable environments and fixtures
• Write 30-50 tasks from real usage with state checks and forbidden rules
• Implement runner with k trials, budgets and isolation
• Add LLM-judge criteria only where state checks cannot decide; calibrate them
• Baseline run; read 10 failed trajectories end to end
• Fix the agent or the tasks (ambiguous tasks are bugs in the eval); rerun
• Wire smoke suite into CI and full suite into release checks

Output template

## Agent eval: <agent> @ <commit>  (<date>)
Model: <id>  Tasks: 64 (held-out 20)  Trials: k=4
| Metric | Value | Prev release |
|---|---|---|
| pass@1 | 0.73 | 0.70 |
| pass^4 | 0.52 | 0.47 |
| Forbidden-action violations | 0 | 1 |
| Median cost / task | $0.21 | $0.24 |
| Median steps | 11 | 13 |
By category: refunds 0.81, account access 0.58 (down 6), escalations 0.90
Top failure modes: 1) retries the same failing tool call 4+ times (9 trials) ...
Traces: <links to 5 representative failures>

Gotchas

• Shared state between trials (same DB, same repo) makes later trials pass or fail because of earlier ones.
• Tasks with more than one valid outcome need checks that accept all of them; otherwise the eval punishes correct creativity.
• Model providers update aliases; pin model versions in eval configs and record them.
• Agents that see eval tasks in their prompt examples or memory overfit; keep eval data out of the agent’s retrieval index.