Benchmark Comparisons skill for Codex

A comparison is fair when each contender gets the same workload, hardware and tuning effort, the measurement captures what users would experience, differences are reported with their variance, and anyone can rerun it from the repository. The goal is the truth about a workload, not a winner.

Scope the claim first

Write the question as a sentence the result can answer:

Weak: "Is Bun faster than Node?"
Good: "For our JSON API (p50 2 KB responses, 200 concurrent clients) on a 4-vCPU Linux
      VM, which of Node 22 and Bun 1.x gives lower p99 latency at 5,000 req/s?"

Comparison type	Tooling	Hand-off
CLI commands, builds, scripts	hyperfine	here
Functions inside one codebase	criterion, pytest-benchmark, benchmark.js, JMH	$benchmark-testing
HTTP services under load	k6, wrk2, vegeta (fixed arrival rate)	$load-testing for full test design
Databases	Standard workloads (pgbench, sysbench, YCSB) plus your own queries	here
ML models	Fixed eval set, same prompts, same decoding settings	here

Workflow

- [ ] 1. Write the claim, metric and workload
- [ ] 2. Pin the environment and record it
- [ ] 3. Configure each contender fairly (defaults AND tuned)
- [ ] 4. Validate correctness of outputs
- [ ] 5. Warm up, then run repeated trials, interleaved
- [ ] 6. Analyse distributions and uncertainty
- [ ] 7. Write up with limits; publish the harness

2. Environment

• Same machine, or identical instance types in the same region and zone. Record CPU model, cores, RAM, disk, OS and kernel, runtime versions, and contender versions with exact commit or release.
• Reduce noise: close other workloads, plug in laptops, avoid burstable cloud instances (t-class CPU credits distort results), and run the load generator on a separate host for network services.
• Changing CPU governors, turbo settings or dropping OS caches needs root. Ask the user before any system-level change, and record what was changed.
• In a Codex sandbox, network-bound and multi-host benchmarks usually cannot run; produce the harness and the run instructions, and say results must come from the target host.

3. Fair configuration

• Run each contender with documented defaults and with reasonable tuning; report both. Tuning one side only is the most common unfair benchmark.
• Give each contender the same features enabled (durability, TLS, compression, logging, safety checks). A database with fsync off is not comparable to one with fsync on.
• Ask or invite the maintainers’ recommended configuration when publishing.
• Use realistic data sizes: benchmarks that fit in L2 cache say little about production.

4. Correctness

Verify each contender produced the same, correct output (checksums, row counts, test suite). A fast wrong answer is not a result.

5. Running

# CLI tools: warmup, enough runs, no shell overhead, exported results
hyperfine --warmup 3 --runs 30 -N \
  --export-markdown results.md --export-json results.json \
  'node dist/build.js' 'bun dist/build.js'

# Clean state per run (for cold-cache or fresh-directory comparisons)
hyperfine --prepare 'rm -rf .cache' --runs 20 'tool-a build' 'tool-b build'

• Warmup: discard JIT, cache and connection-pool warmup; report cold start separately if it matters.
• Repetition: at least 10 runs for stable CLI timings, more when variance is high; for load tests, several runs of several minutes each.
• Interleave contenders (A, B, A, B) or randomise order so drift (thermal throttling, background jobs) does not favour one.
• Latency under load: use a fixed arrival rate (k6 constant-arrival-rate, wrk2 -R). Closed-loop tools that wait for each response before sending the next hide queueing delay (coordinated omission) and make slow systems look good.

6. Analysis

• Report median and spread (IQR or standard deviation) for timings; p50, p95, p99 and max for latency, plus throughput and error rate at each load level.
• Never average percentiles across runs; merge raw samples or HDR histograms.
• Use a confidence interval (bootstrap of the median is simple and distribution-free) or a Mann-Whitney U test for “is A faster than B”. If intervals overlap heavily, call it no meaningful difference.
• Report resource use (CPU, memory, disk) alongside speed; a 5% speedup at 2x memory is a trade-off, not a win.
• Plot distributions (box or violin plots, latency-vs-load curves), not just bar charts of means.

Gotchas

• Benchmarking the harness: the client saturates before the server. Check client CPU.
• Dead-code elimination and constant folding make microbenchmarks measure nothing.
• Caches: the second run of a database query reads from memory. Decide cold vs warm and state it.
• Version drift: a comparison against last year’s release of the competitor is misleading.
• Vendor benchmarks are marketing until reproduced on your workload.
• Licence terms: some commercial database and software licences restrict publishing benchmark results (so-called DeWitt clauses). Check before publishing externally, and get the user’s go-ahead before posting any comparison publicly.
• Small differences (under ~5%) on shared or cloud hardware are often noise. Rerun on another day or host before claiming them.

Verification

• Rerunning the harness on the same host reproduces medians within the reported spread.
• Every contender’s output passed the correctness check.
• The write-up states versions, configs (default and tuned), hardware and the commands.
• A maintainer of the losing tool would agree the setup was fair, or the disagreement is documented.

Report template

# <A> vs <B> for <workload> (<date>)
**Claim tested:** <sentence>
**Result:** <A> p99 = 18 ms (95% CI 17-19) vs <B> 24 ms (CI 22-27) at 5k req/s;
  throughput equal; <B> uses 40% less memory. Under 1k req/s: no meaningful difference.

## Setup
Hardware | OS/kernel | Versions (exact) | Data size | Load generator host

## Configuration
| Setting | A default | A tuned | B default | B tuned |

## Method
Warmup, runs, interleaving, arrival rate, duration, correctness check.

## Results
Tables and plots; raw data in results/*.json

## What this does not show
Other workloads, scales, hardware; cold start; cost.

## Reproduce
git clone ... && ./run.sh   (exact commands)