Benchmark Testing skill for Codex

A benchmark result is trustworthy when the measured code cannot be optimised away, inputs look like production, the run includes warmup and enough samples to report variance, and before/after are compared on the same machine with a statistical test. A single “x ms vs y ms” from one run is an anecdote.

Inspect first

1. Read AGENTS.md; find existing benchmarks (rg -n "func Benchmark|criterion_group|benchmark\(|bench\(|@Benchmark" .) and how they are run.
2. Establish the question: compare two implementations, guard against regressions, or find where time goes. Finding where time goes is profiling, not benchmarking (see node-profiling, python-profiling, go-pprof, flamegraphs).
3. Get realistic inputs: sizes and shapes from production (anonymised), not "hello".
4. Note the machine: CPU model, power mode, other load. Laptops on battery and shared CI runners add large noise.

Choose the harness

Stack	Default	Compare with
Go	testing.B with b.Loop() (Go 1.24+)	benchstat old.txt new.txt
Rust	criterion	Saved baselines: --save-baseline / --baseline
Python	pytest-benchmark	--benchmark-autosave, --benchmark-compare
JS / TS (Vitest project)	vitest bench (tinybench under the hood)	--outputJson then --compare
JS / TS standalone	tinybench or mitata	Run both variants in one process, report ratios
JVM	JMH	JMH’s own error bars; never hand-rolled System.nanoTime loops
.NET	BenchmarkDotNet	Baseline column ([Benchmark(Baseline = true)])

Benchmark.js is archived and unmaintained; migrate existing suites to tinybench or Vitest bench.

Rules every benchmark follows

• Keep results alive. Consume outputs so the compiler or JIT cannot delete the work: b.Loop() in Go keeps the loop body alive, std::hint::black_box in Rust, JMH Blackhole, return values from the benchmarked function in JS and Python.
• Set up outside the timed region. Build inputs once; do not measure allocation of the fixture.
• Warm up. JITs (V8, JVM, .NET, PyPy) optimise after many iterations. Harnesses above handle it; hand-written loops do not.
• Measure one thing. Separate benchmarks for parse, transform and serialise; a composite hides which part changed.
• Report variance. Mean without spread is meaningless; look at stddev, MAD or confidence intervals, and the number of samples.
• Test multiple sizes. 10, 1k, 100k elements reveal complexity changes a single size hides.

Go

func BenchmarkParse(b *testing.B) {
	data := loadFixture(b, "testdata/orders-10k.json")
	b.ReportAllocs()
	for b.Loop() {          // Go 1.24+; setup above is excluded automatically
		if _, err := Parse(data); err != nil {
			b.Fatal(err)
		}
	}
}

go test -run='^$' -bench='^BenchmarkParse$' -benchmem -count=10 ./internal/order > old.txt
# apply change
go test -run='^$' -bench='^BenchmarkParse$' -benchmem -count=10 ./internal/order > new.txt
benchstat old.txt new.txt      # go install golang.org/x/perf/cmd/benchstat@latest (network)

benchstat prints the delta with a p-value and ~ when the difference is not significant. Before Go 1.24, use for i := 0; i < b.N; i++ with b.ResetTimer() after setup and assign results to a package-level sink.

Rust (criterion)

[dev-dependencies]
criterion = "0.5"          # use the current release from crates.io

[[bench]]
name = "parse"
harness = false

use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};
use std::hint::black_box;

fn bench_parse(c: &mut Criterion) {
    let mut group = c.benchmark_group("parse");
    for size in [100usize, 10_000] {
        let input = make_orders(size);
        group.throughput(Throughput::Elements(size as u64));
        group.bench_with_input(BenchmarkId::from_parameter(size), &input, |b, i| {
            b.iter(|| mycrate::parse(black_box(i)))
        });
    }
    group.finish();
}

criterion_group!(benches, bench_parse);
criterion_main!(benches);

cargo bench --bench parse -- --save-baseline before    # on the base commit; baselines live in target/criterion, so share CARGO_TARGET_DIR across worktrees
cargo bench --bench parse -- --baseline before         # on the change; reports change and p-value

Python (pytest-benchmark)

def test_parse_orders(benchmark, orders_10k_bytes):
    result = benchmark(parse_orders, orders_10k_bytes)
    assert len(result) == 10_000          # also proves the work was done

pytest benchmarks/ --benchmark-only --benchmark-autosave
pytest benchmarks/ --benchmark-only --benchmark-compare --benchmark-compare-fail=mean:10%

Keep benchmarks out of the normal test run (--benchmark-skip in default addopts, or a separate directory). pytest-benchmark disables itself under pytest-xdist.

JavaScript (Vitest bench)

// slugify.bench.ts
import { bench, describe } from "vitest";
import { slugifyRegex, slugifyLoop } from "./slugify";

const titles = loadTitles();   // realistic: 5k titles with unicode and punctuation

describe("slugify 5k titles", () => {
  bench("regex", () => { for (const t of titles) slugifyRegex(t); });
  bench("char loop", () => { for (const t of titles) slugifyLoop(t); });
});

npx vitest bench --run --outputJson bench-main.json      # on the base commit
npx vitest bench --run --compare bench-main.json         # on the change

Comparing results

• Run before and after back to back on the same machine, same power settings, nothing else heavy running. Interleave runs (A, B, A, B) if noise is high.
• Treat differences under the noise floor (overlapping intervals, benchstat ~) as no change, and say so.
• Report relative change with interval and sample count, plus absolute times at the sizes that matter.
• Check memory and allocations alongside time; a faster path that allocates 3x can lose under real load.

Benchmarks in CI

Shared CI runners vary by 10-30% run to run. Options, in order of preference:

1. Instruction-count or simulated measurement (CodSpeed, iai-callgrind, Cachegrind) for stable deltas on noisy hardware.
2. Dedicated or bare-metal runners with fixed CPU frequency, comparing PR and base in the same job.
3. Hosted runners with generous thresholds (fail only above 15-25%) and trend tracking (github-action-benchmark), treating alerts as prompts to investigate.

Never fail a PR on a single noisy wall-clock comparison.

Output template

## Benchmark: <what> (<commit A> vs <commit B>)
Machine: <CPU, OS, power mode>. Harness: <tool + version>. Samples: <n> per variant.
| Case | Before | After | Delta | Significance |
|---|---|---|---|---|
| parse/10k | 4.21 ms ± 1% | 2.95 ms ± 2% | -29.9% | p=0.000, n=10 |
| allocs/op | 1,204 | 310 | -74% | |
Conclusion: <faster / no measurable change / slower> at production-sized inputs. Caveats: <noise, sizes not tested>.

Gotchas

• Constant inputs let compilers fold the whole computation; pass inputs through black_box or load them at runtime.
• Benchmarking with a debugger, coverage or -race enabled measures the instrumentation.
• Unoptimised builds produce numbers that mean nothing for production: a dev-mode JS bundle, a Python package built without its C extension, a JVM run with a debugger agent attached.
• Microbenchmark wins can vanish in the real system when the code is not on the hot path; confirm with a profile or an end-to-end measurement.