Fuzz Testing skill for Codex

Fuzzing pays off on code that consumes untrusted bytes. Done well means: a harness that reaches deep logic quickly, sanitizers that turn silent memory corruption into crashes, invariants that turn wrong answers into failures, a committed seed corpus, every crash minimised into a regression test, and a time-boxed fuzz run in CI.

Inspect first

1. Read AGENTS.md and the build system. Find existing fuzz targets: rg -n "func Fuzz|LLVMFuzzerTestOneInput|fuzz_target!|atheris|@FuzzTest" .
2. Rank candidate targets by exposure and complexity:
   • parsers and decoders (file formats, image/font/archive handling, protocol frames)
   • deserialisers, template engines, query or expression parsers, regex-like matchers
   • round-trip pairs (encode/decode, compress/decompress, serialise/parse)
   • code with hand-written bounds arithmetic, especially in C, C++ and unsafe Rust
3. Check the toolchain: clang with -fsanitize=fuzzer (Apple’s Xcode clang does not ship libFuzzer; use Homebrew LLVM or Linux), Go 1.18+, nightly Rust for cargo-fuzz, a Python version Atheris has wheels for.
4. Installing toolchains or corpora needs network; fuzzing itself runs offline.

Choose the engine

Language	Engine	Run
Go	Native testing.F	go test -run='^$' -fuzz='^FuzzParse$' -fuzztime=5m ./pkg/parser
C / C++	libFuzzer + ASan/UBSan (AFL++ for binaries without source changes)	./fuzz_parse corpus/ -max_total_time=300
Rust	cargo-fuzz (libFuzzer)	cargo +nightly fuzz run parse -- -max_total_time=300
Python	Atheris	python fuzz_parse.py corpus/ -max_total_time=300
Java / Kotlin	Jazzer (JUnit @FuzzTest)	JAZZER_FUZZ=1 ./gradlew test --tests '*FuzzTest'
Wide-input properties in any language	Property-based testing	see property-based-testing

Fuzzing and property-based testing overlap: use fuzzing for byte-level inputs and long unattended runs guided by coverage; use property tests for typed domain inputs inside the normal test suite.

Harness rules

• Deterministic: no clock, randomness, network or global state that survives between inputs.
• Fast: thousands of executions per second. Avoid logging, disk I/O and expensive setup per input; initialise once.
• Never exit or panic on invalid input as a matter of course: reject it and return. A panic should mean a bug.
• Check invariants, not only crashes: round-trip equality, output size bounds, no exception types other than the documented ones.
• Bound input size (-max_len) and recursion so timeouts point at real algorithmic bugs.

Go

func FuzzParseHeader(f *testing.F) {
	f.Add([]byte("GET / HTTP/1.1\r\nHost: a\r\n\r\n"))
	f.Add([]byte(""))
	f.Fuzz(func(t *testing.T, data []byte) {
		h, err := ParseHeader(data)
		if err != nil {
			return // invalid input is fine; a panic is not
		}
		out := h.Marshal()
		h2, err := ParseHeader(out)
		if err != nil || !h.Equal(h2) {
			t.Fatalf("round trip failed: %q -> %q (%v)", data, out, err)
		}
	})
}

Failing inputs are written to testdata/fuzz/FuzzParseHeader/<hash>; commit them, and plain go test replays them as regression cases from then on. Only one fuzz target runs per go test -fuzz invocation.

C / C++ (libFuzzer)

#include <cstdint>
#include <cstddef>
#include "parser.h"

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  if (size > 1 << 16) return 0;
  Doc doc;
  if (parse_doc(data, size, &doc) == PARSE_OK) {
    free_doc(&doc);
  }
  return 0;
}

clang++ -g -O1 -fsanitize=fuzzer,address,undefined fuzz_parse.cc parser.cc -o fuzz_parse
mkdir -p corpus && cp tests/fixtures/*.doc corpus/
./fuzz_parse corpus/ -dict=doc.dict -max_len=65536 -max_total_time=600 -jobs=4
./fuzz_parse crash-<sha1>             # reproduce one input
./fuzz_parse -minimize_crash=1 -runs=100000 crash-<sha1>

Rust (cargo-fuzz)

cargo install cargo-fuzz          # network
cargo fuzz init && cargo fuzz add parse

// fuzz/fuzz_targets/parse.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    if let Ok(msg) = mycrate::Message::decode(data) {
        let bytes = msg.encode();
        assert_eq!(mycrate::Message::decode(&bytes).unwrap(), msg);
    }
});

cargo fuzz tmin parse <artifact> minimises; derive arbitrary::Arbitrary on input structs to fuzz typed APIs instead of raw bytes.

Python (Atheris)

import sys
import atheris

with atheris.instrument_imports():
    import mylib.config_parser as cp

def test_one_input(data: bytes) -> None:
    fdp = atheris.FuzzedDataProvider(data)
    text = fdp.ConsumeUnicodeNoSurrogates(fdp.ConsumeIntInRange(0, 4096))
    try:
        cp.parse(text)
    except cp.ConfigError:
        pass          # documented error; anything else is a finding

atheris.Setup(sys.argv, test_one_input)
atheris.Fuzz()

Atheris lags new CPython releases; if pip install atheris has no wheel for the project’s version, run the fuzzer under the newest supported Python.

Corpus and dictionaries

• Seed with small, valid, diverse samples: test fixtures, spec examples, minimised real files. Ten good seeds beat ten thousand random ones.
• Add a dictionary of format tokens (magic bytes, keywords, delimiters) for grammars: kw1="<?xml" lines in a .dict file.
• Periodically minimise the corpus (-merge=1 into a fresh directory, cargo fuzz cmin).
• Store the corpus in the repo if small, or in CI cache or a bucket if large.

Triage a crash

1. Reproduce with the single input and the sanitizer build.
2. Minimise it.
3. Classify: memory safety (ASan report: heap-buffer-overflow, use-after-free), undefined behaviour, panic or unexpected exception, invariant violation, timeout or OOM.
4. Deduplicate by top stack frames; one bug often yields many crash files.
5. Fix the root cause, then add the minimised input as a regression test (Go testdata/fuzz, a unit test that loads the file, or the seed corpus that CI replays).
6. Memory-safety crashes in code that handles untrusted input are potential security bugs; tell the user before sharing details publicly and follow the project’s disclosure policy.

CI

• PRs: replay the corpus and regression inputs (fast, deterministic), plus a short fuzz run (2-10 minutes) on changed targets. ClusterFuzzLite provides GitHub Actions for this.
• Scheduled: long runs (hours) with corpus persistence; open projects can apply to OSS-Fuzz.
• Fail the job on any new crash and upload the crashing input as an artifact.

Verification

• Inject a known bug (off-by-one in a length check) and confirm the fuzzer finds it within minutes; if not, improve the harness or seeds.
• Check coverage of the harness (-print_final_stats=1, go test -fuzz reports “new interesting” inputs, cargo fuzz coverage) reaches the logic you care about.
• Executions per second stay in the thousands; if not, profile the harness.

Gotchas

• Fuzzing without sanitizers in C and C++ misses most memory bugs; they corrupt silently.
• A harness that catches every exception hides real bugs; catch only documented errors.
• Timeouts from huge inputs are noise unless the input size is realistic; set -max_len.
• Go fuzzing caches its generated corpus in $GOCACHE/fuzz; CI without that cache restarts from the seeds every run.