CPU Flame Graphs skill for Codex

Done well means: the profile was sampled at a sensible rate for long enough under the real workload, stacks are complete and symbolised (no walls of [unknown]), on-CPU and off-CPU time are not confused, and the reading names the wide frame that matters plus what code owns it.

How to read one

• x-axis: share of samples (alphabetical order of frames, not time). Width = cost.
• y-axis: stack depth; the top edge is what was on CPU. Wide plateaus at the top are the hot code. Wide frames lower down with many narrow children mean the cost is spread across callees.
• Colours are usually random within a palette (or encode language/kernel); do not read meaning into hue unless the generator documents it.
• Click to zoom into a subtree; search highlights matching frames and sums their share.
• Icicle graphs are the same data upside down (roots at top), as in Chrome DevTools and speedscope’s “left heavy” view.

Choose the profiler

Target	Tool	Notes
Any Linux process, C/C++/Rust/Go, kernel	perf record + FlameGraph or inferno	Needs perf_event access; best with frame pointers
Low-overhead, production, many processes	BCC profile / bpftrace	Needs eBPF-capable kernel and root/CAP_BPF
Time spent blocked (I/O, locks, sleep)	BCC offcputime (off-CPU flame graph)	Complements on-CPU; never mix the two
JVM (Java, Kotlin, Scala)	async-profiler (asprof)	Handles JIT symbols; cpu, alloc, lock, wall modes
Node.js	--cpu-prof first ($node-profiling); perf with --perf-basic-prof for mixed native
Python	py-spy first ($python-profiling); perf with -X perf (3.12+) for mixed native
Go	pprof flame graph view ($go-pprof)	perf works too; Go keeps frame pointers
macOS	Instruments Time Profiler, or sample <pid>	perf/eBPF are Linux only

Workflow

- [ ] 1. Reproduce the load; note the process (PID), duration and environment
- [ ] 2. Check permissions and symbols (below) before recording
- [ ] 3. Record 30-60 s at 49-99 Hz under steady load
- [ ] 4. Fold stacks and render the SVG / HTML
- [ ] 5. Read: widest top-edge frames, then their ancestors in your code
- [ ] 6. Fix; record again with identical settings; compare (differential graph)

Permissions (ask first)

• perf as non-root depends on kernel.perf_event_paranoid (check with sysctl kernel.perf_event_paranoid); lowering it, or adding CAP_PERFMON/CAP_SYS_ADMIN to a container, changes host security settings. Ask the user before changing either, and revert afterwards.
• Codex sandboxes usually cannot run perf or eBPF; these steps typically need to run on the user’s host or a test machine with approval.
• Never profile production hosts without the user’s explicit go-ahead; sampling at 99 Hz is cheap, but --call-graph dwarf and high frequencies are not.

Linux perf

# One process, 99 Hz, with call graphs, 30 s
sudo perf record -F 99 -g -p <pid> -- sleep 30
# Whole system
sudo perf record -F 99 -a -g -- sleep 30

sudo perf script > out.perf
# FlameGraph scripts (git clone https://github.com/brendangregg/FlameGraph, needs network)
./FlameGraph/stackcollapse-perf.pl out.perf > out.folded
./FlameGraph/flamegraph.pl out.folded > cpu.svg
# Or inferno (cargo install inferno)
sudo perf script | inferno-collapse-perf | inferno-flamegraph > cpu.svg

Use 99 Hz (not 100) to avoid lockstep with timers. 49 Hz for long or production captures.

Broken stacks and [unknown]

Symptom	Fix
Stacks only 1-2 frames deep	Binary built without frame pointers: rebuild with -fno-omit-frame-pointer (Rust: -C force-frame-pointers=yes), or perf record --call-graph dwarf (bigger, slower)
[unknown] frames in JIT code (Node)	Run with node --perf-basic-prof (writes /tmp/perf-<pid>.map); add --interpreted-frames-native-stack for interpreted frames
Python frames show as _PyEval_EvalFrameDefault	Python 3.12+: python -X perf app.py or PYTHONPERFSUPPORT=1
Library frames unsymbolised	Install debug symbol packages (-dbg/-debuginfo) or use debuginfod
Containerised process	Run perf in the same PID and mount namespace, or copy binaries and perf maps for symbolisation

Recent Ubuntu (24.04+) and Fedora releases build system packages with frame pointers, which makes stacks through libc and system libraries far more complete.

eBPF

# On-CPU, folded output, one process (Ubuntu package names end in -bpfcc)
sudo profile-bpfcc -F 99 -df -p <pid> 30 > cpu.folded
./FlameGraph/flamegraph.pl cpu.folded > cpu.svg

# Off-CPU: where threads block, in microseconds
sudo offcputime-bpfcc -df -p <pid> 30 > offcpu.folded
./FlameGraph/flamegraph.pl --color=io --title="Off-CPU" --countname=us offcpu.folded > offcpu.svg

# bpftrace one-liner: user + kernel stacks for a PID
sudo bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack, kstack] = count(); }'

Off-CPU graphs answer “why is it slow while CPU is low”: lock waits, disk I/O, network, sleeps, page faults. Width there is blocked time, not CPU.

async-profiler (JVM)

asprof -d 30 -e cpu -f /tmp/cpu.html <pid>        # interactive HTML flame graph
asprof -d 30 -e alloc -f /tmp/alloc.html <pid>    # allocation flame graph
asprof -d 30 -e lock -f /tmp/lock.html <pid>      # contended locks
asprof -d 30 -e wall -t -f /tmp/wall.html <pid>   # wall clock per thread, includes waiting

In containers without perf events, use -e itimer for CPU. Output can also be JFR (-f out.jfr) for JDK Mission Control. For JVM-level tuning see $jvm-tuning.

Differential flame graphs

./FlameGraph/difffolded.pl before.folded after.folded | ./FlameGraph/flamegraph.pl > diff.svg

Red = more samples after, blue = fewer. Normalise capture conditions (same load, same duration, same frequency) or the diff shows load differences, not code differences.

Reading checklist

1. Total samples and duration: is the profile big enough (thousands of samples)?
2. Top-edge plateaus: which leaf functions dominate?
3. Walk down to the first frame in project code: that is the owner to change.
4. Kernel frames large? Syscall-heavy code (small writes, futex contention, page faults).
5. GC or runtime frames large? Allocation pressure: use the allocation profiler of the runtime.
6. Many processes? Filter the folded file with grep before rendering.

Report template

## Flame graph: <service> on <host>, <load>, perf 99 Hz x 30 s (14,800 samples)
On-CPU hot paths:
1. 34% json_encode -> encode_string (src/api/serialize.c:210): per-field escaping, fix: precomputed escape table
2. 18% kernel copy_user_enhanced: many 4 KB writes, fix: buffer output
Off-CPU: 41% futex_wait in pool.acquire, connection pool too small
Files: cpu.svg, offcpu.svg (attached, not committed)
After fix (same settings): encode_string 34% -> 9%; diff.svg