Go pprof Profiling skill for Codex

Done well means: the profile type matches the symptom, it was captured under realistic load from a binary built like production, the conclusion names a function and line with its share of samples, and the fix is proven with benchstat or a diffed profile.

Pick the profile

Symptom	Profile	Endpoint / flag
High CPU, slow handlers	CPU	/debug/pprof/profile?seconds=30, -cpuprofile
Memory high now	heap, inuse_space	/debug/pprof/heap
GC busy, allocation churn	heap, alloc_space / alloc_objects	/debug/pprof/allocs
Goroutine count climbs, hangs	goroutine	/debug/pprof/goroutine?debug=2
Latency high, CPU low, channels/IO waits	block	needs runtime.SetBlockProfileRate
Lock contention	mutex	needs runtime.SetMutexProfileFraction
Scheduling, GC pauses, latency of one request	execution trace	/debug/pprof/trace?seconds=5, go tool trace

Expose pprof safely

import (
    "net/http"
    _ "net/http/pprof" // registers /debug/pprof/* on http.DefaultServeMux
)

func init() {
    go func() {
        // Separate, loopback-only listener: never on the public port
        _ = http.ListenAndServe("127.0.0.1:6060", nil)
    }()
}

• If the app serves on http.DefaultServeMux, importing net/http/pprof exposes the endpoints publicly. Use a dedicated mux or loopback listener and confirm with curl -s localhost:<public-port>/debug/pprof/ that it is not reachable.
• Reach a pod’s loopback with kubectl port-forward pod/<name> 6060 (requires cluster access; ask before touching production).
• Block and mutex profiling cost CPU; enable with modest rates (SetBlockProfileRate(10000) in ns, SetMutexProfileFraction(100)) and only for the investigation.

Workflow

- [ ] 1. Read AGENTS.md / go.mod: Go version, build flags, how the service runs
- [ ] 2. Reproduce with load (k6, hey, vegeta) or a benchmark with fixed input
- [ ] 3. Capture the matching profile during steady load (30 s CPU)
- [ ] 4. top / list / web view: find the hot function and line
- [ ] 5. Fix one thing; capture again; compare with -diff_base or benchstat
- [ ] 6. Consider PGO with a representative production CPU profile

Reading profiles

go tool pprof -http=:8080 http://127.0.0.1:6060/debug/pprof/profile?seconds=30
go tool pprof -http=:8080 -sample_index=alloc_space http://127.0.0.1:6060/debug/pprof/allocs
go tool pprof -top -nodecount=25 cpu.pprof
go tool pprof -list 'orders\.BuildReport' cpu.pprof     # per-line costs in that function
go tool pprof -http=:8080 -diff_base=before.pprof after.pprof

In the web UI: Flame Graph for shape, Top for flat (self) vs cum (inclusive), Source for lines. flat high = this function’s own code; cum high and flat low = it calls something expensive.

Recognise common frames:

Hot frames	Meaning	Usual fix
runtime.mallocgc, runtime.gcBgMarkWorker, scanobject	Allocation pressure	Reuse buffers, preallocate slices (make([]T, 0, n)), avoid []byte↔string copies, sync.Pool for hot short-lived objects
runtime.mapassign, mapaccess	Map-heavy hot loop	Pre-size maps, use slices for dense keys, avoid map of structs copied by value
encoding/json.*, reflect.*	Reflection-based serialisation	Avoid re-encoding, stream with json.Encoder, consider generated or faster codecs after measuring
runtime.convT*, runtime.assertE2I	Interface boxing in hot path	Concrete types, generics in hot loops
sync.(*Mutex).Lock in mutex profile	Contention	Shard the lock, shorten critical section, sync.RWMutex only for read-heavy, atomics for counters
syscall.Syscall, internal/poll in CPU	Many small I/O calls	Buffer with bufio, batch writes
regexp.(*Regexp).doExecute	Regex in a loop	Compile once at package level; strings functions when possible

Escape analysis explains surprise heap allocations: go build -gcflags='-m' ./internal/orders 2>&1 | grep 'escapes to heap'.

Benchmarks with profiles

go test -run '^$' -bench 'BenchmarkBuildReport' -benchmem -count 10 \
  -cpuprofile cpu.pprof -memprofile mem.pprof ./internal/orders > old.txt
# make the change, then the same command > new.txt
go run golang.org/x/perf/cmd/benchstat@latest old.txt new.txt

Use -count 10 or more and benchstat; a single run is noise. Use b.Loop() (Go 1.24+) or b.ResetTimer() so setup is excluded. Benchstat needs network the first time.

Goroutine leaks

curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head -60   # grouped counts

Thousands of goroutines parked at the same line (chan receive, select, IO wait) are the leak. Typical causes: missing ctx.Done() case, unbuffered channel send with no reader after a timeout, HTTP response bodies not closed. In tests, go.uber.org/goleak catches leaks automatically.

Runtime knobs (after code fixes, not instead)

• GOMEMLIMIT (soft memory limit, e.g. 90% of container memory) lets the GC work harder near the limit instead of OOMing; pair with sensible GOGC.
• Go 1.25+ sets GOMAXPROCS from the container CPU limit; on older versions use go.uber.org/automaxprocs to avoid throttling.
• PGO: save a representative production CPU profile as default.pgo in the main package directory; go build uses it automatically (Go 1.21+). Typical gain is a few percent of CPU; refresh it periodically.
• Go 1.25 adds runtime/trace.FlightRecorder for capturing the last few seconds of trace around a rare slow request.

Gotchas

• A CPU profile only shows on-CPU time. A handler waiting on the database is invisible there; use block profiles or tracing.
• inuse_space shows what is live now; alloc_space shows churn since start. Leaks are inuse; GC pressure is alloc.
• Profiles from -race or debug builds (-gcflags=all=-N -l) are not representative.
• Heap profiles are sampled (every 512 KB by default); small one-off allocations can be missed, totals are estimates.

Report template

## pprof: <service> (Go <ver>, <load>)
Profile: CPU 30 s at 2k rps; allocs during same window
| Function | flat | cum | Finding |
|---|---|---|---|
| orders.BuildReport | 4% | 38% | json.Marshal per row inside loop |
| runtime.mallocgc | 17% | 22% | allocation churn from the same loop |
Fix: encode once with json.Encoder over a preallocated slice
benchstat: BuildReport -61% time/op, -83% allocs/op (n=10, p=0.000)
Load test: p99 210 ms -> 88 ms, CPU 3.1 -> 1.4 cores