Python Profiling skill for Codex

Done well means: the slow operation is reproducible with a fixed input, the profiler matches the question (where is time spent, which line, who allocates), the hotspot is named down to a function and line, and the fix is proven by re-running the same input with the same measurement.

Choose the tool

Question	Tool	Overhead	Needs code change?
Which functions take the time (script, test, local repro)?	cProfile + pstats / snakeviz	High (2x+), skews tiny functions	No
What is a running process doing right now (prod, worker, hung job)?	py-spy top / py-spy dump / py-spy record	Low, sampling	No, attaches by PID
Which line inside this one function is slow?	line_profiler (kernprof)	Very high on profiled functions	@profile decorator
Who allocates memory, what is peak?	memray	Moderate	No
What grew between two points in time?	tracemalloc snapshots	Moderate	A few lines
CPU vs native vs memory in one view	scalene	Low-moderate	No
Async code: which coroutine is slow?	py-spy record --idle or pyinstrument (async-aware)	Low	No

Install with the project’s tool (uv add --dev py-spy memray line_profiler, poetry, pip); needs network. Read AGENTS.md for the package manager and how to run the app.

Workflow

- [ ] 1. Reproduce with a fixed input and time it (`time`, or `python -m timeit` for small units)
- [ ] 2. Whole-program profile (cProfile locally, py-spy for running services)
- [ ] 3. Identify top cumulative and top self-time functions in project code
- [ ] 4. Drill into the hottest function with line_profiler if it is not obvious
- [ ] 5. Fix one thing; re-run the same input; compare wall time and the profile
- [ ] 6. Add a regression guard if the path is critical (benchmark test)

cProfile

python -m cProfile -o out.prof -m myapp.cli build --input data/sample.csv
python - <<'PY'
import pstats
s = pstats.Stats("out.prof").strip_dirs()
s.sort_stats("cumulative").print_stats(25)   # where time accumulates
s.sort_stats("tottime").print_stats(15)      # where time is actually spent
s.print_callers("expensive_fn")              # who calls it, how often
PY

Read ncalls first: a cheap function called 2 million times is usually an algorithm problem (nested loop, per-row query), not a micro-optimisation target. Visualise with snakeviz out.prof (opens a local browser).

Profiling a single block:

import cProfile, pstats
with cProfile.Profile() as pr:
    run_report(params)
pstats.Stats(pr).sort_stats("cumulative").print_stats(20)

py-spy for running processes

py-spy top --pid 12345                       # live view, like top for functions
py-spy dump --pid 12345                      # current stack of every thread (hung process)
py-spy record -o profile.svg --pid 12345 --duration 60      # flame graph
py-spy record -o profile.speedscope.json -f speedscope -- python -m myapp.worker

Useful flags: --subprocesses (gunicorn/uwsgi/multiprocessing workers), --idle (include waiting threads, needed to see I/O and lock waits), --gil (only samples holding the GIL), --native (C extension frames, e.g. numpy). Attaching needs ptrace permission: on macOS run with sudo, in Docker add --cap-add SYS_PTRACE. Both change security settings, so ask the user before doing either, and never attach to production processes without approval.

line_profiler

from line_profiler import profile   # line_profiler 4+

@profile
def build_rows(records): ...

LINE_PROFILE=1 python -m myapp.cli build     # writes and prints results
kernprof -l -v script.py                      # classic form

Remove the decorators before committing.

Memory

memray run -o run.bin -m myapp.cli build --input data/big.csv
memray flamegraph run.bin        # HTML flame graph of allocations at peak
memray stats run.bin             # top allocators, peak, totals
memray run --native ...          # include C/C++ frames (numpy, pandas internals)

import tracemalloc
tracemalloc.start(25)
before = tracemalloc.take_snapshot()
process_batch()
after = tracemalloc.take_snapshot()
for stat in after.compare_to(before, "lineno")[:15]:
    print(stat)

Growth that never comes back across repeated batches is a leak: $memory-leak-hunting.

Fix patterns

Hotspot	Fix
Per-row DB queries, per-item HTTP calls	Batch (IN (...), bulk endpoints), prefetch: $n-plus-one-queries
Python loops over numeric data	Vectorise with numpy/pandas/polars; avoid df.apply row-wise and iterrows
x in list inside a loop	set or dict lookup
String building with += in loops	"".join(parts), or write to io.StringIO
Repeated pure computation	functools.lru_cache / cache with bounded size
JSON or regex in hot loop	Compile regex once; orjson/msgspec for heavy serialisation
CPU-bound in threads	GIL serialises it: ProcessPoolExecutor, or native code; free-threaded 3.13t/3.14t only after checking extension support
Blocking calls inside async def	asyncio.to_thread or an async client; py-spy shows the event loop thread stuck in sync I/O

Gotchas

• cProfile adds overhead per call, so code with many tiny calls looks slower than it is relative to C-heavy code. Confirm conclusions with py-spy or wall-clock timing.
• Profiling under pytest includes fixture setup; profile the function, not the session.
• Django DEBUG=True stores every SQL query and slows everything; profile with production-like settings.
• First call includes import time and caches warming; measure steady state separately (python -X importtime for import cost).
• Python 3.12+ can expose Python frames to Linux perf with -X perf; see $flamegraphs for system-wide profiles that mix Python and native code.

Verification

Re-run with the same input and report wall time (median of 3 or more), the hotspot’s share of the new profile, and peak memory if that was the problem. For critical paths add a benchmark (pytest-benchmark, see $benchmark-testing) so the regression cannot return unnoticed.

Report template

## Python profile: <command / endpoint> (Python <ver>, input <size>)
Wall time: 48.2 s -> 6.1 s (median of 3). Peak RSS: 3.1 GB -> 0.9 GB
Tool: cProfile (local), memray (peak)
| Hotspot | Before | Cause | Fix |
|---|---|---|---|
| reports/build.py:112 enrich_row (1.2M calls) | 61% cum | per-row dict rebuild + list membership | precomputed set, vectorised merge |
Profiles: out.prof, run.bin (not committed)