Jupyter Notebooks skill for Codex

A notebook is done when a fresh kernel can run it top to bottom with one command and get the same results, its diff in git shows code changes rather than base64 blobs, and any logic someone else depends on lives in an importable, tested module rather than a cell.

Inspect first

1. Read AGENTS.md, pyproject.toml or requirements*.txt, and any environment.yml. Note the package manager (uv, pip, conda) and Python version.
2. Check the notebook’s kernel and execution order without opening the UI:

python - analysis.ipynb <<'PY'
import json, sys
nb = json.load(open(sys.argv[1]))
print("kernel:", nb["metadata"].get("kernelspec", {}).get("name"))
counts = [c.get("execution_count") for c in nb["cells"] if c["cell_type"] == "code"]
print("execution counts:", counts)
print("out of order:", [c for c in counts if c] != sorted(c for c in counts if c))
PY

3. Look for hidden state: cells that read variables defined lower down, %store, %run of other notebooks, absolute paths (/Users/...), and credentials in cells.

Codex edits .ipynb JSON reliably only through a tool. Prefer editing the paired .py file (Jupytext) or using nbformat in a script; never hand-edit cell JSON with string replacement.

Make it run from a clean kernel

jupyter nbconvert --to notebook --execute analysis.ipynb --output analysis.executed.ipynb \
  --ExecutePreprocessor.timeout=600

If it fails, fix in this order:

1. Imports and installs: move every import and any %pip install to the first cells; replace %pip install with entries in the project’s dependency file.
2. Out-of-order definitions: move the defining cell above its first use.
3. Paths: resolve relative to the repo (Path(__file__) does not exist in notebooks; use a ROOT = Path.cwd() anchor or a project config) and never hard-code a home directory.
4. Randomness: set seeds once at the top (np.random.default_rng(42), random.seed, framework seeds) and pass the generator around.
5. External data: read from a declared location and print its row count and a hash, so reruns on changed data are visible.

Executing a notebook that downloads data or calls APIs needs network; in a sandbox with network off, say so and ask for approval or a local copy of the data.

Kernel and environment

uv add --dev ipykernel                     # or: pip install ipykernel
uv run python -m ipykernel install --user --name myproject --display-name "myproject (uv)"

Set the notebook’s kernelspec to that name so collaborators get the project environment, not whatever python3 is on their path. Pin dependencies in the lockfile, not in cells.

Version control

Problem	Tool	Setup
Outputs and execution counts bloat diffs	nbstripout	nbstripout --install (git filter) or as a pre-commit hook
Reviewers cannot read .ipynb diffs	Jupytext pairing	jupytext --set-formats ipynb,py:percent analysis.ipynb
Lint and format notebook code	nbQA	nbqa ruff analysis.ipynb, or ruff directly (it lints .ipynb natively)
Diff and merge notebooks	nbdime	nbdime config-git --enable

With Jupytext pairing, the .py:percent file is the reviewable source; jupytext --sync updates whichever side is older. Commit both, or commit only the .py and generate the notebook (decide once and record it in AGENTS.md).

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/kynan/nbstripout
    rev: 0.8.1
    hooks: [{ id: nbstripout }]
  - repo: https://github.com/mwouts/jupytext
    rev: v1.17.2
    hooks: [{ id: jupytext, args: [--sync] }]

Check the latest rev values with pre-commit autoupdate (needs network). Keep outputs only for notebooks that are deliverables (a rendered report), and say so in the README.

Parameterised and scheduled runs

Tag one cell parameters (cell metadata "tags": ["parameters"]) holding defaults:

# parameters
start_date = "2025-01-01"
region = "EU"

papermill report.ipynb runs/report-EU-2025-10.ipynb -p start_date 2025-10-01 -p region EU

Papermill injects a new cell after the tagged one and saves the executed copy, which is the audit record of that run. For anything that runs daily in production, refactor to a module and call it from the orchestrator; a notebook in a scheduler is a stopgap.

Refactor notebook code into modules

Move code out when a function is reused across notebooks, is needed by a pipeline or app, or is longer than about 20 lines of logic.

- [ ] 1. Execute the notebook from a clean kernel and save outputs as the baseline
- [ ] 2. Identify cells that are pure transformations (inputs -> outputs, no display)
- [ ] 3. Move each into src/<package>/<module>.py as a function with typed arguments
- [ ] 4. Replace globals with parameters; return values instead of mutating outer frames
- [ ] 5. Write a pytest test per function using a tiny fixture DataFrame
- [ ] 6. Import the functions back into the notebook; keep cells for narrative and plots
- [ ] 7. Re-execute and compare key outputs with the baseline

# first cell during refactoring, so module edits apply without restarting
%load_ext autoreload
%autoreload 2

from mypkg.features import add_session_features

# src/mypkg/features.py
import pandas as pd

def add_session_features(events: pd.DataFrame, gap_minutes: int = 30) -> pd.DataFrame:
    """Assign session ids per user where gaps exceed gap_minutes."""
    events = events.sort_values(["user_id", "ts"])
    gap = events.groupby("user_id")["ts"].diff() > pd.Timedelta(minutes=gap_minutes)
    return events.assign(session_id=gap.groupby(events["user_id"]).cumsum())

Install the package in editable mode (uv pip install -e . or pip install -e .) so the notebook imports the same code the tests run. Never sys.path.append("..") in a cell.

Testing notebooks

• Smoke test: execute each notebook in CI with nbconvert or papermill on a small sample dataset (an environment variable or parameter selects it). Failing execution fails CI.
• pytest --nbval-lax notebooks/ runs notebooks as tests and only checks that cells do not raise; strict --nbval compares outputs and is brittle for plots and timings.
• Assertions on results belong in cells (assert df["revenue"].ge(0).all()) or, better, in the extracted module’s unit tests.

Gotchas

• display() of a 10 million row DataFrame or a large plot in a loop makes the file hundreds of MB. Show head() and aggregates; save big artifacts to files.
• Secrets pasted into cells persist in outputs and git history even after deleting the cell. Read them from environment variables; if one was committed, tell the user it must be rotated.
• %cd and os.chdir in a cell change paths for every later cell and every rerun.
• A notebook that passes with “Run All” in the UI may still rely on an old kernel’s imports. Only a fresh kernel (--execute) counts.
• Widgets and interactive plots do not render in CI or static HTML exports; provide a static fallback for anything in a report.

Verification

jupyter nbconvert --to notebook --execute analysis.ipynb --output /tmp/check.ipynb
pytest -q tests/                          # extracted module tests
git diff --stat                           # .ipynb diffs small after nbstripout

Compare two or three headline numbers in the executed copy with the baseline. If they differ, find the cell where they diverge before declaring the refactor done.

Report template

**Notebook:** notebooks/churn_analysis.ipynb
**Clean-kernel run:** passes in 2m 10s (kernel: churn (uv), Python 3.12)
**Changes:** imports consolidated; 3 out-of-order cells moved; seed set; paths made repo-relative
**Extracted:** src/churn/features.py (4 functions, 9 tests), src/churn/io.py (2 functions)
**Repo hygiene:** nbstripout + jupytext pairing via pre-commit; paired file notebooks/churn_analysis.py
**Results check:** churn rate 4.82% and AUC 0.771 match the baseline run
**Follow-up:** daily run should move from papermill cron to the Airflow DAG