scikit-learn Pipelines skill for Codex

Done well means: every transformation that learns from data lives inside one Pipeline fitted only on training folds, the CV splitter mirrors how the model will meet new data, a dumb baseline is beaten by a margin larger than fold-to-fold noise, and the test set is touched exactly once.

Before modelling

1. Read AGENTS.md, the data loading code and sklearn.__version__ (APIs below assume 1.4+).
2. Define the prediction moment: what is known at the time the model is used? Columns computed after that moment (refund flags for a fraud model at checkout) are leakage.
3. Identify grouping (customer, patient, device) and time ordering. These decide the splitter.
4. Hold out a test set now; do not look at it again until the final report.

Choose the splitter

Data	Splitter
i.i.d. rows, classification	StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
Several rows per entity (user, patient, store)	GroupKFold or StratifiedGroupKFold with groups=
Ordered in time, predicting the future	TimeSeriesSplit(n_splits=5, gap=...); or a single forward split
Very small data (< 500 rows)	RepeatedStratifiedKFold for stable estimates

The holdout split must follow the same rule (group-aware or time-based), not just CV.

Reference pipeline

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, make_column_selector as sel
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X, y = df.drop(columns=["churned"]), df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)

numeric = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                    ("scale", StandardScaler())])
categorical = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                        ("onehot", OneHotEncoder(handle_unknown="infrequent_if_exist", min_frequency=20))])
pre = ColumnTransformer([("num", numeric, sel(dtype_include="number")),
                         ("cat", categorical, sel(dtype_include=["object", "category"]))])

linear = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=2000, class_weight="balanced"))])
hgb = HistGradientBoostingClassifier(categorical_features="from_dtype", early_stopping=True, random_state=0)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scoring = {"auc": "roc_auc", "ap": "average_precision", "logloss": "neg_log_loss"}
for name, model in {"dummy": DummyClassifier(strategy="prior"), "linear": linear, "hgb": hgb}.items():
    r = cross_validate(model, X_train, y_train, cv=cv, scoring=scoring, n_jobs=-1)
    print(name, {k: f"{r['test_' + k].mean():.3f} +/- {r['test_' + k].std():.3f}" for k in scoring})

• HistGradientBoosting handles NaN natively and, with categorical_features="from_dtype", pandas category columns without one-hot. It is the strong default for tabular data; cast string columns with X[c].astype("category") first.
• pipe.set_output(transform="pandas") keeps column names through transforms for debugging.

Leakage checklist

- [ ] No fit/fit_transform on the full dataset before splitting (scalers, imputers, encoders, PCA, feature selection, SMOTE)
- [ ] Target encoding inside the pipeline (TargetEncoder cross-fits during fit_transform)
- [ ] No feature derived from the label or from post-outcome events
- [ ] Duplicates and near-duplicates do not straddle train and validation
- [ ] Group or time structure respected by every split
- [ ] Hyperparameters chosen on CV folds, never on the test set

Smell test: validation AUC above 0.95 on a messy business problem, or one feature with enormous importance, means investigate leakage before celebrating. Check permutation importance on validation data, then read how the top feature is produced.

Hyperparameter search

from scipy.stats import loguniform, randint
from sklearn.model_selection import RandomizedSearchCV

search = RandomizedSearchCV(
    hgb,
    {"learning_rate": loguniform(0.01, 0.3), "max_leaf_nodes": randint(15, 127),
     "min_samples_leaf": randint(10, 200), "l2_regularization": loguniform(1e-4, 10)},
    n_iter=40, cv=cv, scoring="average_precision", n_jobs=-1, random_state=0, refit=True,
)
search.fit(X_train, y_train)

Name pipeline params with double underscores (clf__C). Prefer randomized or HalvingRandomSearchCV over exhaustive grids above three parameters. If tuning and model selection both use the same CV, the best score is optimistic; nested CV gives an honest estimate.

Imbalance, probabilities, thresholds

Need	Tool
Rare positive class	Score with average_precision; class_weight="balanced" before any resampling
Calibrated probabilities (risk scores, expected cost)	CalibratedClassifierCV(model, method="isotonic", cv=5) (sigmoid when data is small)
A decision threshold other than 0.5	TunedThresholdClassifierCV(model, scoring="f1") or a cost-based scorer (1.5+)
Resampling (SMOTE)	Only inside an imblearn.pipeline.Pipeline so it runs on training folds only

Check calibration with CalibrationDisplay.from_estimator and Brier score on validation.

Final evaluation and persistence

from sklearn.metrics import classification_report

final = search.best_estimator_
print(classification_report(y_test, final.predict(X_test)))   # once, at the end

import joblib, sklearn
joblib.dump({"model": final, "sklearn": sklearn.__version__, "features": list(X.columns)}, "model.joblib")

• Pickle/joblib files execute code on load: load only files you created. For sharing across trust boundaries, use skops.io or ONNX (skl2onnx), see $ml-model-deployment.
• Load with the same scikit-learn version; cross-version unpickling is unsupported.
• Validate the input schema at inference (column names, dtypes) before predict.

Gotchas

• cross_val_score on an unfitted pipeline is correct; on pre-transformed X it leaks.
• OneHotEncoder(handle_unknown="error") crashes in production on a new category.
• LabelEncoder is for targets, not features; use OrdinalEncoder for features.
• StandardScaler on sparse one-hot output needs with_mean=False.
• Reporting accuracy on a 95/5 dataset hides a useless model; always show the dummy baseline.

Report template

**Task:** churn in 90 days | **Rows:** 48k train / 12k test | **Split:** StratifiedGroupKFold by account
**Baseline:** dummy AP 0.08 | linear AP 0.41 +/- 0.02 | HGB tuned AP 0.53 +/- 0.02
**Test (once):** AP 0.52, AUC 0.86; threshold 0.31 tuned for F1 on CV
**Leakage review:** dropped `cancellation_reason` (post-outcome); top features plausible
**Artifact:** model.joblib (sklearn 1.x), input schema checked at load

For building and versioning the features themselves, use $feature-engineering.