Feature Engineering skill for Codex

Done well means: every feature value in the training set is exactly what the system could have computed at that row’s prediction time, the same code (or the same definition) produces features for training and serving, and each feature earns its place by a measured gain on validation.

Start with the prediction contract

Write this down before creating any feature:

- Entity: customer_id            - Prediction time: order checkout (event_ts)
- Label: chargeback within 60 days of event_ts
- Data available at prediction time: profile, past orders, past sessions (lag <= 5 min)
- Not available: anything after event_ts, chargeback outcomes newer than 60 days before event_ts

Read AGENTS.md, the existing feature code, warehouse schemas and any feature store config. Note how fresh each source really is in production (batch nightly vs streaming).

Point-in-time correctness

The most common and most expensive bug: joining a snapshot table (current customer state) onto historical training rows. Every join must be “latest value as of the row’s timestamp, minus the source’s latency”.

import pandas as pd

labels = labels.sort_values("event_ts")
balances = balances.sort_values("updated_at")

train = pd.merge_asof(
    labels, balances,
    left_on="event_ts", right_on="updated_at",
    by="customer_id",
    direction="backward",                       # only past values
    tolerance=pd.Timedelta("30D"),              # stale beyond this -> NaN
    allow_exact_matches=False,                  # same-timestamp updates may be post-event
)

-- As-of join in SQL (DuckDB/Snowflake support ASOF JOIN; otherwise use a window)
SELECT l.*, b.balance
FROM labels l
ASOF LEFT JOIN balances b
  ON l.customer_id = b.customer_id AND l.event_ts > b.updated_at;

For aggregates, compute windows that end strictly before the prediction time:

orders = orders.sort_values("order_ts").set_index("order_ts")
roll = (orders.groupby("customer_id")["amount"]
        .rolling("30D", closed="left")          # excludes the current row
        .agg(["sum", "count"]).reset_index())

Feature menu by data type

Raw data	Features that usually help	Watch out
Numeric	Raw value, log1p for heavy tails, ratios (debt/income), clipped values	Tree models do not need scaling; linear models do
Categorical, low cardinality (< 30)	One-hot	Unseen categories at serving: handle_unknown
Categorical, high cardinality (IDs, zip codes)	Target encoding with cross-fitting, frequency/count encoding, hashing	Plain target mean on full data leaks the label
Timestamps	Hour, day of week, month, holiday flag, time since last event, age of account	Time zone of the user, not the server
Event logs	Counts, sums and distinct counts over 1h/1d/7d/30d windows, recency, trend (7d vs 30d)	Window must close before prediction time
Text	Length, keyword flags, TF-IDF, or embeddings ($openai-embeddings)	Embedding model versions must match train and serve
Missing values	Missing indicator columns (missingness is often signal)	Imputation statistics fitted on train folds only
Geography	Distance to store, region aggregates, geohash buckets	Region aggregates computed with the row’s own label leak

Prefer a few strong, explainable features over hundreds of generated ones; add aggregation windows in a grid only when validation shows gains.

Encoding rules

from sklearn.preprocessing import TargetEncoder, OneHotEncoder

# TargetEncoder.fit_transform cross-fits internally (no row sees its own label);
# keep it inside the model Pipeline so CV refits it per fold.
te = TargetEncoder(target_type="binary", smooth="auto", cv=5, random_state=0)

• Rare categories: group below a minimum count (min_frequency= on OneHotEncoder) into “other” so the model is not fitting noise.
• Hashing (FeatureHasher) suits huge, open-ended vocabularies; collisions are acceptable noise, but the feature is not interpretable.
• Keep encoders and imputers in the model pipeline ($scikit-learn-pipelines); the feature layer produces raw, point-in-time-correct columns.

Train/serve parity

Approach	When
Same Python function used in batch training and online serving	Small teams, one service
SQL/dbt definitions materialized to offline and online stores	Warehouse-centric stacks
Feature store (Feast, Tecton, Databricks, Vertex)	Many models share features; online low-latency lookup needed

Feast in brief:

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64

customer = Entity(name="customer", join_keys=["customer_id"])
stats = FeatureView(
    name="customer_30d_stats", entities=[customer], ttl=timedelta(days=30),
    schema=[Field(name="orders_30d", dtype=Int64), Field(name="spend_30d", dtype=Float32)],
    source=FileSource(path="data/customer_stats.parquet", timestamp_field="event_ts"),
)

feast apply                                        # register definitions
feast materialize-incremental $(date -u +%Y-%m-%dT%H:%M:%S)   # load online store

Training sets come from store.get_historical_features(entity_df=labels, features=[...]) (point-in-time join by event_timestamp), serving from store.get_online_features(...). feast apply and materialization change shared infrastructure: confirm the target environment with the user first.

Measuring a feature’s value

1. Baseline model on the current feature set, fixed CV splits, 3 seeds.
2. Add one feature group; rerun; keep it if the gain exceeds the seed/fold spread.
3. Check permutation importance on validation and ablation (drop the group) for the final set.
4. Inspect the top features’ definitions for leakage; a suspiciously dominant feature is usually computed with future information.

Skew and drift checks

• Log served feature values for a sample of requests; compare distributions with the training set (population stability index, KS test, or simple quantile tables).
• Recompute features offline for logged requests and diff against what was served: any mismatch is a parity bug, not drift.
• Track null rates per feature in production; a jump usually means an upstream break.

Gotchas

• groupby().transform("mean") over the whole dataset includes future rows and the row’s own label-correlated values.
• Data refreshed late in production but on time in the warehouse makes training features fresher than serving features; simulate the real lag in the as-of join.
• Deleting or backfilling source rows changes historical features; snapshot training sets and record their version.
• Time zones: compute day-of-week and hour in the user’s local time.

Output contract

| Feature | Definition (as of prediction time) | Source / lag | Gain (val AP) |
|---|---|---|---|
| orders_30d | count of orders in [t-30d, t) | orders, 5 min | +0.021 |
| days_since_signup | t - signup_ts in days | accounts, nightly | +0.008 |
| merchant_te | cross-fitted target encoding of merchant_id | in model pipeline | +0.015 |

Parity check: 1,000 served requests recomputed offline, 0 mismatches.