codexskills.dev

A/B Test Analysis skill for Codex

SRM check first, intervals over p-values. Codex checks an experiment's health before its result: SRM, cross-arm contamination and analysis unit come first. It then picks the right test for each metric, reports effects as confidence intervals, corrects for multiple metrics, refuses to bless peeked results, and ends with a ship recommendation. A standard-library script handles SRM, two-proportion tests and sample size offline.

  • Data & Analytics
  • $ab-test-analysis
  • 174 lines
  • 3 files
  • ~2,000 tokens
  • MIT
  • Updated
  • Editor's pick

Install

curl -fsSL codexskills.dev/install.sh \  | sh -s -- ab-test-analysis

Installs to ~/.agents/skills/ab-test-analysis/ for every project on this machine.

Download ab-test-analysis.zip

104-line POSIX script. No sudo, no telemetry. Read install.sh

Use it in Codex

Start a prompt with $ab-test-analysis, or pick it from /skills. You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.

What Codex reads before loading it

Analyses randomised A/B and multivariate experiments correctly - sample ratio mismatch (SRM) check first, then two-proportion z-tests, Welch t-tests, delta-method ratio metrics, CUPED variance reduction, confidence intervals, power and minimum detectable effect, guardrail metrics and multiple-testing control. Use when reading out an experiment, a test result looks too good, a team wants to stop early, designing sample size for a planned test, or reviewing an experiment readout before a ship decision.

Use it when

  • An experiment has finished and needs a readout and recommendation
  • Results look too good, or the split between arms looks uneven
  • Planning sample size, MDE and duration for a new test
  • Reviewing someone else's experiment analysis before a ship decision

Use something else when

  • You are designing variants, copy and hypotheses for a marketing test: use ab-testing
  • You are setting up product event tracking for experiments: use analytics-tracking-plan

How to check the result

Arm totals recomputed from raw exposure and event tables match the platform, the SRM test passes, and an A/A run through the same code shows no effect.

The mistake it prevents

Stopping the test the first day the p-value dips below 0.05, which makes the reported significance meaningless.

~/.agents/skills/ab-test-analysis/SKILL.md

--- name: ab-test-analysis description: Analyses randomised A/B and multivariate experiments correctly - sample ratio mismatch (SRM) check first, then two-proportion z-tests, Welch t-tests,… license: MIT ---

A trustworthy readout checks the experiment’s health before its result, reports the effect as an interval rather than a p-value alone, analyses only the metrics and window that were planned, and states the ship recommendation with the risk it carries. If the assignment is broken, there is no result to report.

Before computing anything

  1. Find the plan: hypothesis, primary metric, guardrails, unit of randomisation (user, session, account), planned sample size or duration, and alpha. If none was written, say so in the report; post-hoc choices weaken every conclusion.
  2. Confirm the analysis unit matches the randomisation unit. Randomised by user but analysed per session or pageview means observations are correlated and naive standard errors are too small; aggregate to the user, or use the delta method.
  3. Read AGENTS.md and the data source (warehouse table, PostHog, Statsig, GrowthBook, Optimizely export). Use the platform’s exposure table, not a reconstruction.
  4. Use only users exposed to the experiment (triggered), from exposure time onward.

Workflow

- [ ] 1. SRM check on assignment counts
- [ ] 2. Pre-period balance and exposure sanity (dates, platforms, duplicates across arms)
- [ ] 3. Primary metric: effect, CI, p-value at the planned horizon
- [ ] 4. Guardrails: did anything get worse beyond its threshold?
- [ ] 5. Secondary metrics and pre-registered segments, corrected for multiplicity
- [ ] 6. Practical significance vs the MDE and cost of shipping
- [ ] 7. Recommendation with caveats

Step 1: sample ratio mismatch

python3 scripts/ab_stats.py srm --observed 50900 49100              # 50/50 split expected
python3 scripts/ab_stats.py srm --observed 33100 33500 33400 --expected 1 1 1

The script runs a chi-square goodness-of-fit test and exits 1 when p < 0.001. On SRM, stop and investigate before reading any metric: bot filtering applied unevenly, redirects dropping users in one arm, assignment logged after a crash-prone code path, caching, or users counted in both arms. Report SRM as the headline finding.

Also check: users appearing in more than one arm (exclude them and report the count) and exposure starting on the same date in all arms.

Step 3: choose the test

MetricTestNotes
Conversion (user converted yes/no)Two-proportion z-testscripts/ab_stats.py proportions
Mean per user (revenue, sessions)Welch’s t-testHeavy tails: winsorise at p99.9 pre-registered, or bootstrap
Ratio of sums (CTR = clicks/views, AOV = revenue/orders)Delta method or bootstrap by userDenominator varies per user; a plain t-test on per-event rows is wrong
Count with many zerosWelch t on per-user valuesFine at large n; bootstrap for small n
Time-to-eventSurvival analysisOut of scope here; ask for a statistician
python3 scripts/ab_stats.py proportions --control 1200 50000 --treatment 1290 50100

The script prints both rates, absolute difference with CI, relative lift with a delta-method CI, z and two-sided p. It uses only the standard library, so it runs in a sandbox without installs.

For continuous metrics in Python (needs scipy, which may require an install):

import numpy as np
from scipy import stats

def welch(control: np.ndarray, treatment: np.ndarray, alpha: float = 0.05):
    res = stats.ttest_ind(treatment, control, equal_var=False)
    diff = treatment.mean() - control.mean()
    se = np.sqrt(treatment.var(ddof=1) / len(treatment) + control.var(ddof=1) / len(control))
    z = stats.norm.ppf(1 - alpha / 2)
    return diff, (diff - z * se, diff + z * se), res.pvalue

Delta method for a ratio metric R = sum(Y)/sum(X), per user i with n users in an arm:

def ratio_var(y: np.ndarray, x: np.ndarray) -> float:
    n = len(y)
    my, mx = y.mean(), x.mean()
    vy, vx = y.var(ddof=1), x.var(ddof=1)
    cov = np.cov(y, x, ddof=1)[0, 1]
    return (vy / mx**2 - 2 * my * cov / mx**3 + my**2 * vx / mx**4) / n

The difference between arms then has variance ratio_var(t) + ratio_var(c).

CUPED

When a pre-experiment covariate exists (same metric in the 2-4 weeks before exposure), CUPED cuts variance, often by 20-50% for engagement metrics:

theta = np.cov(y_all, x_pre_all, ddof=1)[0, 1] / x_pre_all.var(ddof=1)   # pooled across arms
y_adj = y - theta * (x_pre - x_pre_all.mean())

Run the same test on y_adj. Estimate theta on both arms pooled, and use a covariate measured strictly before exposure, otherwise the treatment leaks into the adjustment.

Power and duration (design time)

python3 scripts/ab_stats.py sample-size --baseline 0.04 --mde 0.05 --relative
  • Choose the MDE from what is worth shipping, not from what the traffic allows.
  • Run whole weeks (at least 1-2 full weekly cycles) to cover day-of-week effects.
  • If the required sample exceeds available traffic, say so: options are a bigger MDE, a more sensitive metric, CUPED, or not running the test.

Hard rules

  • No peeking. Looking at p-values daily and stopping at the first p < 0.05 inflates false positives several-fold. Analyse at the planned horizon, or use a method built for continuous monitoring (sequential tests, always-valid p-values) that the platform supports. If someone stopped early, say the p-value is not valid as reported.
  • Multiple metrics and segments. Primary metric at full alpha. For secondary metrics apply Holm or Benjamini-Hochberg; label unplanned segment results “exploratory”.
  • Novelty and primacy. Plot the daily treatment effect; an effect that decays over the first week will not persist.
  • Non-significant is not “no effect”. Report the CI: “between -0.4% and +1.1%” tells the reader whether a meaningful effect is ruled out.
  • Interference. Marketplaces, social features and shared budgets break the independence assumption; flag it and suggest cluster or switchback designs.
  • Do not give personalised investment or financial advice from revenue experiments; the readout is about the product decision.

Interpretation table

Primary resultGuardrailsRecommendation
CI excludes 0, effect >= MDEAll within thresholdsShip
CI excludes 0, effect < MDEFineShip if cost is low; effect smaller than planned
CI includes 0, narrow around 0FineNo meaningful effect; decide on other grounds
CI includes 0, wideAnyUnderpowered; extend or redesign, do not call it neutral
AnyA guardrail degraded beyond thresholdDo not ship without an explicit trade-off decision
SRM detected-No conclusion; fix assignment and rerun

Verification

  • Recompute the primary metric’s arm totals from raw exposure and event tables and match the platform’s numbers within rounding; a mismatch means the metric definitions differ.
  • Run an A/A comparison (or the pre-period) through the same code: it should show no effect in about 95% of reruns or splits.
  • Confirm the analysis window and filters match the plan exactly.

Readout template

# Experiment: <name> (<start> to <end>, <n> days)
**Hypothesis:** <change> increases <primary metric>
**Unit:** user, 50/50, triggered on checkout page view
**Health:** SRM p = 0.41 (ok); 112 users in both arms excluded; no exposure gaps

| Metric | Control | Treatment | Diff (95% CI) | p |
|---|---|---|---|---|
| Checkout conversion (primary) | 2.400% | 2.575% | +0.17 pts (-0.02, +0.37) | 0.076 |
| Revenue per user (guardrail) | 1.82 | 1.84 | +0.02 (-0.05, +0.09) | 0.58 |

**Interpretation:** effect is positive but the CI includes zero; the test reached 65% of its planned sample.
**Recommendation:** extend to the planned 2 weeks; do not ship on this readout.
**Caveats:** novelty effect visible days 1-3; mobile segment result is exploratory (not pre-registered).

What the zip contains

Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.

agents/openai.yamlgenerated from this page

Gives the Codex app a display name, a one-line description and a starter prompt for this skill.

interface:
  display_name: "A/B Test Analysis"
  short_description: "SRM check first, intervals over p-values"
  default_prompt: "$ab-test-analysis Read out this checkout experiment and tell me whether we should ship"
scripts/ab_stats.py152 lines, 6.0 KB 
#!/usr/bin/env python3
"""Standard-library A/B test helpers: SRM check, two-proportion test, sample size.

Usage:
  python3 ab_stats.py srm --observed 50310 49690 [--expected 0.5 0.5]
  python3 ab_stats.py proportions --control 1200 50000 --treatment 1290 50100 [--alpha 0.05]
  python3 ab_stats.py sample-size --baseline 0.04 --mde 0.05 --relative [--alpha 0.05] [--power 0.8]

Exit codes: 0 ok, 1 SRM detected (srm subcommand), 2 bad input.
"""
from __future__ import annotations

import argparse
import math
import sys
from statistics import NormalDist

N = NormalDist()


def _gamma_q(s: float, x: float) -> float:
    """Regularised upper incomplete gamma Q(s, x)."""
    if x <= 0:
        return 1.0
    if x < s + 1:  # series for P, then Q = 1 - P
        term = total = 1.0 / s
        a = s
        for _ in range(10_000):
            a += 1
            term *= x / a
            total += term
            if abs(term) < abs(total) * 1e-15:
                break
        return max(0.0, 1.0 - total * math.exp(-x + s * math.log(x) - math.lgamma(s)))
    # continued fraction (modified Lentz)
    tiny = 1e-300
    b = x + 1 - s
    c = 1 / tiny
    d = 1 / b
    h = d
    for i in range(1, 10_000):
        an = -i * (i - s)
        b += 2
        d = an * d + b
        d = tiny if abs(d) < tiny else d
        c = b + an / c
        c = tiny if abs(c) < tiny else c
        d = 1 / d
        delta = d * c
        h *= delta
        if abs(delta - 1) < 1e-15:
            break
    return math.exp(-x + s * math.log(x) - math.lgamma(s)) * h


def chi2_sf(stat: float, df: int) -> float:
    return _gamma_q(df / 2.0, stat / 2.0)


def cmd_srm(a: argparse.Namespace) -> int:
    obs = a.observed
    k = len(obs)
    exp_w = a.expected or [1.0 / k] * k
    if len(exp_w) != k or k < 2:
        print("error: need >= 2 observed counts and matching expected shares", file=sys.stderr)
        return 2
    total = sum(obs)
    wsum = sum(exp_w)
    expected = [total * w / wsum for w in exp_w]
    stat = sum((o - e) ** 2 / e for o, e in zip(obs, expected))
    p = chi2_sf(stat, k - 1)
    for i, (o, e) in enumerate(zip(obs, expected)):
        print(f"arm {i}: observed {o:>12,}  expected {e:>14,.1f}  share {o / total:.4%}")
    print(f"chi2 = {stat:.3f}, df = {k - 1}, p = {p:.3g}")
    if p < a.threshold:
        print(f"SRM DETECTED (p < {a.threshold}): do not trust effect estimates until explained")
        return 1
    print(f"no SRM at threshold {a.threshold}")
    return 0


def cmd_proportions(a: argparse.Namespace) -> int:
    xc, nc = a.control
    xt, nt = a.treatment
    if min(nc, nt) <= 0 or not (0 <= xc <= nc and 0 <= xt <= nt):
        print("error: counts must satisfy 0 <= conversions <= users, users > 0", file=sys.stderr)
        return 2
    pc, pt = xc / nc, xt / nt
    pooled = (xc + xt) / (nc + nt)
    se_pooled = math.sqrt(pooled * (1 - pooled) * (1 / nc + 1 / nt))
    z = (pt - pc) / se_pooled if se_pooled > 0 else 0.0
    p_value = 2 * (1 - N.cdf(abs(z)))
    se_diff = math.sqrt(pc * (1 - pc) / nc + pt * (1 - pt) / nt)
    zcrit = N.inv_cdf(1 - a.alpha / 2)
    lo, hi = (pt - pc) - zcrit * se_diff, (pt - pc) + zcrit * se_diff
    print(f"control   {xc:,}/{nc:,} = {pc:.4%}")
    print(f"treatment {xt:,}/{nt:,} = {pt:.4%}")
    print(f"absolute diff = {pt - pc:+.4%}  {100 * (1 - a.alpha):.0f}% CI [{lo:+.4%}, {hi:+.4%}]")
    if pc > 0:
        # delta-method CI on relative lift (log ratio)
        se_log = math.sqrt((1 - pc) / (pc * nc) + (1 - pt) / (pt * nt)) if pt > 0 else float("nan")
        ratio = pt / pc
        rlo = math.exp(math.log(ratio) - zcrit * se_log) - 1 if pt > 0 else float("nan")
        rhi = math.exp(math.log(ratio) + zcrit * se_log) - 1 if pt > 0 else float("nan")
        print(f"relative lift = {ratio - 1:+.2%}  CI [{rlo:+.2%}, {rhi:+.2%}]")
    print(f"z = {z:.3f}, two-sided p = {p_value:.4g}, alpha = {a.alpha}")
    return 0


def cmd_sample_size(a: argparse.Namespace) -> int:
    p1 = a.baseline
    p2 = p1 * (1 + a.mde) if a.relative else p1 + a.mde
    if not (0 < p1 < 1 and 0 < p2 < 1):
        print("error: baseline and baseline+effect must be in (0, 1)", file=sys.stderr)
        return 2
    za = N.inv_cdf(1 - a.alpha / 2)
    zb = N.inv_cdf(a.power)
    pbar = (p1 + p2) / 2
    n = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
    n = math.ceil(n)
    print(f"baseline {p1:.4%} -> {p2:.4%} (absolute {p2 - p1:+.4%}, relative {p2 / p1 - 1:+.2%})")
    print(f"alpha {a.alpha} two-sided, power {a.power}")
    print(f"users per arm: {n:,}  (total for 2 arms: {2 * n:,})")
    return 0


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = ap.add_subparsers(dest="cmd", required=True)
    s = sub.add_parser("srm", help="sample ratio mismatch chi-square test")
    s.add_argument("--observed", type=int, nargs="+", required=True)
    s.add_argument("--expected", type=float, nargs="+", help="expected shares, default equal")
    s.add_argument("--threshold", type=float, default=0.001)
    s.set_defaults(fn=cmd_srm)
    p = sub.add_parser("proportions", help="two-proportion z-test with CIs")
    p.add_argument("--control", type=int, nargs=2, metavar=("CONVERSIONS", "USERS"), required=True)
    p.add_argument("--treatment", type=int, nargs=2, metavar=("CONVERSIONS", "USERS"), required=True)
    p.add_argument("--alpha", type=float, default=0.05)
    p.set_defaults(fn=cmd_proportions)
    z = sub.add_parser("sample-size", help="users per arm for a conversion metric")
    z.add_argument("--baseline", type=float, required=True)
    z.add_argument("--mde", type=float, required=True, help="minimum detectable effect")
    z.add_argument("--relative", action="store_true", help="interpret mde as relative lift")
    z.add_argument("--alpha", type=float, default=0.05)
    z.add_argument("--power", type=float, default=0.8)
    z.set_defaults(fn=cmd_sample_size)
    args = ap.parse_args()
    return args.fn(args)


if __name__ == "__main__":
    sys.exit(main())

About this skill

Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.

Related skills

All 30 Data & Analytics skills
esc