codexskills.dev

A/B Testing skill for Codex

Size it, check SRM, then trust the readout. Codex plans experiments before touching code: one primary metric, guardrails, a decision rule and a sample size computed from baseline and minimum detectable effect. It wires deterministic assignment and exposure logging into the flag tool you already use, runs a bundled offline calculator for duration, sample ratio mismatch and significance, and writes a readout with confidence intervals instead of a bare winner.

  • Growth & SEO
  • $ab-testing
  • 188 lines
  • 3 files
  • ~2,150 tokens
  • MIT
  • Updated

Install

curl -fsSL codexskills.dev/install.sh \  | sh -s -- ab-testing

Installs to ~/.agents/skills/ab-testing/ for every project on this machine.

Download ab-testing.zip

104-line POSIX script. No sudo, no telemetry. Read install.sh

Use it in Codex

Start a prompt with $ab-testing, or pick it from /skills. You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.

What Codex reads before loading it

Designs, implements and analyses A/B and multivariate experiments on websites and apps - hypothesis, primary and guardrail metrics, sample size from baseline and minimum detectable effect, deterministic assignment, exposure logging, sample ratio mismatch checks and an honest readout. Works with GrowthBook, Statsig, Optimizely, PostHog, LaunchDarkly or a home-grown flag. Use when asked to set up an experiment, size a test, decide how long to run it, check whether a result is significant, or explain why two variants show different numbers.

Use it when

  • You are planning an experiment and need a hypothesis, metrics and a sample size
  • An experiment needs assignment and exposure tracking implemented in code
  • A finished test needs a significance check and a written readout
  • Arm sizes look uneven and you suspect a sample ratio mismatch

Use something else when

How to check the result

Final arm counts pass the SRM check (p of at least 0.001) and the primary metric recomputed from raw events matches the platform's figure.

The mistake it prevents

Stopping a fixed-horizon test the first day it shows p below 0.05; repeated peeking multiplies false positives.

~/.agents/skills/ab-testing/SKILL.md

--- name: ab-testing description: Designs, implements and analyses A/B and multivariate experiments on websites and apps - hypothesis, primary and guardrail metrics, sample size from… license: MIT ---

A test is done well when the decision rule was written before launch, the sample size came from a calculation rather than a deadline, assignment was checked for sample ratio mismatch, and the readout reports an interval and the guardrails, not just “variant B won”.

Before writing any code

  1. Read AGENTS.md and find what the project already uses for flags or experiments: rg -n "growthbook|statsig|optimizely|posthog|launchdarkly|unleash|getFeatureFlag|useFeatureIsOn" -g '!node_modules'. Reuse it. A second experimentation system in one product creates conflicting assignments.
  2. Find where the primary metric is already tracked (analytics events, a warehouse table). If the metric is not logged today, the first task is instrumentation, not the test.
  3. Ask for, or estimate from data, the baseline rate and the daily eligible traffic of the page or flow under test. Without both, no duration can be promised.

Write the experiment brief first

## Experiment: <short-slug>   (flag key: exp_<slug>_<yyyymm>)
**Hypothesis:** Because <evidence>, changing <element> for <audience> will increase
<primary metric> by at least <MDE>.
**Primary metric:** <one metric, unit = user, e.g. checkout completion per assigned user>
**Guardrails:** <metrics that must not drop: revenue per user, refund rate, page errors, LCP>
**Secondary (exploratory only):** <...>
**Audience and trigger:** <who is eligible; the event that counts as exposure>
**Split:** 50/50   **Alpha:** 0.05 two-sided   **Power:** 0.8
**Sample size:** <n per arm> -> <days> (whole weeks)
**Decision rule:** ship if primary CI excludes 0 and no guardrail degrades beyond <x>;
otherwise keep control. Inconclusive = keep control and log the learning.

One primary metric. Every extra metric read as “the” result raises the false-positive rate.

Sample size and duration

Use the bundled calculator (standard library only, works offline):

python3 scripts/ab_calc.py size --baseline 0.042 --mde 0.10 --daily 6000
python3 scripts/ab_calc.py size --baseline 0.042 --mde 0.005 --absolute --arms 3
  • --mde is relative by default (0.10 = a 10% lift on the baseline).
  • With more than two arms it applies a Bonferroni correction across treatment-vs-control comparisons, so each extra arm costs traffic.
  • It rounds duration up to whole weeks to cover weekday and weekend behaviour.
SituationDecision
Required duration > 6-8 weeksRaise the MDE, pick a higher-traffic page, or use a metric closer to the change (click-through instead of revenue)
Duration < 7 daysStill run 7 days minimum; weekly cycles bias shorter tests
Baseline under 1% and low trafficDo not A/B test; ship behind a flag and monitor, or run qualitative research
Need to stop early when results are clearUse a sequential method your platform supports (GrowthBook and Statsig both offer sequential testing) and decide that before launch

Assignment and exposure

Assignment must be deterministic per unit and logged at the moment the user actually sees the difference.

// Deterministic bucketing when no platform SDK exists (Node 20+).
import { createHash } from "node:crypto";

export function bucket(experimentKey: string, unitId: string, weights = [0.5, 0.5]): number {
  const h = createHash("sha256").update(`${experimentKey}:${unitId}`).digest();
  const x = h.readUInt32BE(0) / 2 ** 32; // uniform in [0, 1)
  let acc = 0;
  for (let i = 0; i < weights.length; i++) {
    acc += weights[i];
    if (x < acc) return i;
  }
  return weights.length - 1;
}

Rules:

  • Unit: user ID when logged in, a first-party stable ID otherwise. Session IDs inflate sample size and break independence for anything with repeat visits.
  • Salt with the experiment key so users are not in the same bucket in every test.
  • Exposure event fires when the variant renders, not when the flag is evaluated on the server for every request. Analyse only exposed users (triggered analysis).
  • Server-side or edge assignment for above-the-fold changes. Client-side swaps cause flicker, which is itself a treatment and biases results.
  • Never change weights mid-test (e.g. 10/90 to 50/50) without restarting; time-varying ratios confound the comparison (Simpson’s paradox).
  • Keep assignment stable across devices only if the unit is a login ID; say so in the brief.

GrowthBook example (React SDK, current API):

import { useFeatureValue } from "@growthbook/growthbook-react";

export function PricingCta() {
  const variant = useFeatureValue("exp_pricing_cta_202610", "control");
  return variant === "annual_first" ? <AnnualFirstCta /> : <DefaultCta />;
}

Wire the SDK’s trackingCallback to send experiment_viewed with experiment and variation IDs to the analytics tool; without it there is no exposure data to analyse. Installing an SDK needs network; in a sandbox with network off, ask for approval before npm install.

While the test runs

  • Check sample ratio mismatch on day 1-2 and again at the end:
python3 scripts/ab_calc.py srm 50211 48302            # 50/50 planned
python3 scripts/ab_calc.py srm 33000 33500 16700 --ratios 2 2 1

p < 0.001 means assignment or logging is broken (bots filtered on one side, redirects losing users, a crash in one variant). Stop, fix, restart. Never analyse an SRM test.

  • Watch guardrails (errors, latency, refunds) daily; a broken variant is stopped regardless of the primary metric.
  • Do not peek at the primary metric to decide when to stop on a fixed-horizon test. Ten peeks at alpha 0.05 push the real false-positive rate well above 20%.

Analysis

python3 scripts/ab_calc.py result 1210 25000 1325 25100   # conv_A n_A conv_B n_B
Metric typeMethod
Conversion (binary per user)Two-proportion z-test (the script), or the platform’s engine
Revenue / continuous per userWelch’s t-test on per-user values; cap or winsorise extreme outliers at a pre-set percentile
Ratio metrics (clicks per session)Delta method; naive per-session tests understate variance
Pre-period data availableCUPED variance reduction (Statsig, GrowthBook, Eppo support it)

Interpreting:

  • Report the absolute and relative effect with a confidence interval. “+9% (CI +1% to +17%)” tells a stakeholder far more than “p = 0.025”.
  • Significant but tiny lift with a CI touching zero-ish values: the true effect may be small; weigh against maintenance cost.
  • Not significant does not mean “no effect”; it means the test could not detect one of the size planned.
  • Segment cuts (device, country, new vs returning) are exploratory. A “winning segment” found after the fact needs its own follow-up test.
  • Novelty and primacy effects: compare week 1 against week 2+ for UI changes users notice.

Gotchas

  • Bots and internal traffic in one arm only (e.g. a variant page that a crawler fetches) create SRM. Filter by the same rule on both arms.
  • Redirect tests (split URL) lose users on the redirect and leak through caches; prefer rendering both variants at the same URL.
  • Caching: a CDN caching the first variant it saw serves everyone the same page. Vary the cache key on the bucket cookie or assign at the edge.
  • Overlapping experiments on the same surface interact. Use mutually exclusive layers or namespaces for tests touching the same element.
  • Ending a test does not remove its code. Ship the winner, delete the flag and the losing branch in the same PR (see feature-flags for cleanup).

Shipping

Turning a winner on for 100% of production traffic is a production change: show the readout and get explicit user confirmation before changing rollout in the platform or merging the cleanup PR.

Verification

  • Run the SRM check on final counts; p >= 0.001.
  • Confirm exposure counts in analytics match assignment counts within a few percent.
  • Recompute the primary metric from raw events (SQL or export) and compare with the platform’s number; a mismatch means the metric definition differs.

Readout template

## Result: exp_pricing_cta_202610 (ran 2026-09-01 to 2026-09-15, 14 days)
**Decision:** ship B | keep control | inconclusive (keep control)
**Primary:** checkout completion 4.84% -> 5.28%, +9.1% relative (95% CI +1.1% to +17.0%), p = 0.025
**Guardrails:** revenue/user +2.0% (CI -3% to +7%); refund rate flat; JS errors flat
**Sample:** 25,000 / 25,100 exposed; SRM p = 0.64
**Caveats:** mobile drove most of the lift (exploratory, not powered)
**Next:** remove flag and control branch; follow-up test on mobile-only copy

What the zip contains

Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.

agents/openai.yamlgenerated from this page

Gives the Codex app a display name, a one-line description and a starter prompt for this skill.

interface:
  display_name: "A/B Testing"
  short_description: "Size it, check SRM, then trust the readout"
  default_prompt: "$ab-testing How long do we need to run a test on the signup page if we want to detect a 10% lift?"
scripts/ab_calc.py165 lines, 6.7 KB 
#!/usr/bin/env python3
"""A/B test arithmetic with the standard library only.

Subcommands:
  size    sample size per arm for a conversion-rate test
  srm     sample ratio mismatch check (chi-square goodness of fit)
  result  two-proportion z-test with a confidence interval for the lift

Examples:
  python3 ab_calc.py size --baseline 0.042 --mde 0.10 --daily 6000
  python3 ab_calc.py srm 50211 49302
  python3 ab_calc.py srm 33000 33500 16700 --ratios 2 2 1
  python3 ab_calc.py result 1210 25000 1325 25100

Exit codes: 0 ok, 1 SRM detected (srm), 2 bad input.
"""
from __future__ import annotations

import argparse
import math
import sys
from statistics import NormalDist

Z = NormalDist()


def sample_size(p1: float, p2: float, alpha: float, power: float, two_sided: bool) -> int:
    """Per-arm n for a two-proportion test (pooled variance under H0)."""
    za = Z.inv_cdf(1 - alpha / 2) if two_sided else Z.inv_cdf(1 - alpha)
    zb = Z.inv_cdf(power)
    pbar = (p1 + p2) / 2
    num = za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil(num ** 2 / (p2 - p1) ** 2)


def gamma_q(a: float, x: float) -> float:
    """Regularized upper incomplete gamma Q(a, x) (Numerical Recipes style)."""
    if x <= 0:
        return 1.0
    gln = math.lgamma(a)
    if x < a + 1:  # series for P, return 1 - P
        ap, total, delta = a, 1.0 / a, 1.0 / a
        for _ in range(1000):
            ap += 1
            delta *= x / ap
            total += delta
            if abs(delta) < abs(total) * 1e-14:
                break
        return max(0.0, 1.0 - total * math.exp(-x + a * math.log(x) - gln))
    b = x + 1 - a  # continued fraction for Q
    c, d = 1.0 / 1e-300, 1.0 / b
    h = d
    for i in range(1, 1000):
        an = -i * (i - a)
        b += 2
        d = an * d + b
        d = 1e-300 if abs(d) < 1e-300 else d
        c = b + an / c
        c = 1e-300 if abs(c) < 1e-300 else c
        d = 1.0 / d
        delta = d * c
        h *= delta
        if abs(delta - 1) < 1e-14:
            break
    return math.exp(-x + a * math.log(x) - gln) * h


def cmd_size(a: argparse.Namespace) -> int:
    p1 = a.baseline
    p2 = p1 * (1 + a.mde) if not a.absolute else p1 + a.mde
    if not (0 < p1 < 1 and 0 < p2 < 1):
        print("error: baseline and baseline+effect must be between 0 and 1", file=sys.stderr)
        return 2
    comparisons = max(1, a.arms - 1)
    alpha = a.alpha / comparisons  # Bonferroni across treatment-vs-control comparisons
    n = sample_size(p1, p2, alpha, a.power, not a.one_sided)
    total = n * a.arms
    print(f"baseline {p1:.4%} -> target {p2:.4%} ({'absolute' if a.absolute else 'relative'} MDE {a.mde:g})")
    print(f"alpha {a.alpha} ({'Bonferroni ' + format(alpha, '.4f') if comparisons > 1 else 'single comparison'}), "
          f"power {a.power}, {'one' if a.one_sided else 'two'}-sided")
    print(f"per arm: {n:,}   total ({a.arms} arms): {total:,}")
    if a.daily:
        days = math.ceil(total / a.daily)
        weeks = max(1, math.ceil(days / 7))
        print(f"at {a.daily:,} eligible users/day: {days} days -> run {weeks * 7} days (whole weeks)")
    return 0


def cmd_srm(a: argparse.Namespace) -> int:
    counts = a.counts
    ratios = a.ratios or [1.0] * len(counts)
    if len(ratios) != len(counts) or len(counts) < 2 or any(c < 0 for c in counts):
        print("error: give at least two counts and one ratio per count", file=sys.stderr)
        return 2
    total = sum(counts)
    expected = [total * r / sum(ratios) for r in ratios]
    chi2 = sum((o - e) ** 2 / e for o, e in zip(counts, expected))
    p = gamma_q((len(counts) - 1) / 2, chi2 / 2)
    for i, (o, e) in enumerate(zip(counts, expected)):
        print(f"arm {i}: observed {o:,}  expected {e:,.0f}  ({(o - e) / e:+.2%})")
    print(f"chi-square {chi2:.3f}, df {len(counts) - 1}, p = {p:.6f}")
    if p < a.threshold:
        print(f"SRM DETECTED (p < {a.threshold}): do not read the results; find the assignment or logging bug")
        return 1
    print("no sample ratio mismatch at this threshold")
    return 0


def cmd_result(a: argparse.Namespace) -> int:
    c1, n1, c2, n2 = a.conv_a, a.n_a, a.conv_b, a.n_b
    if min(n1, n2) <= 0 or not (0 <= c1 <= n1 and 0 <= c2 <= n2):
        print("error: conversions must be between 0 and visitors", file=sys.stderr)
        return 2
    p1, p2 = c1 / n1, c2 / n2
    pooled = (c1 + c2) / (n1 + n2)
    se0 = math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
    z = (p2 - p1) / se0 if se0 else 0.0
    pval = 2 * (1 - Z.cdf(abs(z)))
    se = math.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
    zc = Z.inv_cdf(1 - a.alpha / 2)
    lo, hi = (p2 - p1) - zc * se, (p2 - p1) + zc * se
    print(f"control   {c1:,}/{n1:,} = {p1:.4%}")
    print(f"treatment {c2:,}/{n2:,} = {p2:.4%}")
    print(f"absolute diff {p2 - p1:+.4%}  ({1 - a.alpha:.0%} CI {lo:+.4%} to {hi:+.4%})")
    if p1 > 0:
        print(f"relative lift {(p2 - p1) / p1:+.2%}  (CI approx {lo / p1:+.2%} to {hi / p1:+.2%})")
    print(f"z = {z:.3f}, two-sided p = {pval:.5f} -> {'significant' if pval < a.alpha else 'not significant'} at alpha {a.alpha}")
    return 0


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = ap.add_subparsers(dest="cmd", required=True)

    s = sub.add_parser("size", help="sample size per arm")
    s.add_argument("--baseline", type=float, required=True, help="control conversion rate, e.g. 0.042")
    s.add_argument("--mde", type=float, required=True, help="minimum detectable effect (relative by default)")
    s.add_argument("--absolute", action="store_true", help="treat --mde as absolute percentage points")
    s.add_argument("--alpha", type=float, default=0.05)
    s.add_argument("--power", type=float, default=0.8)
    s.add_argument("--arms", type=int, default=2, help="total arms including control")
    s.add_argument("--one-sided", action="store_true")
    s.add_argument("--daily", type=int, help="eligible users per day across all arms")
    s.set_defaults(fn=cmd_size)

    r = sub.add_parser("srm", help="sample ratio mismatch check")
    r.add_argument("counts", type=int, nargs="+", help="assigned users per arm")
    r.add_argument("--ratios", type=float, nargs="+", help="planned split, e.g. 50 50 or 2 2 1")
    r.add_argument("--threshold", type=float, default=0.001)
    r.set_defaults(fn=cmd_srm)

    t = sub.add_parser("result", help="two-proportion z-test")
    t.add_argument("conv_a", type=int)
    t.add_argument("n_a", type=int)
    t.add_argument("conv_b", type=int)
    t.add_argument("n_b", type=int)
    t.add_argument("--alpha", type=float, default=0.05)
    t.set_defaults(fn=cmd_result)

    args = ap.parse_args()
    return args.fn(args)


if __name__ == "__main__":
    sys.exit(main())

About this skill

Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.

Related skills

All 32 Growth & SEO skills
esc
New

Introducing Codex Skills Pro

79 professional skills that take Codex through complete engagements, each with its own scripts, references and a report that verifies the result.

All-AccessOne-time payment, no subscription
$49
Get All-AccessOr browse individual skills from $2.99 →

Secure checkout by Creem