Synthetic Data Generation skill for Codex

Synthetic data is useful when it is reproducible (seeded, versioned), covers the cases you chose on purpose rather than whatever the generator likes, is validated against the same schema as real data, and is labelled as synthetic everywhere it lands. It never replaces a real held-out test set for claiming production quality.

Pick the generator by purpose

Purpose	Generator	Notes
Unit and integration test fixtures	Faker + factories (see test-data-factories)	Seeded, deterministic, no network
Property-based edge cases	Hypothesis / fast-check (see property-based-testing)	Shrinks failures automatically
Database seed for dev/staging	Faker + domain rules (see database-seeding)	Referential integrity first
LLM eval or red-team cases	LLM with an attribute grid	Needs dedup, validation, human spot check
Rare-class augmentation for a classifier	LLM paraphrase of real examples	Train/dev only, never test
Stand-in for a sensitive production table	SDV or similar synthesizer	Measure fidelity and privacy before sharing

Seeded fixtures (no LLM)

from faker import Faker
import random

Faker.seed(20260101)          # class-level seed: same output on every run
random.seed(20260101)
fake = Faker(["en_US", "de_DE"])

def customer(i: int) -> dict:
    return {
        "id": f"cus_{i:06d}",
        "name": fake.name(),
        "email": f"user{i}@example.test",     # reserved test domain, never a real address
        "country": random.choice(["US", "DE"]),
        "created_at": fake.date_time_between("-2y", "now").isoformat(),
    }

• Use example.com, example.test and documented test card numbers; Faker can emit plausible real-looking emails and phone numbers that belong to someone.
• Validate every generated record with the production schema (Pydantic Model.model_validate, JSON Schema, DB constraints). Generators drift from schemas.
• Write the seed and generator version into the output metadata.

LLM-generated eval cases

Never ask “generate 100 diverse test questions”. You get 100 variations of the same three. Define the coverage yourself as an attribute grid, then generate per cell.

import itertools, json

GRID = {
    "intent": ["refund", "cancel", "upgrade", "bug_report", "account_access"],
    "persona": ["new user", "power user", "angry customer", "non-native speaker"],
    "difficulty": ["direct", "ambiguous", "multi-intent", "missing key detail"],
    "channel": ["chat", "email"],
}
cells = [dict(zip(GRID, combo)) for combo in itertools.product(*GRID.values())]
# 160 cells; sample if too many, but sample cells, not free-form prompts.

PROMPT = """Write one realistic customer message for a SaaS support inbox.
Intent: {intent}. Persona: {persona}. Difficulty: {difficulty}. Channel: {channel}.
Return JSON: {{"message": str, "expected_intent": str, "notes": str}}.
Do not mention the persona or difficulty explicitly. Vary length and tone."""

Rules:

• Use structured outputs for the record shape; reject anything that fails validation.
• Generate with a different model (or at least a different prompt) from the system under test, or the eval rewards the generator’s own style.
• Store the grid cell with each record so you can report quality per slice.
• Seed with 10-20 real anonymized examples in the prompt when available; it anchors realism (typos, length, jargon).
• Generate expected answers separately from inputs, and have a human verify a sample. LLM-written gold labels inherit the generator’s mistakes.

Edge and adversarial cases

Add these deliberately, as their own grid dimension or set:

• Empty, whitespace, very long (near the context limit), mixed language, emoji, RTL text.
• Malformed inputs: truncated JSON, wrong encodings, HTML fragments.
• Prompt-injection attempts in user content and in retrieved documents (see prompt-injection-defense for the attack catalogue).
• Out-of-scope requests the system must refuse or route.

Dedup and quality filters

def shingles(text: str, n: int = 5) -> set[str]:
    toks = text.lower().split()
    return {" ".join(toks[i:i + n]) for i in range(max(1, len(toks) - n + 1))}

def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if a | b else 1.0

1. Exact dedup on normalized text.
2. Near-dup: drop pairs with 5-gram Jaccard above ~0.6, or embedding cosine above ~0.92 (tune by eyeballing 20 pairs around the cut-off).
3. Validate schema and business rules; drop failures, log the rate. Over ~10% failures means the prompt is wrong.
4. Diversity report: counts per grid cell, length distribution, distinct-n (unique n-grams / total). Fill empty cells with targeted generation.
5. Human review: 30-50 random records, scored realistic / unrealistic / wrong label.

Tabular synthesizers

SDV (pip install sdv, needs network for install; check its Business Source License terms for commercial use):

from sdv.metadata import Metadata
from sdv.single_table import GaussianCopulaSynthesizer
from sdv.evaluation.single_table import run_diagnostic, evaluate_quality

metadata = Metadata.detect_from_dataframe(data=df, table_name="orders")
metadata.validate()                         # fix detected types and primary keys by hand
synth = GaussianCopulaSynthesizer(metadata)
synth.fit(df)
fake = synth.sample(num_rows=10_000)

run_diagnostic(df, fake, metadata)          # structural validity
evaluate_quality(df, fake, metadata)        # column shapes and pair trends

• Drop direct identifiers (names, emails, ids) before fitting; regenerate them with Faker.
• Privacy check before sharing: for each synthetic row, distance to the closest real row. Exact or near-exact copies of real rows mean the model memorized; do not share. Rare categories (a single customer in a small town) re-identify people; bucket them first.
• Synthetic tabular data is not anonymization under GDPR by default. Have the data protection owner approve before it leaves the controlled environment.

Workflow

• Read AGENTS.md; find schemas, existing fixtures and factory helpers
• State the purpose and pick the generator from the table
• Define coverage (grid, edge cases) and record seeds and versions
• Generate in batches with structured output; validate every record
• Dedup, diversity report, fill gaps
• Human spot check; fix prompt or rules; regenerate
• Write to a versioned file (evals/data/support_v3.jsonl) with a README line on how it was made
• Mark records "synthetic": true

Output contract

Each dataset ships with a manifest:

{
  "name": "support_intents_v3",
  "purpose": "eval",
  "records": 640,
  "generator": {"model": "<id>", "prompt_sha": "<hash>", "seed": 20260101},
  "grid": {"intent": 5, "persona": 4, "difficulty": 4, "channel": 2},
  "filters": {"schema_fail": 21, "near_dup": 37},
  "human_review": {"sampled": 40, "unrealistic": 3, "wrong_label": 1},
  "created": "2026-10-28"
}

Gotchas

• Synthetic eval sets overestimate quality: they are cleaner and more on-topic than real traffic. Report them separately from real-data results.
• Training on LLM-paraphrased test items leaks the test set. Keep generation inputs and test items disjoint.
• Unseeded Faker in tests makes failures unreproducible.
• Large generation runs cost real money; estimate tokens per record times count and confirm with the user before generating more than a few thousand LLM records.