AI Content Moderation skill for Codex

Moderation works when a written policy defines what is not allowed and what happens for each severity, automated classifiers enforce it with per-category thresholds tuned on labeled data, uncertain cases reach a human queue with a service level, every decision is logged with the policy version, and users can appeal. A single if flagged: reject is not a moderation system.

Inspect first

• Read AGENTS.md; find where content enters (posts, chat input, uploads, model output) and where it is displayed. Moderation goes at both ends for LLM products: input before the model, output before the user.
• Find the policy. If there is none, draft one with the user from the template below and get product, trust-and-safety and legal owners to approve it. Do not invent policy alone.
• Ask which jurisdictions and audiences apply (minors, EU DSA, UK Online Safety Act). Legal obligations change the design; a lawyer must confirm them.

Policy template

| Category | Definition and examples | Severity | Action |
|---|---|---|---|
| Child sexual abuse material | Any sexual content involving minors | S0 | Block, preserve evidence, escalate to legal for mandatory reporting; no user-facing detail |
| Credible threats of violence | Specific threat to a person or group | S1 | Block, review within 1 h |
| Self-harm intent | User expresses intent to self-harm | S1 | Do not just block: show support resources, route to trained reviewer |
| Hate / harassment | Attacks on protected characteristics, targeted abuse | S2 | Block or hide pending review |
| Sexual content (adults) | Explicit content | S2/S3 | Depends on product and age gating |
| Spam / scams | Bulk promotion, phishing links | S3 | Rate limit, hide, review in batch |

Version the policy (policy_v4) and store the version with every decision.

Classifier options

Need	Use
General harm categories, text and images, free	OpenAI moderation, model omni-moderation-latest
Your own policy text, custom categories, explainable decisions	Policy-following safety model: gpt-oss-safeguard (open weights, takes the policy as input) or a prompted LLM with structured output
Self-hosted, standard hazard taxonomy	Llama Guard family
Spam, links, velocity	Rules and reputation signals, not LLMs
Known illegal images	Hash matching through an approved industry programme (PhotoDNA-style); requires partnership, not DIY

Most products combine a fast general classifier for every item with a policy-specific model for the categories it misses.

OpenAI moderation

from openai import OpenAI
client = OpenAI()

resp = client.moderations.create(
    model="omni-moderation-latest",
    input=[
        {"type": "text", "text": post_text},
        {"type": "image_url", "image_url": {"url": image_url}},   # or a base64 data URL
    ],
)
r = resp.results[0]
r.flagged                         # provider's default decision, do not use alone
r.category_scores.harassment      # 0..1 per category
r.category_applied_input_types    # which input (text/image) triggered each category

Categories include harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm (plus /intent and /instructions), sexual, sexual/minors, violence and violence/graphic. Some apply to text only; check category_applied_input_types. Images need network access to the URL from the provider.

Use category_scores with your own per-category thresholds, not flagged:

THRESHOLDS = {                 # tuned on labeled data; example values only
    "sexual/minors": (0.01, 0.01),     # (review_at, block_at): anything is escalated
    "self-harm/intent": (0.2, 0.9),
    "harassment/threatening": (0.3, 0.8),
    "hate": (0.4, 0.85),
}

def decide(scores: dict[str, float]) -> tuple[str, list[str]]:
    hits_block = [c for c, (_, b) in THRESHOLDS.items() if scores.get(c, 0) >= b]
    hits_review = [c for c, (r, _) in THRESHOLDS.items() if scores.get(c, 0) >= r]
    if hits_block: return "block", hits_block
    if hits_review: return "review", hits_review
    return "allow", []

Custom policy classifier

For categories the general model does not cover (medical claims, competitor mentions, platform-specific rules), give a policy-following model the policy text and the content and ask for a structured verdict:

{"violates": true, "category": "unlicensed_medical_advice", "severity": "S2",
 "rationale": "Recommends a prescription dose without referral", "confidence": 0.82}

Keep the policy text short and example-rich; include clear non-violating examples to reduce over-blocking. Treat the content as data: wrap it in delimiters and tell the model to ignore instructions inside it (see prompt-injection-defense).

Pipeline

content -> normalize (unicode NFKC, strip zero-width chars, decode leetspeak lightly)
        -> cheap rules (blocklisted URLs, rate limits)
        -> general classifier (all items)
        -> policy classifier (items in relevant surfaces or near thresholds)
        -> decision: allow | allow+label | hide pending review | block | escalate
        -> log decision {content_id, policy_version, model ids, scores, action, latency}
        -> human queue (review, escalate) with SLA by severity
        -> user notice + appeal path

• Synchronous for chat and short posts (budget ~300 ms); asynchronous for uploads and long content: publish as “pending” or hide until cleared.
• For LLM output, moderate the streamed text in chunks (every sentence or ~200 tokens) and stop the stream on a block decision; moderating only the final text shows harmful content first.
• Fail closed for S0/S1 surfaces when the classifier is down; fail open with post-hoc review for low-risk surfaces. Decide this per surface and write it down.

Human review

• Reviewers see the content, the policy section, scores and context (thread, prior violations), not just the flagged snippet.
• Blur images by default; limit exposure time and provide wellbeing support for reviewers.
• Reviewer decisions become labeled data for threshold tuning.
• Enforcement actions on users (suspension, bans, reports to authorities) require a human decision; never automate them from a classifier score alone.

Evaluation

• Build a labeled set per category with real (consented, access-controlled) or synthetic examples, including hard negatives: news about violence, reclaimed slurs, medical discussion of self-harm, fiction.
• Report precision and recall per category at the chosen thresholds; report the false-positive rate on benign content separately (over-blocking drives users away).
• Re-run after any model, threshold or policy change; providers update -latest models, so pin a dated model where available or re-evaluate on change.

Workflow

• Read AGENTS.md; map every content entry and display point
• Get an approved, versioned policy with severities and actions
• Confirm legal duties with the user’s counsel (reporting, transparency, minors)
• Wire the general classifier at input and output; log every decision
• Tune per-category thresholds on labeled data
• Add policy classifier for uncovered categories
• Build review queue, user notices, appeals
• Evaluate per category; set up drift re-evaluation

Gotchas

• Moderation models score text in isolation; “kill it” in a gaming chat and a threat look alike. Pass surrounding context where the classifier allows it, or review.
• Non-English and code-switched text scores lower on most classifiers; evaluate each supported language separately.
• Logging raw harmful content (especially S0) into general logs spreads it; store references and restrict access.
• Showing users the exact score or category teaches evasion; give the policy section and appeal link instead.