OpenAI Batch API skill for Codex

Done well means: the input file is validated offline before upload, every request has a stable custom_id that maps back to a source row, results are joined by custom_id (never by position), failures and expirations are resubmitted automatically, and the job is idempotent so a rerun does not double-process.

When batch is the right tool

Situation	Choice
Results needed within seconds/minutes for a user	Synchronous API
Thousands+ of independent requests, results within 24 h OK	Batch (50% cheaper, separate rate-limit pool)
Eval suite over a fixed dataset, run nightly	Batch
Steps depend on each other’s output (agent loops, multi-turn)	Synchronous, or chain batches stage by stage
Under ~100 requests	Synchronous with a concurrency limit; batch overhead not worth it

Limits to design around: up to 50,000 requests and 200 MB per input file, one model per file, one endpoint per batch, completion_window="24h" only, embeddings batches capped at 50,000 total inputs, and a per-model cap on enqueued prompt tokens (see the organization limits page). Output files are deleted 30 days after completion — copy results to your own storage.

Supported endpoints: /v1/responses, /v1/chat/completions, /v1/embeddings, /v1/completions, /v1/moderations, /v1/images/generations, /v1/images/edits.

Workflow

• 1. Read AGENTS.md; find where jobs, storage and secrets live
• 2. Build the JSONL from source rows with deterministic custom_ids
• 3. Validate offline with scripts/batch_jsonl.py check
• 4. Upload, create the batch, persist the batch id
• 5. Poll until a terminal status
• 6. Download output and error files; merge by custom_id
• 7. Resubmit failures; write results idempotently

2. Build the input file

import hashlib, json, os

MODEL = os.environ["OPENAI_MODEL"]  # check the models page; one model per file
SCHEMA = {...}  # strict JSON Schema for the label, see openai-structured-outputs

def cid(row) -> str:
    # Stable across reruns: source id + prompt version, not a random uuid or row index
    return f"{row['id']}:{PROMPT_VERSION}"

with open("batch_input.jsonl", "w", encoding="utf-8") as fh:
    for row in rows:
        fh.write(json.dumps({
            "custom_id": cid(row),
            "method": "POST",
            "url": "/v1/responses",
            "body": {
                "model": MODEL,
                "instructions": INSTRUCTIONS,          # identical prefix -> cache-friendly
                "input": row["text"],
                "text": {"format": {"type": "json_schema", "name": "label",
                                     "strict": True, "schema": SCHEMA}},
                "max_output_tokens": 300,
                "store": False,
            },
        }) + "\n")

Split into multiple files when you exceed 50,000 lines or 200 MB, or when enqueued tokens would exceed the model’s batch queue limit (estimate input tokens per line × lines).

3. Validate before spending

python3 scripts/batch_jsonl.py check batch_input.jsonl

It checks JSON per line, unique custom_id, POST, a single supported endpoint, a single model, no stream: true, and the line/size limits. Runs offline; exit code 1 on any error. Also send 3–5 of the lines synchronously first to confirm the body works — a schema typo fails every request in the batch 24 hours later.

4–5. Submit and poll

Submitting needs network and spends money; confirm the row count and model with the user before the first large submission.

from openai import OpenAI
client = OpenAI()

f = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=f.id,
    endpoint="/v1/responses",
    completion_window="24h",
    metadata={"job": "ticket-labels", "prompt_version": PROMPT_VERSION},
)
save_job(batch.id)  # persist; the poller may run in a different process

TERMINAL = {"completed", "failed", "expired", "cancelled"}
batch = client.batches.retrieve(batch_id)
if batch.status not in TERMINAL:
    reschedule(minutes=10)           # cron/queue; do not sleep-loop in a web worker
print(batch.status, batch.request_counts)

Status values: validating, failed (input invalid — check batch.errors), in_progress, finalizing, completed, expired, cancelling, cancelled.

6. Collect and merge

def download(file_id, path):
    if file_id:
        with open(path, "w", encoding="utf-8") as fh:
            fh.write(client.files.content(file_id).text)

download(batch.output_file_id, "out.jsonl")
download(batch.error_file_id, "err.jsonl")

python3 scripts/batch_jsonl.py merge batch_input.jsonl out.jsonl err.jsonl --retry retry.jsonl

Each output line has custom_id, response.status_code, response.body (the normal endpoint response) and error. Output order does not match input order. The merge command reports succeeded/failed/missing counts, failures by code, and writes every request without a 200 result to retry.jsonl.

7. Retry and persist

• Resubmit retry.jsonl as a new batch. Cap resubmissions (e.g. 3 rounds); after that, send the remainder synchronously or report them.
• batch_expired lines are normal under load: completed requests are kept and billed, unfinished ones land in the error file.
• Write results with an upsert keyed on custom_id so reruns are idempotent.
• Parse response.body with the same code you use synchronously (output items, status == "incomplete" checks, refusals).

Gotchas

• Using the row index as custom_id: any filter or reorder corrupts the join.
• Mixing models or endpoints in one file fails validation for the whole batch.
• A batch that is completed can still contain many per-request errors — always read request_counts and the error file.
• Changing the prompt between the first round and the retry round mixes versions; keep the prompt version in custom_id and metadata.
• Cancelling takes up to ~10 minutes (cancelling); requests already done are billed.
• Large repeated instructions: put them first and identical across lines so prompt caching can apply where supported.

Verification

1. check passes and a synchronous dry run of 3–5 lines returns valid output.
2. After completion, merge shows succeeded == requests (or a known remainder).
3. Spot-check 10 random results against inputs by custom_id.
4. Compare cost from the usage dashboard with the synchronous estimate.

Output contract

**Job:** ticket-labels v3 — 42,180 requests, model from OPENAI_MODEL, /v1/responses
**Batches:** batch_abc (completed), batch_def (retry round 1, 37 requests)
**Result:** 42,180 succeeded, 0 outstanding; written to warehouse.ticket_labels
**Failures seen:** 29 batch_expired, 8 status 500 — all succeeded on retry
**Cost:** <batch cost> vs <synchronous estimate>