Multi-Agent Orchestration skill for Codex

Done well means: there is a measured reason for more than one agent, each agent has one job, a narrow toolset and a typed input/output contract, the orchestrator enforces budgets and deadlines, a failed worker degrades the result instead of crashing or looping, and one trace shows the whole run.

First, should this be multi-agent at all?

Signal	Verdict
One agent with < ~15 tools completes the task within its context	Stay single-agent; tune tools and prompt
Subtasks are independent and can run in parallel (research N sources, review N files)	Orchestrator-worker
Distinct domains with separate tools and permissions (billing vs tech support)	Router or handoffs
Fixed stages with different expertise (extract -> validate -> write)	Pipeline (often plain code, not agents)
Output quality improves with critique against clear criteria	Evaluator-optimizer loop, capped iterations
Context overflows because of accumulated tool output	Workers with fresh context that return summaries

Multi-agent systems cost several times the tokens of a single agent and add coordination failure modes. Require a baseline: run the single-agent version on the same eval tasks first.

Patterns

Pattern	Control	Use	Main risk
Orchestrator-worker	Lead agent plans, spawns workers, synthesises	Breadth tasks, parallel research	Duplicate work, vague worker briefs
Router	Classifier picks one specialist	Support triage, intent routing	Misroutes; add a fallback specialist
Handoff	Active agent transfers the conversation	Domain switches mid-conversation	Ping-pong between agents
Pipeline	Code calls agents in fixed order	Document processing, ETL with LLM steps	Error propagation; validate between stages
Evaluator-optimizer	Generator + critic loop	Writing, code fixes with tests	Endless loops; cap rounds

Prefer deterministic code for control flow wherever the next step is knowable; reserve LLM decisions for what genuinely needs judgment.

Workflow

- [ ] 1. Baseline the single-agent version on 20+ eval tasks (success, tokens, latency)
- [ ] 2. Pick the pattern from the table; draw the flow with who-decides-what
- [ ] 3. Define task contracts (input brief, output schema) for each worker
- [ ] 4. Set budgets: max workers, max depth, tokens/cost, wall-clock deadline per level
- [ ] 5. Implement failure policy: timeout, retry, partial results, cancellation
- [ ] 6. Make side effects idempotent and gate outward-facing actions on human approval
- [ ] 7. Trace every agent call under one run ID; compare against the baseline

Task contracts

Workers fail most often because the brief was vague. Every delegation carries:

from pydantic import BaseModel, Field

class WorkerBrief(BaseModel):
    objective: str                 # one sentence, the question to answer
    scope: str                     # what to cover and what NOT to cover
    sources_or_inputs: list[str]   # files, URLs, IDs the worker should use
    output_format: str             # e.g. "WorkerResult JSON"
    budget_tool_calls: int = 15
    deadline_seconds: int = 120

class WorkerResult(BaseModel):
    status: str = Field(pattern="^(complete|partial|failed)$")
    findings: list[str]
    evidence: list[str]            # citations: file:line, URL, record ID
    open_questions: list[str]
    tokens_used: int

• The orchestrator writes briefs that do not overlap (“worker A: pricing pages only”).
• Workers return structured results with evidence, not prose essays. The synthesiser cites evidence; anything without evidence is flagged.
• Pass references (file paths, record IDs, storage keys) instead of large blobs.

Orchestrator with budgets and partial failure (asyncio)

import asyncio

MAX_WORKERS = 6
sem = asyncio.Semaphore(4)               # concurrency cap, respects provider rate limits

async def run_worker(brief: WorkerBrief) -> WorkerResult:
    async with sem:
        try:
            return await asyncio.wait_for(worker_agent(brief), timeout=brief.deadline_seconds)
        except asyncio.TimeoutError:
            return WorkerResult(status="failed", findings=[], evidence=[],
                                open_questions=[f"timed out: {brief.objective}"], tokens_used=0)
        except Exception as exc:          # one worker must not sink the run
            log.exception("worker failed", extra={"objective": brief.objective})
            return WorkerResult(status="failed", findings=[], evidence=[],
                                open_questions=[f"error {type(exc).__name__}: {brief.objective}"], tokens_used=0)

async def orchestrate(task: str, budget: Budget) -> Report:
    briefs = (await plan(task))[:MAX_WORKERS]
    results = await asyncio.gather(*(run_worker(b) for b in briefs))
    budget.charge(sum(r.tokens_used for r in results))
    ok = [r for r in results if r.status != "failed"]
    if len(ok) < max(1, len(briefs) // 2):
        return Report.incomplete(task, results)          # say so instead of guessing
    return await synthesize(task, ok, missing=[r for r in results if r.status == "failed"])

Budget rules:

• Hard caps in code: workers per level, recursion depth (workers spawning workers: usually 0 or 1 extra level), total tokens or cost, wall-clock deadline. Prompts are not enforcement.
• Check the budget before each spawn and before each synthesis round; stop with a partial report when exceeded.
• Scale effort to the task: a simple lookup gets 1 worker, a broad comparison 3-6.

Handoffs and routing

• Each specialist has a description written for the router (“Handles invoices, refunds and payment failures. Not for login problems.”).
• Pass a handoff summary (customer, goal, what was tried) rather than the full transcript when the receiving agent does not need it.
• Prevent ping-pong: count handoffs per conversation; after 2-3, route to a human or a generalist.
• Permissions follow the agent: the billing agent’s tools are not available to the FAQ agent.
• Framework specifics: $openai-agents-sdk (handoffs, guardrails) or $langgraph-agents (subgraphs, Send, Command(goto=...)).

Evaluator-optimizer loops

• Critic uses explicit criteria or a rubric and returns pass/fail plus specific fixes.
• Cap rounds (2-3). If still failing, return the best version with the critic’s remaining issues.
• Prefer objective checks (tests, schema validation, linters) over an LLM critic whenever they exist.

Side effects and consistency

• Workers that write (files, tickets, records) use idempotency keys derived from run ID + task ID, so retries do not duplicate.
• Parallel workers must not write the same resource. Assign ownership (one worker per file or record) or funnel writes through the orchestrator.
• Outward-facing actions (sending messages, publishing, payments, deploys) happen once, after synthesis, with explicit user confirmation. Never let a worker do them directly.

Observability

• One run_id propagated to every agent and tool call; parent/child spans for orchestrator -> worker -> tool ($llm-observability).
• Record per agent: brief, status, tool calls, tokens, latency, cost.
• Dashboards to watch: tokens per successful task, worker failure rate, timeouts, handoff loops.

Gotchas

• Workers duplicating each other’s searches: briefs lacked scope boundaries.
• Synthesiser hallucinating to fill gaps from failed workers: pass the missing list and instruct it to state gaps.
• Shared mutable memory between parallel agents leads to races and contradictory state; share read-only context, merge results in the orchestrator.
• Rate limits: N parallel workers multiply request rate; the semaphore and backoff are mandatory.
• Long-running orchestrations need durable state (checkpoints or a job table) to resume after a crash instead of restarting and re-paying.

Verification

• Eval comparison against the single-agent baseline on the same tasks: success rate, tokens, p95 latency, cost. Ship only if success improves enough to justify cost.
• Fault injection: force one worker to time out and one to raise; the run completes with a partial report that names the gaps.
• Budget test: a task that would exceed the cap stops at the cap.

Report template

**Pattern:** orchestrator-worker, max 6 workers, depth 1, deadline 180 s, budget 400k tokens
**Baseline vs multi-agent (40 tasks):** success 62% -> 81%, tokens x3.4, p95 95 s -> 140 s
**Failure handling:** timeout and exception -> partial result; < 50% workers ok -> incomplete report
**Side effects:** none in workers; ticket creation after synthesis with user confirmation
**Tracing:** run_id spans in Langfuse; dashboard "tokens per successful task"