OpenAI Realtime Voice skill for Codex

Done well means: the standard API key never reaches a browser, the transport matches the client (WebRTC for browsers and mobile, WebSocket for servers and telephony bridges), turn detection is tuned against real recordings, interruptions stop playback and truncate the model’s memory of unheard audio, and tool calls are fast or acknowledged out loud.

Decide the architecture

Client	Transport	Auth
Browser / mobile app	WebRTC to https://api.openai.com/v1/realtime/calls	Ephemeral client secret minted by your server
Your backend (agents, call center bridge)	WebSocket wss://api.openai.com/v1/realtime?model=...	Standard API key, server only
Browser, but session config must stay server-side	“Unified” WebRTC: browser sends SDP to your server, server posts multipart sdp + session to /v1/realtime/calls	Standard key on server
Want tools, handoffs, guardrails without raw events	Agents SDK RealtimeAgent / RealtimeSession (@openai/agents/realtime)	Ephemeral secret

Model: read from config (REALTIME_MODEL) and check the models page for the current gpt-realtime* model. The GA interface does not use the old OpenAI-Beta: realtime=v1 header — remove it when migrating beta code.

Workflow

• 1. Read AGENTS.md; find where secrets and server routes live
• 2. Server route that mints a client secret (or proxies SDP)
• 3. Client connection with mic track, remote audio, oai-events data channel
• 4. session.update with instructions, voice, turn detection, transcription
• 5. Event handler: transcripts, tool calls, errors, interruptions
• 6. Tools: fast, idempotent, time-boxed
• 7. Test with recorded audio in noisy and quiet conditions

2. Mint an ephemeral secret (server)

// POST /api/realtime-token  — server only; OPENAI_API_KEY never leaves here
app.post("/api/realtime-token", requireUser, async (req, res) => {
  const r = await fetch("https://api.openai.com/v1/realtime/client_secrets", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      session: {
        type: "realtime",
        model: process.env.REALTIME_MODEL,
        instructions: "You are a concise booking assistant. Speak in short sentences.",
        audio: { output: { voice: "marin" } },
      },
    }),
  });
  if (!r.ok) return res.status(502).send(await r.text());
  const data = await r.json();
  res.json({ value: data.value });   // short-lived key for this user only
});

Authenticate the user before minting; rate-limit the route — each token opens a billable session.

3. Browser WebRTC

const { value: key } = await (await fetch("/api/realtime-token", { method: "POST" })).json();
const pc = new RTCPeerConnection();
const audioEl = Object.assign(document.createElement("audio"), { autoplay: true });
pc.ontrack = (e) => (audioEl.srcObject = e.streams[0]);

const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});
pc.addTrack(mic.getTracks()[0]);

const dc = pc.createDataChannel("oai-events");
dc.onmessage = (m) => handleServerEvent(JSON.parse(m.data));

await pc.setLocalDescription(await pc.createOffer());
const sdp = await fetch("https://api.openai.com/v1/realtime/calls", {
  method: "POST",
  body: pc.localDescription!.sdp,
  headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/sdp" },
});
await pc.setRemoteDescription({ type: "answer", sdp: await sdp.text() });

Echo cancellation must stay on, or the model hears itself through the speakers and interrupts its own replies.

4. Session configuration

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "…",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "turn_detection": { "type": "semantic_vad", "eagerness": "auto" }
      },
      "output": { "voice": "marin" }
    },
    "tools": [ { "type": "function", "name": "check_availability", "description": "…", "parameters": { "type": "object", "properties": { "date": { "type": "string" } }, "required": ["date"] } } ]
  }
}

Voice is fixed once the model has produced audio in the session; choose it up front.

Turn detection

Symptom / need	Setting
Default, clean audio, predictable timing	server_vad; tune silence_duration_ms (start ~500) and threshold (raise in noise)
Users pause mid-sentence and get cut off	semantic_vad with eagerness: "low"
Fast back-and-forth, short answers	semantic_vad eagerness: "high" or lower silence_duration_ms
Push-to-talk, kiosks, noisy rooms	turn_detection: null; client sends input_audio_buffer.commit then response.create on button release
Background noise triggers barge-in	Raise threshold, keep browser noise suppression, or set interrupt_response: false

5. Events to handle

• session.created / session.updated: confirm the config actually applied.
• input_audio_buffer.speech_started: user barged in — stop local playback now.
• response.output_audio_transcript.delta / response.output_text.delta: captions.
• conversation.item.input_audio_transcription.completed: user transcript (only if input transcription is configured).
• response.done: final status, usage, and any function_call items.
• error: log the full event; most are config or schema mistakes.

Interruptions (barge-in)

• WebRTC: the server truncates unplayed audio for you. For push-to-talk, send output_audio_buffer.clear to drop queued audio.
• WebSocket: you own playback. On speech_started, stop the player and send conversation.item.truncate with the assistant item_id, content_index: 0 and audio_end_ms equal to what was actually played. Without it the model believes the user heard the whole answer.

WebSocket audio format

Send base64 PCM16 mono at 24 kHz, little-endian, via input_audio_buffer.append in chunks of roughly 20–100 ms. Resample telephony audio (8 kHz mu-law) or configure a matching input format instead of sending mismatched bytes — mismatches produce silence or noise with no error.

6. Tool calls in a live conversation

1. Read function_call items from response.done (stream arguments with response.function_call_arguments.delta if you need them early).
2. Execute, then send conversation.item.create with {"type": "function_call_output", "call_id": ..., "output": "<json string>"}.
3. Send response.create so the model speaks the result.

Keep tools under ~1 s; for slower work, instruct the model to say “let me check” first. Confirm destructive or paid actions verbally and in your own code before executing.

Gotchas

• Shipping the standard key to the browser. Always mint ephemeral secrets server-side.
• Sample-rate mismatch (48 kHz mic data labelled 24 kHz) gives chipmunk or slowed audio.
• Long instructions and many tools add latency to every response; keep both lean.
• Forgetting response.create after a tool output leaves the call in dead silence.
• Sessions have a maximum duration; handle disconnects and reconnect with a fresh secret.
• Record consent: tell users the call is AI-handled and whether audio is stored.

Verification

• Measure time from speech_stopped to first response.output_audio.delta; track p50/p95.
• Replay 10+ recorded utterances (quiet, noisy, long pauses, interruptions) and log false barge-ins and premature turn ends per VAD setting.
• Assert the token route rejects unauthenticated requests and never returns the real key (grep -rn "OPENAI_API_KEY" client/ web/ public/ must find nothing).
• For WebSocket, check audio_end_ms in truncate events matches playback position.

Output contract

**Transport:** WebRTC (browser) with ephemeral secret route `/api/realtime-token`
**Model / voice:** from REALTIME_MODEL / marin
**Turn detection:** semantic_vad eagerness=low — chosen because <reason>
**Tools:** check_availability (read, <300 ms), book_slot (write, verbal confirm)
**Latency:** p50 <x> ms, p95 <y> ms end-of-speech to first audio
**Open issues:** <noise handling, reconnect, consent copy>