LLM Streaming skill for Codex

Streaming works when the first token reaches the user within the provider’s time-to-first-token plus ~100 ms, nothing between server and browser buffers it, a disconnect cancels the upstream call within a second, and errors that happen after headers are sent still reach the client as a typed event.

Inspect first

1. Read AGENTS.md and find the runtime (Node, edge, Python), framework and hosting (nginx, Cloudflare, Vercel, AWS ALB, Cloud Run). Most “streaming is broken” bugs are a buffering layer, not code.
2. Reproduce with curl, which never buffers: curl -N -X POST localhost:3000/api/chat -H 'content-type: application/json' -d '{"q":"count to 20"}'. If curl streams locally but the browser does not through the deployed URL, the problem is a proxy, compression or the client parser.
3. Check the provider SDK: OpenAI Responses (stream=True, typed events such as response.output_text.delta), Gemini, local servers. Use the SDK’s stream iterator; do not parse provider SSE by hand.

Protocol choice

Need	Use
Server pushes tokens for one request	SSE over a POST response (text/event-stream) read with fetch
Bidirectional, interrupts, audio	WebSocket (see websockets-realtime) or the provider’s realtime API
Simple GET with auto-reconnect, no auth headers	Browser EventSource
Internal service to service	Provider SDK stream or gRPC server streaming

EventSource only does GET and cannot set an Authorization header. For chat, POST plus fetch and a stream reader is the default.

SSE framing

Each event is field: value lines ended by a blank line. Send JSON in data: with your own event names so the client can branch.

event: delta
data: {"text":"Hel"}

event: tool_call
data: {"id":"call_1","name":"search","args":{"q":"x"}}

event: done
data: {"finish_reason":"stop","usage":{"input":812,"output":233}}

: keep-alive comment, ignored by clients

Required response headers:

Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache, no-transform
Connection: keep-alive          (HTTP/1.1 only; omit on HTTP/2)
X-Accel-Buffering: no           (tells nginx not to buffer this response)

Node: route handler with cancellation

// Next.js App Router / any Fetch-API runtime
import OpenAI from "openai";
const client = new OpenAI();

export async function POST(req: Request) {
  const { input } = await req.json();
  const encoder = new TextEncoder();
  const send = (event: string, data: unknown) =>
    encoder.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

  const stream = new ReadableStream({
    async start(controller) {
      const heartbeat = setInterval(() => controller.enqueue(encoder.encode(": ping\n\n")), 15000);
      try {
        const upstream = await client.responses.create(
          { model: process.env.MODEL!, input, stream: true },
          { signal: req.signal },               // client disconnect aborts the provider call
        );
        for await (const ev of upstream) {
          if (ev.type === "response.output_text.delta") controller.enqueue(send("delta", { text: ev.delta }));
          if (ev.type === "response.completed") controller.enqueue(send("done", { usage: ev.response.usage }));
        }
      } catch (err) {
        if (!req.signal.aborted) controller.enqueue(send("error", { message: "generation_failed" }));
      } finally {
        clearInterval(heartbeat);
        try { controller.close(); } catch {}
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

Express: res.on("close", () => controller.abort()), call res.flushHeaders() once, and respect backpressure: if res.write(chunk) returns false, await once(res, "drain") before writing more. Disable compression() for the route or call res.flush() after each write; gzip middleware otherwise holds bytes until its buffer fills.

Python: FastAPI

import json, asyncio
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

app, client = FastAPI(), AsyncOpenAI()

def sse(event: str, data) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"

@app.post("/api/chat")
async def chat(request: Request, body: dict):
    async def gen():
        stream = await client.responses.create(model=MODEL, input=body["input"], stream=True)
        try:
            async for ev in stream:
                if await request.is_disconnected():
                    break
                if ev.type == "response.output_text.delta":
                    yield sse("delta", {"text": ev.delta})
                elif ev.type == "response.completed":
                    yield sse("done", {"usage": ev.response.usage.model_dump()})
        except Exception:
            yield sse("error", {"message": "generation_failed"})
        finally:
            await stream.close()          # releases the upstream HTTP connection
    return StreamingResponse(gen(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Use an async generator with an async client. A sync generator with a sync SDK runs in a threadpool and blocks a worker per stream. sse-starlette (EventSourceResponse) adds ping and disconnect handling if you prefer a library.

Client: parse a POST stream

const res = await fetch("/api/chat", { method: "POST", body, signal: controller.signal,
  headers: { "content-type": "application/json" } });
if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buf = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += value;
  let i;
  while ((i = buf.indexOf("\n\n")) !== -1) {          // events can split across chunks
    const raw = buf.slice(0, i); buf = buf.slice(i + 2);
    let event = "message", data = "";
    for (const line of raw.split("\n")) {
      if (line.startsWith("event:")) event = line.slice(6).trim();
      else if (line.startsWith("data:")) data += line.slice(5).trimStart();
    }
    if (data) handle(event, JSON.parse(data));
  }
}

Network chunks do not align with events or even with UTF-8 characters; TextDecoderStream handles multi-byte splits, the buffer handles event splits.

Buffering layers to fix

Layer	Fix
nginx	proxy_buffering off; for the location, or the X-Accel-Buffering: no header; proxy_read_timeout 300s;
Compression middleware	Exclude text/event-stream, or flush per write
Cloudflare	Streams pass through, but proxied requests time out after ~100 s with no bytes: heartbeats keep it alive
AWS ALB / API Gateway	ALB idle timeout default 60 s (raise it, send heartbeats); REST API Gateway buffers responses: use a Lambda function URL with response streaming or another path
Serverless platforms	Check the function’s max duration; long agent runs need a background job plus a resumable stream
Dev servers / Next.js	Do not await the full generation before returning the Response; return the stream immediately

Errors and resumption

• Once headers are sent you cannot change the status code. Send an error event with a stable code and let the client mark the message failed.
• Always send a terminal event (done or error). A stream that just ends is ambiguous: the client should treat a missing terminal event as a dropped connection.
• For long generations, decouple generation from delivery: write events to a store (Redis stream, DB) with incrementing ids, send id: lines, and let the client reconnect with Last-Event-ID to replay from where it stopped. Cancel the job only on an explicit Stop, not on disconnect.

Workflow

• Reproduce with curl -N locally and against the deployed URL
• Use the provider SDK stream and map its events to your own small event set
• Set SSE headers, heartbeat every 15 s, terminal done/error event
• Propagate client abort to the provider call (signal, is_disconnected, close())
• Remove buffering in proxies and compression
• Parse on the client with a buffered event splitter
• Add a test that asserts incremental delivery and cancellation

Verification

• curl -N shows deltas arriving over time, not in one burst; measure time to first byte.
• Start a long generation, kill curl after 2 s, and confirm in server logs or the provider dashboard that the upstream request ended early.
• Run behind the real proxy (or a local nginx with the production config) and repeat.
• Unit test the client parser with events split mid-line and a multi-byte character split across chunks.
• Leave a stream idle for 2 minutes behind the proxy; heartbeats keep it open.

Gotchas

• res.json() or await response.text() anywhere in the chain silently turns a stream into a single blob.
• Serverless “streaming” responses can still buffer if the platform or framework version does not support it; test on the deployed URL, not only locally.
• Python print-style logging of every token adds seconds to long streams; log at the end.
• Retrying a failed stream from scratch can double-charge; retry only before the first token, otherwise surface the partial answer.