vLLM Serving skill for Codex

Done well means: the server starts reproducibly from a pinned image and model revision, memory settings are chosen from the real context and concurrency the product needs, the endpoint is authenticated, and throughput and latency are measured under a realistic load before anyone calls it production.

Inputs to collect first

• Model, revision, license, and precision available (bf16, FP8, AWQ/GPTQ int4).
• Traffic shape: typical and p95 prompt length, output length, peak concurrent requests, latency target (time to first token, inter-token latency) versus throughput target.
• GPUs: model, count, VRAM, NVLink or PCIe (nvidia-smi topo -m).
• Read AGENTS.md and existing deploy manifests (Dockerfile, Helm chart, compose file).

Pulling images and weights needs network; in a Codex sandbox ask for approval. Deploying to a shared or production cluster requires explicit user confirmation.

Sizing

Weights: parameters x bytes (bf16 2, FP8 1, int4 about 0.55). KV cache per token is about 2 x layers x kv_heads x head_dim x bytes. Example: Llama 3.1 8B in bf16 is about 128 KB per token, so 32 concurrent 8k-token sequences need about 32 GB of KV cache on top of 16 GB of weights.

Model size	Typical setup
7-8B	1x 24-48 GB GPU (bf16) or 1x 24 GB with FP8
14-32B	1x 80 GB (bf16/FP8), or TP=2 on 48 GB cards
70B	TP=4 on 80 GB (bf16), TP=2 on 80 GB with FP8
Larger MoE	TP within a node, add --enable-expert-parallel; check the model’s recipe

Tensor parallel size should divide the attention head count and stay inside one node (NVLink). For more throughput across nodes, run replicas (data parallel) behind a load balancer.

Baseline command

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --revision <commit-sha> \
  --served-model-name llama-8b \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --api-key "$VLLM_API_KEY" \
  --host 0.0.0.0 --port 8000

Docker (pin the tag; never latest in production):

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN -e VLLM_API_KEY \
  vllm/vllm-openai:<version> \
  --model meta-llama/Llama-3.1-8B-Instruct --max-model-len 16384 --api-key "$VLLM_API_KEY"

--ipc=host (or a large --shm-size) is required for tensor parallelism.

Flag decisions

Goal	Flag
Fit more concurrent sequences	--kv-cache-dtype fp8; lower --max-model-len to what traffic needs
Startup OOM	Lower --gpu-memory-utilization if other processes share the GPU; otherwise shorter --max-model-len or more TP
Shared system prompts / RAG prefixes	Prefix caching (on by default in the V1 engine; --enable-prefix-caching on older)
Lower time to first token under load	Tune --max-num-batched-tokens (chunked prefill budget); smaller favors decode latency
Higher throughput	Raise --max-num-seqs; FP8 weights on Hopper/Ada/Blackwell
Pre-quantized checkpoint	Usually auto-detected from config; --quantization awq/gptq/fp8 if needed
Multiple fine-tunes on one base	--enable-lora --lora-modules sql=/adapters/sql support=/adapters/support --max-lora-rank 64
Tool calling	--enable-auto-tool-choice --tool-call-parser <parser> (hermes, llama3_json, mistral, …) per model docs
Reasoning models	--reasoning-parser <name> so reasoning returns separately from content
Custom chat template	--chat-template ./template.jinja when the tokenizer’s template is missing or wrong

Check vllm serve --help for the installed version; flag names and defaults shift between releases. Pin the version in requirements or the image tag.

Calling the server

import os
from openai import OpenAI

client = OpenAI(base_url="http://gpu-host:8000/v1", api_key=os.environ["VLLM_API_KEY"])

r = client.chat.completions.create(
    model="llama-8b",
    messages=[{"role": "user", "content": "Return the order id from: 'Order #A-1182 shipped'"}],
    response_format={"type": "json_schema", "json_schema": {"name": "order", "schema": {
        "type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}},
    temperature=0,
)

LoRA adapters are addressed by their name in model=. Offline batch jobs can skip the server: from vllm import LLM, SamplingParams; LLM(model=...).generate(prompts, SamplingParams(...)).

Observability and load testing

GET /metrics exposes Prometheus metrics. Watch:

Metric	Meaning
vllm:num_requests_running / vllm:num_requests_waiting	Waiting > 0 for long means saturated
KV cache usage (vllm:kv_cache_usage_perc, gpu_cache_usage_perc on older versions)	Near 1.0 causes preemption
vllm:num_preemptions_total	Rising means too little KV memory for the concurrency
vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds (names vary by version)	Latency SLO inputs
vllm:prompt_tokens_total, vllm:generation_tokens_total	Throughput

vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions --model llama-8b \
  --dataset-name random --random-input-len 2000 --random-output-len 300 \
  --num-prompts 500 --max-concurrency 64

Use input/output lengths from real traffic. Report p50/p95 TTFT, inter-token latency and output tokens/s at the target concurrency, then change one flag at a time.

Gotchas

• --api-key protects /v1 routes only; still put the server behind a private network or gateway, and keep /metrics internal.
• --max-model-len larger than the KV cache can hold fails at startup with a message giving the maximum; read it instead of guessing.
• Different max-num-seqs, batching or TP can change outputs slightly even at temperature 0; tests should compare semantics, not exact strings.
• Long cold start: weight download plus CUDA graph capture. Cache weights on a volume, and set readiness probes on /health with a generous initial delay.
• Spot-check quantized checkpoints on your eval set before switching ($model-quantization).
• trust_remote_code runs repo code; pin the revision and review it.

Rollout checklist

- [ ] Image tag and model revision pinned
- [ ] max-model-len, memory utilization, max-num-seqs chosen from traffic data
- [ ] Auth and network exposure reviewed; metrics scraped
- [ ] Benchmark at target concurrency recorded (p50/p95 TTFT, ITL, tok/s)
- [ ] Eval suite passes against the served endpoint
- [ ] User confirmed before deploying to the production cluster

Report template

**Model:** Llama-3.1-8B-Instruct @ <sha>, bf16, vllm/vllm-openai:<version>, 1x L40S
**Config:** max-model-len 16384, gpu-mem 0.90, kv fp8, max-num-seqs 128
**Load (2k in / 300 out, 64 concurrent):** TTFT p50 0.31 s / p95 0.92 s; ITL p95 38 ms; 2,950 out tok/s
**Saturation:** waiting queue > 0 above ~90 concurrent; preemptions 0 at target