LLM Observability skill for Codex

Done well means: any production answer can be traced to its exact prompt version, model, retrieved context, tool calls, tokens, cost and latency within a minute; dashboards show cost and quality per feature and per user segment; sensitive content is masked before it leaves the process; and alerts fire on cost, latency, error and quality regressions.

What every trace must carry

Field	Why
Trace per user request; spans for LLM call, retrieval, rerank, each tool	Reconstruct the run
Model requested and model returned	Provider-side aliases change
Input/output tokens, cached tokens, reasoning tokens	Cost and cache effectiveness
Cost (computed or from the platform’s price table)	Spend per feature/user
Latency, plus time to first token for streams	UX
Prompt name + version	Tie regressions to prompt changes
user_id, session_id, feature/route, environment, release	Slicing and debugging
Retrieved doc IDs and scores	RAG debugging
Errors, finish reason, retries	Reliability
Scores: user feedback, eval or judge results	Quality over time

Choose the stack

Situation	Default
Need LLM-specific UI (traces, prompts, evals, cost) fast	Langfuse (cloud or self-hosted)
Already run an OTel pipeline (Tempo, Jaeger, Datadog, Honeycomb)	OTel GenAI instrumentation, export to it; optionally also to Langfuse via OTLP
LangChain/LangGraph-heavy team already on LangSmith	Keep LangSmith; same principles apply
Strict data residency	Self-host Langfuse or keep spans in your own OTel backend

Langfuse’s current Python SDK (v4) is built on OpenTelemetry, so spans from OTel instrumentations can land in Langfuse too. Check the installed SDK version first; v3 and v4 method names differ.

Workflow

- [ ] 1. Read AGENTS.md; find LLM call sites, frameworks and existing tracing (grep for clients)
- [ ] 2. Decide backend and data policy (what content may be stored, retention, masking)
- [ ] 3. Instrument: auto-instrument the provider SDK/framework, add spans for app steps
- [ ] 4. Attach attributes: user, session, feature, prompt version, release
- [ ] 5. Verify one end-to-end trace locally; flush in scripts and on shutdown
- [ ] 6. Build dashboards (cost, latency, errors, quality) and alerts
- [ ] 7. Wire feedback and eval scores to traces

Langfuse (Python SDK v4)

export LANGFUSE_PUBLIC_KEY=pk-lf-...  LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com   # or your self-hosted URL / EU or US region

from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()

@observe()
def retrieve(question: str) -> list[dict]:
    return vector_search(question, k=8)              # becomes a child span with input/output

@observe()
def answer(question: str, user_id: str, session_id: str) -> str:
    with propagate_attributes(user_id=user_id, session_id=session_id,
                              tags=["support-bot"], metadata={"release": RELEASE}):
        docs = retrieve(question)
        with langfuse.start_as_current_observation(
            name="answer-llm", as_type="generation", model=MODEL,
        ) as gen:
            resp = llm_call(question, docs)
            gen.update(output=resp.text,
                       usage_details={"input": resp.input_tokens, "output": resp.output_tokens})
        return resp.text

• Provider and framework integrations (the Langfuse OpenAI wrapper, LangChain callback handler, or OpenInference/OTel instrumentations) capture model calls automatically; add @observe for your own steps.
• v4 exports only Langfuse spans, spans with gen_ai.* attributes and known LLM instrumentation scopes by default. HTTP or DB spans need a should_export_span filter on the client if you want them.
• Short-lived scripts and serverless handlers: call langfuse.flush() before exit or events are lost.
• Link prompts managed in Langfuse to generations so dashboards can split by prompt version.
• On SDK v3 the equivalents are start_as_current_span / start_as_current_generation and update_current_trace; follow the installed version’s docs.

OpenTelemetry GenAI conventions

Use the GenAI semantic conventions so any backend understands the spans:

Attribute	Example
gen_ai.operation.name	chat, embeddings, execute_tool
gen_ai.provider.name (older: gen_ai.system)	openai
gen_ai.request.model / gen_ai.response.model	gpt-5-mini / dated snapshot
gen_ai.usage.input_tokens / gen_ai.usage.output_tokens	812 / 164
gen_ai.response.finish_reasons	["stop"]

Span name convention: {operation} {model}, e.g. chat gpt-5-mini. The conventions are still marked experimental; pin instrumentation versions and check attribute names when upgrading.

# OpenAI SDK auto-instrumentation (package: opentelemetry-instrumentation-openai-v2)
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
OpenAIInstrumentor().instrument()

Message content capture is off by default in the OTel GenAI instrumentations; enable it only after the data policy allows it (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true). To send OTel spans to Langfuse, point an OTLP HTTP exporter at Langfuse’s OTel endpoint (/api/public/otel) with basic auth from the key pair. JS/TS: the Vercel AI SDK emits OTel spans when telemetry is enabled ($vercel-ai-sdk).

Privacy and cost of tracing

• Decide per field: store, mask, or drop. Mask emails, phone numbers, card numbers, tokens before export (Langfuse client supports a masking function; OTel needs a span processor).
• Never log API keys or auth headers; check that HTTP instrumentation does not capture them.
• Retention: align trace retention with the privacy policy; prompts and outputs often contain personal data.
• High volume: sample (for example 10-20% of successful requests, 100% of errors and of requests with negative feedback). Keep usage/cost metrics unsampled as aggregated metrics.

Dashboards and alerts

Panel	Slice by	Alert example
Cost per day and per 1k requests	feature, model, tenant	Daily cost > 1.5x 7-day average
Tokens per request (input/output/cached)	feature, prompt version	Input tokens p95 +40% after release
Latency p50/p95, TTFT p95	model, route	TTFT p95 > 2.5 s for 15 min
Error and retry rate, finish_reason=length	provider, model	Errors > 2% for 10 min
Tool call failures	tool name	Any tool > 5% failures
Quality: thumbs-down rate, judge scores	prompt version, feature	Thumbs-down +50% day over day
Retrieval: empty results, low top score	index, tenant	Empty retrieval > 5%

Cost per successful task (not per call) is the number product owners can act on.

Feedback and evals on traces

• Return the trace ID to the client with each answer; the feedback endpoint posts a score (thumbs, 1-5, category) against that trace ID through the platform’s score API.
• Run LLM-as-judge or rule-based evaluators on a sample of production traces (faithfulness, refusal, PII in output) and store results as scores. Offline eval sets live in $llm-evals.
• Turn bad traces into regression test cases: export input, context and expected behaviour.

Gotchas

• Async and streaming: spans end when the function returns, not when the stream finishes; end the generation span in the stream’s completion callback and record TTFT separately.
• Background workers and threads lose trace context unless you propagate it (OTel context propagation or passing the trace ID explicitly).
• Double instrumentation (framework callback plus provider auto-instrumentation) produces duplicate generations and double-counted cost. Pick one per call path.
• Cost from list prices ignores discounts, batch pricing and cached-token rates; reconcile monthly with the provider invoice.
• Do not block requests on telemetry; exporters must be async with bounded queues.

Verification

• Trigger one request locally; the trace shows retrieval, LLM and tool spans with tokens, cost, model, prompt version, user and session, and masked PII.
• Kill a script without flush() once to confirm events are lost, then fix it.
• Compare a day’s traced token totals with the provider usage dashboard (within ~5%).
• Fire a synthetic alert (e.g. temporarily low threshold) and confirm it reaches on-call.

Output summary template

**Backend:** Langfuse Cloud EU (SDK v4), OTel GenAI spans from openai-v2 instrumentation
**Coverage:** /chat, /summarize, nightly digest job (flush on exit)
**Attributes:** user_id, session_id, feature, prompt name@version, release, tenant
**Privacy:** email/phone/card masking, content capture on, 30-day retention, 20% sampling of successes
**Dashboards:** cost per 1k requests by feature, TTFT p95, tool failures, thumbs-down by prompt version
**Alerts:** daily cost > 1.5x baseline, error rate > 2%, TTFT p95 > 2.5 s