Distributed Tracing skill for Codex

Done well means: you go from a symptom (slow p99, error spike, a customer’s request ID) to the specific span and code path responsible, with evidence from several traces, not one anecdote; and the tracing setup keeps the traces you will need next time.

Inspect first

1. Read AGENTS.md; identify the tracing backend and how to query it (Grafana Tempo with TraceQL, Jaeger UI/API, vendor UI). Querying needs network access to the backend.
2. Confirm propagation format (W3C traceparent is the default) and sampling setup: head ratio in SDKs, tail sampling in a Collector, or vendor-side.
3. Get the symptom precisely: which endpoint, which time window, which percentile, which customers or regions. “Checkout is slow” is not enough.

Find the right traces

Starting point	How to get traces
Latency spike on a metric graph	Exemplars on the histogram panel (Grafana) link to trace IDs from that bucket
Error log line	trace_id field in the log; open it directly
Customer report with a request ID	Search logs for the ID, take trace_id
General slowness	Query the backend for slow traces of that endpoint

TraceQL (Tempo) examples:

{ resource.service.name = "checkout" && span.http.route = "/api/orders" && duration > 2s }

{ resource.service.name = "checkout" && status = error }

{ span.db.system.name = "postgresql" && duration > 500ms } | count() > 10

{ resource.service.name = "api-gateway" } >> { resource.service.name = "inventory" && status = error }

The >> operator selects descendants, so the last query finds gateway requests where inventory failed somewhere underneath. Pick at least five slow traces and five normal ones for the same endpoint; comparing them is the method.

Reading a waterfall

Work through these questions in order:

1. Critical path. Which chain of spans determines the end time? Shortening a span off the critical path changes nothing.
2. Gaps. Time inside a parent with no child span means uninstrumented work: CPU, lock waits, connection pool waits, GC, or a missing instrumentation library. Add a span or profile there.
3. Sequential calls that could be parallel. A staircase of sibling spans to independent services.
4. N+1. Dozens or hundreds of identical short DB or HTTP spans under one parent. See $n-plus-one-queries.
5. Retries and timeouts. Repeated client spans to the same target, often ending at a round number (1 s, 5 s, 30 s) that matches a configured timeout.
6. Queueing. A large gap between a producer span and the consumer span (linked, not parented) means backlog, not slow processing.
7. Fan-out amplification. One request causing calls to many shards or services, where the slowest one dominates (tail at scale).
8. Errors. The deepest span with error status is usually the origin; parents just propagate it. Check whether the error was retried and recovered.

Compare slow and fast traces side by side: which span grows? If every span is proportionally slower, suspect the host (CPU throttling, noisy neighbour, GC) rather than any one dependency; check $kubernetes-observability metrics for that node.

Clock skew and odd shapes

• A child span that starts before its parent or ends after it means clocks differ between hosts; durations within one span are still reliable, cross-host offsets are not.
• Very short client spans with a long server span (or the reverse) suggest the server is timing something other than the request, or a proxy is buffering.
• Missing root span (trace starts at an internal service) means the edge proxy or gateway is not instrumented or not propagating.

Fixing broken propagation

Symptoms: traces end at a service boundary, the downstream service starts new root traces, or async work is detached.

Boundary	Typical cause	Fix
HTTP client	Custom client not instrumented	Use the instrumented client or inject traceparent manually
Reverse proxy, API gateway	Strips unknown headers or starts its own trace format	Allow traceparent/tracestate; enable OTel in nginx/Envoy/the gateway
Message queue	Context not in message headers/attributes	Inject on publish, extract on consume, link spans
Thread pools, executors, goroutines	Context not passed	Pass ctx (Go), wrap executors (Java agent does most), use context.with() (Node)
Mixed formats	One side B3, other side W3C	Configure both propagators during migration

Test propagation with one request and check that every service shows the same trace ID and correct parent IDs; see $opentelemetry for SDK setup.

Keep the traces you need: tail sampling

Head sampling at 1-10% discards most rare errors. A Collector tier with tail sampling keeps all errors and slow traces plus a baseline:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

All spans of a trace must reach the same Collector instance: put a first tier using the loadbalancing exporter with routing_key: traceID in front of the tail-sampling tier. Size decision_wait above your longest normal trace duration and num_traces for traces per second times decision wait, with headroom; watch the Collector’s dropped spans metrics.

Workflow

• Pin down the symptom: endpoint, window, percentile, segment
• Collect 5+ slow or failing traces and 5 normal ones
• Walk the waterfall questions; note the span and pattern that differs
• Confirm with metrics (is this span’s latency up across all traffic?)
• Confirm with code or logs at that span; form one hypothesis
• Fix, deploy, compare the same trace query before and after
• Fix propagation or sampling gaps discovered along the way

Verification

After the fix, rerun the same TraceQL or search for the same window length and compare span duration distributions, and check the RED metrics for the endpoint. One fast trace is not proof.

Output

## Trace investigation: <symptom>
- Window and scope: <time, endpoint, percentile>
- Traces examined: <ids, slow vs normal>
- Finding: <span / pattern> on the critical path accounts for <x of y ms>
- Evidence: <metric or log confirming beyond the sampled traces>
- Cause: <code path, config, dependency>
- Fix and result: <change, before/after numbers>
- Instrumentation gaps found: <propagation breaks, missing spans, sampling>

Gotchas

• Sampled traces are biased to whatever the sampler keeps; do not infer rates or percentages from trace counts. Use metrics for “how often”.
• Auto-instrumentation creates spans for libraries it knows; a slow custom TCP client appears as a gap, not a span.
• Span durations from the browser include network and queuing on the client device; compare with the server span before blaming the backend.