OpenTelemetry skill for Codex

Done well means: every service reports a correct service.name, one request produces one connected trace across all services it touches, span names and attributes follow semantic conventions so backends understand them, sampling is a deliberate decision, and telemetry leaves the process through a Collector rather than straight to a vendor.

Inspect first

1. Read AGENTS.md, the dependency manifest and the entrypoint. Note language, framework and runtime version (zero-code support differs by version).
2. Search for existing telemetry: rg -n "opentelemetry|@opentelemetry|dd-trace|newrelic|elastic-apm|sentry_sdk.*traces". Two tracers patching the same libraries cause duplicate or broken spans; pick one.
3. Find where telemetry should go: an existing Collector (otel-collector, Alloy, vendor agent with OTLP intake) or nothing yet. Ports: OTLP gRPC 4317, OTLP HTTP 4318.
4. Check the deployment manifests for env injection (Helm values, task definitions, the OpenTelemetry Operator’s Instrumentation resource).

Package installs need network; in a sandbox with network off, ask before installing.

Order of work

1. Zero-code instrumentation for the framework, HTTP clients, database drivers and messaging libraries. It covers 80% of the value with no code changes.
2. Resource attributes and exporter via environment variables.
3. Manual spans only around business operations the auto-instrumentation cannot see (a pricing calculation, a batch loop, a cache decision).
4. Metrics and logs: enable the SDK meter and the logging bridge so logs carry trace_id/span_id.
5. Collector for batching, filtering, sampling and fan-out to backends.

Zero-code setup by language

Language	Install	Run
Node.js	npm i @opentelemetry/api @opentelemetry/auto-instrumentations-node	node --import @opentelemetry/auto-instrumentations-node/register app.js (ESM); --require for CommonJS
Python	pip install opentelemetry-distro opentelemetry-exporter-otlp then opentelemetry-bootstrap -a install	opentelemetry-instrument python app.py
Java	download opentelemetry-javaagent.jar from the project’s GitHub releases	java -javaagent:opentelemetry-javaagent.jar -jar app.jar
.NET	OpenTelemetry.AutoInstrumentation install script, or the SDK packages with AddOpenTelemetry()	per the install script’s env vars
Go	no agent; use go.opentelemetry.io/otel SDK plus otelhttp, otelgrpc and driver wrappers	code changes required

Configure with standard environment variables, which every SDK honours:

OTEL_SERVICE_NAME=checkout-api
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=prod,service.version=2026.03.1,service.namespace=shop
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_PROPAGATORS=tracecontext,baggage
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

OTEL_EXPORTER_OTLP_ENDPOINT is a base URL; for HTTP the SDK appends /v1/traces, /v1/metrics, /v1/logs. The signal-specific variables (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) are used verbatim, so they need the full path. Mixing these up is the most common “no data” cause.

Manual spans

Node:

import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('checkout');

export async function priceCart(cart) {
  return tracer.startActiveSpan('price_cart', async (span) => {
    span.setAttribute('cart.item_count', cart.items.length);
    try {
      return await computePrice(cart);
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Python:

from opentelemetry import trace
tracer = trace.get_tracer("checkout")

def price_cart(cart):
    with tracer.start_as_current_span("price_cart") as span:
        span.set_attribute("cart.item_count", len(cart.items))
        return compute_price(cart)  # exceptions are recorded and status set on exit

Rules:

• Span names are low-cardinality: GET /orders/{id}, price_cart; never include IDs. Put IDs in attributes.
• Use semantic-convention names when one exists: http.request.method, http.response.status_code, http.route, url.full, server.address, db.system.name (older SDKs: db.system), messaging.system. Custom attributes get a namespace (cart.item_count, tenant.tier).
• Span status: server spans are ERROR for 5xx only; client spans are ERROR for 4xx and 5xx. Leave status unset on success.
• Always end spans; an un-ended span is never exported.
• Do not put PII, tokens or full SQL with literal values into attributes.

Propagation

• Default to W3C Trace Context (traceparent, tracestate) plus baggage. Use B3 only for interop with an existing Zipkin estate (OTEL_PROPAGATORS=tracecontext,baggage,b3multi).
• Auto-instrumented HTTP and gRPC clients inject headers. Custom transports (queues, job tables, WebSockets) need manual propagation.inject(context.active(), carrier) on send and propagation.extract on receive.
• For queue consumers processing batches, start a new trace per message and add a span link to the producer context rather than parenting hundreds of messages to one span.
• Baggage is sent to every downstream service, including third parties; never put secrets or personal data in it.

Collector

Minimal gateway config (otelcol-contrib):

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }
  resource:
    attributes:
      - { key: deployment.environment.name, value: prod, action: upsert }
  batch: {}
exporters:
  otlphttp/backend:
    endpoint: https://otlp.example-backend.com
    headers: { authorization: "Bearer ${env:BACKEND_TOKEN}" }
  debug: { verbosity: basic }
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, resource, batch], exporters: [otlphttp/backend] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/backend] }
    logs:    { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/backend] }

memory_limiter goes first, batch last before exporters. The old logging exporter was removed; use debug. Validate with otelcol-contrib validate --config=config.yaml. Tail-based sampling (keep all errors and slow traces) needs the tail_sampling processor and all spans of a trace reaching the same Collector instance; see $distributed-tracing.

Sampling decision

Situation	Choice
Low traffic (< ~50 req/s per service)	Sample everything (parentbased_always_on)
High traffic, cost matters	parentbased_traceidratio at 1-10% in SDKs
Must keep every error and slow request	Head sample generously, add tail sampling in a Collector tier
Downstream of another service	Always parentbased_* so you honour the caller’s decision

Metrics are not sampled; keep RED metrics (rate, errors, duration) from the SDK or from the Collector’s spanmetrics connector so dashboards stay accurate under sampling.

Verification

1. Run locally with OTEL_TRACES_EXPORTER=console (or a Collector with the debug exporter) and make one request. Confirm one trace ID across all spans and that the resource shows the right service.name.
2. Call a downstream service and confirm its spans share the trace ID and the parent span ID matches the client span.
3. Trigger an error path; confirm the span has status ERROR and an exception event.
4. Check the backend: service map shows the expected edges, no unknown_service:* entries.
5. Check overhead: compare p99 latency and CPU with instrumentation on and off under the same load; typical overhead is a few percent.

If traces are disconnected, look for a hop that drops headers (proxy, custom HTTP client, thread pool or async boundary losing context) before touching sampling.

Output

Report: what was instrumented automatically, which manual spans were added and why, the env vars per environment, sampling choice, Collector pipeline, and a trace ID from the verification run.

Gotchas

• unknown_service:node in the backend means OTEL_SERVICE_NAME was not set where the process actually runs.
• In Node, the instrumentation must load before the instrumented modules; importing express before the SDK starts silently disables its instrumentation.
• Python apps under gunicorn or uWSGI with pre-fork need the SDK initialised in each worker (post-fork hook), not only in the master.
• Lambda and other short-lived runtimes must flush the exporter before the process freezes; use the language’s Lambda layer or call forceFlush.
• Semantic conventions changed names (HTTP stabilised in 2023, database later); dashboards built on old names break after SDK upgrades. Check before upgrading.