Datadog APM skill for Codex

Done well means: every service appears once in the service catalog with the right env and version, a trace links to its logs and back, monitors are symptom-based and kept in code, and spend tracks traffic because ingestion, indexing and tag cardinality are deliberate choices.

Inspect first

1. Read AGENTS.md; find the tracer (rg -n "dd-trace|ddtrace|dd-java-agent|Datadog.Trace") and how the Agent runs (DaemonSet/Helm chart, ECS sidecar, host package, serverless extension).
2. Check tagging env vars in manifests: DD_SERVICE, DD_ENV, DD_VERSION, and the Kubernetes labels tags.datadoghq.com/service|env|version.
3. Look for existing monitors as code (rg -n "datadog_monitor|datadog_service_level_objective") versus monitors only in the UI.
4. If cost is the concern, ask for the Usage and Plan pages or the usage attribution export; Codex cannot see billing without it. API calls need network and keys the user provides through environment variables (DD_API_KEY, DD_APP_KEY), never pasted into code.

Instrumentation

Prefer Single Step Instrumentation (Agent-injected) on hosts and Kubernetes when the platform team already uses it; otherwise add the tracer per service:

Language	Install	Run
Python	pip install ddtrace	ddtrace-run gunicorn app:app
Node.js	npm i dd-trace	node --require dd-trace/init app.js; ESM apps add --import dd-trace/initialize.mjs
Java	download dd-java-agent.jar	java -javaagent:dd-java-agent.jar -jar app.jar
.NET, Go, Ruby, PHP	per-language tracer package	see the tracer docs; Go needs code wrappers

Unified service tagging, set by the deploy system rather than in code:

env:
  - name: DD_ENV
    valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/env']" } }
  - name: DD_SERVICE
    valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/service']" } }
  - name: DD_VERSION
    valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/version']" } }
  - name: DD_AGENT_HOST
    valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  - name: DD_LOGS_INJECTION
    value: "true"

• DD_LOGS_INJECTION=true adds dd.trace_id and dd.span_id to logs from supported loggers; logs must be JSON (or parsed by a pipeline) for the link to work.
• Datadog also accepts OTLP. If the org is standardising on OpenTelemetry, send OTLP to the Agent or a Collector instead of adding dd-trace; see $opentelemetry. Do not run both tracers on one process.

Sampling and retention

Two separate controls, often confused:

Control	What it decides	Where
Ingestion sampling	Which traces leave the app and are stored for 15 minutes live search and metrics	Tracer: DD_TRACE_SAMPLING_RULES='[{"service":"checkout","sample_rate":0.2}]'; Agent target traces per second; Ingestion Control page
Retention filters	Which ingested spans are indexed for 15 days of search	APM retention filters in the UI or Terraform

Trace metrics (hits, errors, latency) are computed before sampling, so monitors on them stay accurate at low sample rates. Keep error and high-latency retention filters, and lower the default for healthy high-volume endpoints and health checks.

Monitors

Write monitors on user-facing symptoms, per service and env, in Terraform:

resource "datadog_monitor" "checkout_error_rate" {
  name    = "[checkout] 5xx error rate above 5% in prod"
  type    = "query alert"
  query   = "sum(last_10m):sum:trace.http.request.errors{service:checkout,env:prod}.as_count() / sum:trace.http.request.hits{service:checkout,env:prod}.as_count() > 0.05"
  message = <<-EOT
    Checkout error rate is {{value}}. Runbook: https://runbooks.example.com/checkout/errors
    @pagerduty-checkout
  EOT
  monitor_thresholds {
    critical = 0.05
    warning  = 0.02
  }
  require_full_window = false
  notify_no_data      = false
  tags                = ["service:checkout", "env:prod", "team:payments"]
}

• The trace metric name depends on the integration’s operation name (trace.flask.request.hits, trace.express.request.errors, trace.http.request.*). Copy it from the service page’s metric list before writing the query.
• Use .as_count() for ratios of counts; averaging rates gives wrong ratios.
• Put notification handles and the runbook in message; use {{#is_alert}} blocks to vary text by state.
• Use SLO burn-rate alerts (datadog_service_level_objective plus SLO monitors) instead of many threshold monitors when SLOs exist; see $slo-error-budgets.
• Applying Terraform or calling the monitors API changes shared alerting; run terraform plan and get the user’s confirmation before apply.

Cost control

Cost driver	What inflates it	Lever
Custom metrics	Every unique tag value combination is a billed metric	Remove high-cardinality tags (user, request, pod name on custom metrics); Metrics without Limits to keep only queried tags
Indexed logs	Indexing every debug/info line	Exclusion filters on indexes; archive to object storage; generate metrics from logs instead of indexing
Ingested logs	Volume itself	Drop health checks and noisy debug at the agent or pipeline
APM ingested spans	Sampling at 100% on hot paths	Sampling rules per service; Agent target TPS
Indexed spans	Broad retention filters	Keep errors, slow traces, and a small sample of the rest
Hosts and containers	Agent on short-lived or tiny nodes	Check billing model for containers and serverless

Workflow: pull usage attribution by service/team, sort by cost, and fix the top three. For custom metrics, use the Metrics Summary to find metrics with the most distinct tag values, then trace each back to the code that emits it (rg -n "statsd.*<metric>").

Workflow

• Tracer or OTLP path chosen, one per process
• DD_SERVICE, DD_ENV, DD_VERSION set from deploy metadata
• Logs JSON with trace IDs injected; trace to log link checked
• Sampling rules and retention filters set intentionally
• Monitors in Terraform, symptom-based, with runbooks and owners
• Usage reviewed: top custom metrics, log indexes, span retention

Verification

• One request in staging shows a trace with correct service, env and version, and the “Logs” tab on the trace shows that request’s lines.
• terraform plan shows only intended monitor changes; after apply, use the monitor’s “Test notifications” to confirm routing.
• After cost changes, compare daily usage for the changed product for one week; the service’s monitors and dashboards must still have data.

Output

Report: services instrumented and how, tagging source, sampling and retention settings, monitors added or removed (with plan output), and estimated cost impact by driver.

Gotchas

• Version tag missing means deployment tracking and faulty-deploy detection do not work.
• Setting DD_SERVICE in code and in env differently produces two services.
• Tags on custom metrics such as customer_id can cost more than the rest of the account combined.
• Ingestion sampling at the app means rare paths may never show traces; keep error retention and consider higher rates for low-volume critical services.