Grafana Dashboards skill for Codex

Done well means: someone paged at night can open the dashboard and, within 30 seconds and without scrolling, tell whether the service is healthy, which dependency is not, and where to click next. Every panel has a question, a unit and a reason to exist.

Inspect first

1. Read AGENTS.md; look for dashboards in the repo (rg -l '"panels"' --glob '*.json', *.jsonnet, grafana/, Helm dashboards/ ConfigMaps) and how they are deployed (file provisioning, Terraform grafana_dashboard, Grafana Operator, Git sync).
2. List data sources and their UIDs. Panels should reference data sources by variable or UID, never by display name that differs per environment.
3. Ask (or infer from the service) who uses the dashboard and for what: on-call triage, capacity review, product stakeholders. One audience per dashboard.
4. Check the metrics actually available (curl -s <prom>/api/v1/label/__name__/values or Grafana Explore). Do not design panels around metrics that do not exist.

Structure

Lay dashboards out top to bottom, from “is it broken?” to “why?”:

Row	Content	Panel types
1. Health	SLO status, error rate, p99 latency, traffic, each against its target	Stat with thresholds, small time series
2. RED per endpoint	Rate, errors, duration by route or operation	Time series, sorted table
3. Dependencies	Downstream calls: latency and errors per dependency, DB pool, cache hit ratio	Time series
4. Saturation (USE)	CPU, memory vs limit, throttling, queue depth, connection pools	Time series with limit lines
5. Deploys and context	Annotations for deploys, feature flags, incidents	Annotations, state timeline
Collapsed rows	Deep detail rarely needed	Collapsed by default to keep load fast

Keep the first screen to roughly 6-12 panels. Link to a separate drill-down dashboard rather than growing one page to 60 panels.

Panel rules

• Title states the question or the measure with scope: “p99 latency by route (5m)”.
• Set the unit (s, bytes, percentunit, reqps). A latency graph without a unit invites a 1000x misreading.
• Add thresholds that match the SLO or alert, and draw them (threshold style: line or area). A green/red stat with arbitrary thresholds is worse than none.
• Use $__rate_interval in rate() so ranges are always at least four scrape intervals.
• Use legends with {{route}}-style templates; hide legends for single-series panels.
• Time series for trends, Stat for current value vs target, Table for top-N, Heatmap for latency distributions from histogram buckets, State timeline for up/down states. Avoid pie charts and gauges for anything that changes over time.
• Same metric, same colour across panels (errors always red-ish), and do not stack series that are not parts of a whole.
• Every panel query must be correct PromQL/LogQL on its own; see $prometheus-metrics.

Variables

datasource   type: Data source, query: prometheus
cluster      label_values(up, cluster)
namespace    label_values(kube_pod_info{cluster="$cluster"}, namespace)
service      label_values(http_requests_total{namespace="$namespace"}, service)

• Chain variables so each filters the next. Use Include All with a custom all value of .* only when queries use =~.
• Avoid variables that query high-cardinality labels; they slow every dashboard load.
• Repeat rows or panels by a variable sparingly; 30 repeated panels is a slow dashboard.

Linking signals

• Data links from latency panels to Explore with Tempo/Jaeger, and exemplars enabled on histogram queries so a click on a spike opens a real trace.
• Data links from error panels to Loki or Elasticsearch filtered by the same labels and time range (${__from}, ${__to} and ${__field.labels.service}).
• Dashboard links to the runbook and to related dashboards (upstream, downstream).
• Deploy annotations from CI or from a query on build_info changes.

Dashboards as code

Choose one path and stay with it:

Situation	Tool
Few dashboards, small team	Export JSON, commit, file provisioning
Many similar dashboards	Grafonnet (Jsonnet) or the Grafana Foundation SDK (TypeScript, Go, Python)
Terraform already manages Grafana	grafana_dashboard resource with config_json = file(...)
Kubernetes with kube-prometheus-stack	ConfigMaps labelled for the Grafana sidecar (grafana_dashboard: "1")

File provisioning:

# provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: default
    type: file
    folder: Services
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Before committing exported JSON: set a stable uid, remove id and version, replace hard-coded data source references with ${datasource} or the provisioned UID, and keep the default time range sensible (last 6 h for ops). Never push to a shared production Grafana with the API or Terraform without the user confirming the target instance.

Workflow

• Write the 3-6 questions the dashboard must answer
• Confirm each needed metric exists and has bounded labels
• Build the health row first, with targets and thresholds
• Add RED, dependency and saturation rows
• Add variables, links to logs/traces and the runbook, deploy annotations
• Export as code, strip instance-specific fields, commit
• Verify against an incident or load test

Verification

• Replay a known incident time range: the health row must show the problem on the first screen, and the drill-down must point to the cause.
• Cross-check one number against an independent source (load balancer request count, database stats) to catch wrong aggregation.
• Check load time with the browser network tab or the panel inspector’s query stats; each panel query should return in well under a second over 6 h. Replace slow ones with recording rules.
• Toggle every variable value, including All, and confirm no panel errors or empties.
• Re-import the committed JSON into a scratch folder to prove it is self-contained.

Output

Deliver the dashboard file(s), the provisioning path, a short list of questions each row answers, and any recording rules added for performance.

Gotchas

• Panels showing No data after import usually reference a data source UID that does not exist on that instance.
• Mixed rate() windows across panels make the same metric disagree; standardise.
• A Stat panel’s default calculation is “last” (or “last non-null”); for error rates over a window you may want “mean” or a range query, not the final point.
• Dashboard JSON schema evolves between Grafana major versions; export from the target version and test after upgrades.