Prometheus Metrics skill for Codex

Done well means: each metric answers a question someone will ask during an incident, names and units are unambiguous, labels are bounded, and every query aggregates in the right order so the number on the graph is mathematically correct.

Inspect first

1. Read AGENTS.md; find the client library (prom-client, prometheus_client, client_golang, Micrometer) and the existing registry and /metrics handler.
2. Fetch current exposition locally: curl -s localhost:8080/metrics | head -100.
3. Check scrape config or ServiceMonitor for job and instance labels; do not duplicate them as metric labels.
4. If Prometheus is reachable, check promtool --version and the server version; some features (native histograms, UTF-8 names) depend on it.

Pick the type

Question	Type	Example
How many times did X happen?	Counter	http_requests_total
What is the value right now?	Gauge	queue_depth, inflight_requests
How long / how big, with percentiles across instances?	Histogram	http_request_duration_seconds
Percentiles for a single instance only	Summary (rarely)	avoid; cannot be aggregated
Static build metadata	Gauge set to 1 with labels	build_info{version="1.4.2",commit="ab12"} 1

Never use a gauge for something that only goes up; rate() on it is wrong. Never compute averages in the app; expose _sum and _count (a histogram gives both).

Naming

• <namespace>_<what>_<unit>[_total]: checkout_payment_attempts_total, db_query_duration_seconds, cache_size_bytes.
• Base units only: seconds, bytes, ratios 0-1. Never milliseconds, never percent.
• Counters end in _total. Do not put the type or label names in the metric name.
• One metric with a label beats many metrics: http_requests_total{code="500"}, not http_500_total.

Labels

• Every label value set is a new time series. Allowed: route template, method, status class or code, outcome (success|failure), small enums. Forbidden: user ID, email, raw path, query string, trace ID, error message, timestamp.
• Use the route template (/orders/:id), never the raw URL.
• Initialise known label combinations to zero at startup so rate() and alerts see a series before the first event.
• For cardinality budgets and cleanup, see $metrics-cardinality.

Instrumentation examples

Python (prometheus_client):

from prometheus_client import Counter, Histogram

REQUESTS = Counter("http_requests_total", "HTTP requests", ["method", "route", "code"])
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency", ["method", "route"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)

def handle(req):
    with LATENCY.labels(req.method, req.route_template).time():
        resp = app(req)
    REQUESTS.labels(req.method, req.route_template, str(resp.status)).inc()
    return resp

Go (client_golang):

var latency = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "Request latency.",
    Buckets: prometheus.ExponentialBucketsRange(0.005, 10, 12),
}, []string{"method", "route"})

Node (prom-client): new client.Histogram({ name, help, labelNames, buckets }); call client.collectDefaultMetrics() once.

Bucket rule: put boundaries around your SLO threshold (if the target is 300 ms, have buckets at 0.25 and 0.5 at minimum, ideally 0.3 exactly). histogram_quantile can only interpolate between bucket edges. If the Prometheus server and client support native histograms, they remove most bucket tuning; confirm both sides before switching.

PromQL that is correct

# request rate per route
sum by (route) (rate(http_requests_total[5m]))

# error ratio: aggregate numerator and denominator separately, then divide
sum(rate(http_requests_total{code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# p99 latency per route: sum buckets by le BEFORE histogram_quantile
histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# average latency
sum(rate(http_request_duration_seconds_sum[5m])) / sum(rate(http_request_duration_seconds_count[5m]))

# fraction of requests faster than 300 ms (needs a 0.3 bucket)
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))

# events in the last day
sum(increase(payment_attempts_total[1d]))

# targets down
up == 0

Rules:

• rate() then sum(), never sum() then rate(): summing raw counters hides resets.
• The range must cover at least four scrape intervals; in Grafana use $__rate_interval.
• Averaging percentiles across instances is meaningless; aggregate buckets instead.
• increase() extrapolates, so it returns non-integers for integer counters; that is expected.
• irate() is for spiky, zoomed-in graphs, not for alerts or recording rules.
• Matching two vectors with different labels needs on()/ignoring() and, for many-to-one, group_left. Prefer aggregating both sides to the same labels.

Recording rules

Precompute anything a dashboard or alert evaluates repeatedly. Name with level:metric:operations:

groups:
  - name: http-recording
    interval: 1m
    rules:
      - record: route:http_requests:rate5m
        expr: sum by (route) (rate(http_requests_total[5m]))
      - record: route:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

Verification

curl -s localhost:8080/metrics | promtool check metrics       # naming and help lint
promtool check rules rules/*.yml
promtool test rules tests/*.yml                              # unit tests with input_series

Query the live server and sanity-check magnitudes: the request rate should match load balancer or access-log counts within a few percent; error ratio should be between 0 and

1. If a graph drops to zero on deploy and spikes after, look for a counter reset combined with the wrong aggregation order.

Output

When adding metrics, deliver a table of metric name, type, labels with allowed values, unit, and the question it answers, plus the queries or recording rules that use them.

Gotchas

• Creating a metric inside a request handler re-registers it and throws (or leaks). Define metrics at module scope.
• Multi-process Python servers (gunicorn) need prometheus_client multiprocess mode with PROMETHEUS_MULTIPROC_DIR; otherwise each scrape hits one worker.
• Label values with unescaped quotes or newlines break the exposition format.
• Absent series are not zero: rate(x[5m]) with no samples returns nothing, so alerts silently never fire. Use absent() or initialise series.
• Exposing a summary’s quantile label and then averaging it across pods is a common, wrong dashboard.