Grafana Loki Logs skill for Codex

Done well means: streams are selected by a handful of stable labels, queries narrow with cheap line filters before expensive parsing, metrics derived from logs are correct, and the label set stays small enough that ingestion and queries do not fall over.

Inspect first

1. Read AGENTS.md; find the shipping config (Alloy *.alloy, Promtail promtail.yaml, Fluent Bit, OTel Collector loki/otlphttp exporter) and the Loki version.
2. List labels and their cardinality before writing queries:

export LOKI_ADDR=http://loki-gateway.monitoring:80     # needs network to the cluster
logcli labels
logcli labels namespace
logcli series '{namespace="checkout"}' --analyze-labels --since=1h

3. Look at a sample of raw lines to see the format (JSON, logfmt, plain): logcli query '{namespace="checkout"}' --since=15m --limit=20.

Label design

Labels define streams; every unique label set is a separate stream and index entry.

Good labels (bounded, describe the source)	Never labels (put in the line or structured metadata)
cluster, namespace, app / service_name, env	user_id, request_id, trace_id
container, job	status_code, path, ip
level only if the shipper sets it reliably	timestamps, anything unbounded

• Aim for well under 100k active streams per tenant and a handful of labels per stream; Loki rejects pushes beyond its per-stream label limit (15 by default).
• High-cardinality values that you filter on often (trace ID, user ID) go into structured metadata (Loki 3.x with the OTLP or Alloy pipeline), which can be filtered without being a stream label.
• Changing labels later splits history; decide them once per platform.

LogQL query order

Write queries from cheapest to most expensive:

{namespace="checkout", app="api"}            # 1. stream selector: as specific as possible
  |= "payment"                               # 2. line filters: substring is cheapest
  != "healthz"
  |~ "timeout|deadline exceeded"             #    regex only when needed
  | json                                     # 3. parse only after narrowing
  | status >= 500 and duration_ms > 1000     # 4. label filters on parsed fields
  | line_format "{{.method}} {{.path}} {{.status}} {{.duration_ms}}ms"

• Parse only the fields you need: | json status="response.status", path="request.path" is cheaper than full | json on wide lines.
• Use | logfmt for key=value logs, | pattern "<ip> - - [<_>] \"<method> <path> <_>\" <status> <size>" for fixed formats, and | regexp as the last resort.
• Filter parsing errors deliberately: | json | __error__="" drops unparseable lines; look at them first with | __error__!="" to make sure you are not hiding real ones.
• Structured metadata filters look like label filters after the selector: {app="api"} | trace_id="4bf92f3577b34da6a3ce929d0e0e4736".

Metric queries

# error lines per second by app
sum by (app) (rate({namespace="checkout"} |= "level=error" [5m]))

# 5xx count by status over the last hour
sum by (status) (count_over_time({app="api"} | json | status >= 500 [1h]))

# p99 latency from a numeric field
quantile_over_time(0.99, {app="api"} | json | unwrap duration_ms | __error__="" [5m]) by (route)

# bytes per stream to find noisy sources
topk(10, sum by (app) (bytes_over_time({namespace=~".+"}[1h])))

• rate and count_over_time count lines; unwrap turns a field into a sample value for sum_over_time, avg_over_time, quantile_over_time.
• Always aggregate with sum by (...) to keep result series bounded.
• If you need this metric often, emit it as a Prometheus metric from the app instead; log-derived metrics are expensive to compute repeatedly.

Alerts from logs

Loki’s ruler evaluates LogQL in Prometheus rule format:

groups:
  - name: checkout-logs
    rules:
      - alert: CheckoutPanicLogged
        expr: sum(count_over_time({namespace="checkout"} |= "panic:" [5m])) > 0
        for: 0m
        labels: { severity: ticket }
        annotations:
          summary: "Panic logged in checkout"
          runbook_url: "https://runbooks.example.com/checkout/panic"

Use log alerts for events that only exist in logs; prefer metrics for rates and SLOs. See $alerting-rules for routing and severity.

Shipping with Alloy

Promtail is deprecated in favour of Grafana Alloy; new pipelines should use Alloy. Minimal Kubernetes pod logs to Loki:

discovery.kubernetes "pods" { role = "pod" }

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule { source_labels = ["__meta_kubernetes_namespace"]           target_label = "namespace" }
  rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] target_label = "app" }
  rule { source_labels = ["__meta_kubernetes_pod_container_name"]  target_label = "container" }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.drop_probes.receiver]
}

loki.process "drop_probes" {
  stage.drop { expression = ".*(GET /healthz|GET /readyz).*" }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint { url = "http://loki-gateway.monitoring/loki/api/v1/push" }
}

Check with alloy fmt and the Alloy UI’s component graph. alloy convert translates Promtail configs as a starting point; review its output rather than trusting it blind.

Troubleshooting slow or failing queries

Error or symptom	Cause	Fix
max entries limit	Too many lines returned	Add filters or raise --limit only for export
too many outstanding requests / timeouts	Broad selector over long range	Narrow labels and time; add line filter before parser
maximum of series reached	Metric query grouped by high-cardinality field	Group by fewer labels
Empty result but logs exist	Wrong label name, or parse error filtered out	Check logcli series, inspect __error__
stream limit exceeded on push	Too many label combinations	Remove unbounded labels at the shipper

Verification

• Run each query with logcli query ... --stats (or Grafana’s query inspector) and check bytes processed; a dashboard query scanning tens of GB per refresh needs a narrower selector or a recording rule.
• After label changes, logcli series --analyze-labels shows stream counts within budget.
• For alerts, backfill-check by running the expression over a past incident window.

Output

Deliver the LogQL with a one-line explanation per stage, the label set with expected cardinality, and the shipper config diff when labels or drops changed.

Gotchas

• |= is case-sensitive; use |~ "(?i)error" when case varies.
• level labels added by some shippers guess from text and are often wrong; filter on parsed fields when it matters.
• Logs arriving out of order beyond the accepted window are rejected; check clock sync on hosts.
• Retention is set per tenant or stream in Loki’s limits_config, not per query; deleting logs uses the compactor’s delete API and is irreversible, so confirm with the user.