Kubernetes Observability skill for Codex

Done well means: the cluster, nodes and every workload are scraped through declarative monitors, the default alerts are tuned to the cluster instead of muted, storage and retention are sized from measured series, and the core workload signals (throttling, memory against limits, OOM kills, restarts, scheduling) are one query away.

Inspect first

1. Read AGENTS.md; find how the cluster is managed (Argo CD, Flux, Helmfile, Terraform helm_release) and whether kube-prometheus-stack or another stack (Grafana Alloy, Datadog, a managed Prometheus) is already installed.
2. Read-only checks (need cluster access from the user’s kubeconfig):

kubectl config current-context
helm list -A | rg -i "prometheus|grafana|alloy|loki"
kubectl get prometheus,alertmanager,servicemonitor,podmonitor,prometheusrule -A
kubectl get crd | rg monitoring.coreos.com

3. Confirm the context points at the intended cluster before any install or upgrade; installing into the wrong cluster is a classic incident. Helm installs and upgrades change the cluster and require the user’s explicit go-ahead.

Install or upgrade

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace -f values.yaml --version <pinned-version>

Pin the chart version and read its upgrade notes; major versions often require applying updated CRDs first (Helm does not upgrade CRDs in the crds/ folder). In GitOps setups, commit the values and let Argo CD or Flux apply them instead of running Helm by hand.

Core values:

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 45GB                      # below the volume size, leaves headroom
    replicas: 2
    resources:
      requests: { cpu: "1", memory: 8Gi }
      limits:   { memory: 12Gi }
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources: { requests: { storage: 60Gi } }
    # select monitors from all namespaces, not only those labelled with the release name
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    externalLabels: { cluster: prod-eu-1 }
alertmanager:
  alertmanagerSpec:
    replicas: 2
grafana:
  admin:
    existingSecret: grafana-admin           # never commit the admin password
  sidecar:
    dashboards: { enabled: true, label: grafana_dashboard, searchNamespace: ALL }
kube-state-metrics:
  metricLabelsAllowlist:
    - pods=[app.kubernetes.io/name,app.kubernetes.io/component]

Sizing: plan memory from active series (roughly a few KB per head series plus query headroom) and disk from samples per second x retention x ~1-2 bytes per sample after compression. Measure prometheus_tsdb_head_series after a day and adjust. For long retention or multiple clusters, remote-write to Thanos, Mimir or a managed backend rather than growing local disks.

Getting workloads scraped

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout
  namespace: checkout
spec:
  selector:
    matchLabels: { app.kubernetes.io/name: checkout }
  endpoints:
    - port: metrics          # the Service port NAME, not the number
      path: /metrics
      interval: 30s

When a target does not appear, check in this order:

1. The Prometheus resource’s serviceMonitorSelector and serviceMonitorNamespaceSelector match the monitor (kubectl get prometheus -n monitoring -o yaml). With the NilUsesHelmValues defaults left on, monitors need the label release: <helm release>.
2. The ServiceMonitor selector matches the Service labels, and the Service has a named port matching endpoints.port.
3. The Service has endpoints (kubectl get endpointslices -n checkout).
4. Prometheus has RBAC to that namespace (the chart grants cluster-wide by default).
5. Status in Prometheus UI under Status > Targets (or kubectl port-forward svc/kps-kube-prometheus-stack-prometheus 9090 -n monitoring); look for scrape errors.

Use PodMonitor for pods without a Service (jobs, DaemonSets with host ports). PrometheusRule resources need labels matching ruleSelector the same way.

Key queries

# CPU throttling ratio per container (above ~25% sustained hurts latency)
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/ sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))

# memory working set vs limit (OOM kill happens at the limit)
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/ max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})

# containers whose last termination was an OOM kill
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 0

# pods stuck pending
sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0

# requests vs allocatable: how much of the cluster is reserved
sum(kube_pod_container_resource_requests{resource="cpu"}) / sum(kube_node_status_allocatable{resource="cpu"})

container_memory_working_set_bytes is what the kubelet uses for eviction; usage includes reclaimable cache and overstates pressure.

Default alerts: tune, do not mute

The chart ships the kubernetes-mixin rules. Common adjustments:

Alert	Usual issue	Tune
KubeCPUOvercommit, KubeMemoryOvercommit	Fires on autoscaled clusters	Disable when the cluster autoscaler handles capacity
CPUThrottlingHigh	Noisy for batch and sidecars	Raise threshold or scope by namespace; route as ticket
KubePodCrashLooping, KubePodNotReady	Pages for dev namespaces	Route by namespace label to team receivers
Watchdog	Meant to always fire	Route to a dead man’s switch service, never silence
etcd*, KubeScheduler*, KubeControllerManager* on managed clusters	Control plane not scrapeable	Disable those components in values (kubeEtcd.enabled: false etc.)

Change via defaultRules.disabled or defaultRules.rules.<group>: false in values plus your own PrometheusRule replacements; avoid editing generated rules in-cluster. Route by namespace or a team label in Alertmanager; see $alerting-rules.

Verification

• Status > Targets shows every expected job up; up == 0 returns nothing unexpected.
• A test ServiceMonitor in a new namespace is picked up within a couple of minutes.
• The Watchdog alert reaches the dead man’s switch; a test PrometheusRule fires and routes.
• After 24 h, Prometheus memory is under 70% of its limit and disk usage projects within retentionSize with headroom.

Output

The values file diff, monitors and rules added, alert routing changes, sizing numbers (series, samples per second, disk and memory), and the targets/alerts verification.

Gotchas

• Two Prometheus instances (an old one and the chart’s) scraping the same targets double load and confuse dashboards.
• Monitoring namespace in the same cluster dies with the cluster; send the Watchdog to an external dead man’s switch and consider remote-write.
• kubectl top uses metrics-server, not Prometheus; numbers differ in timing and window.
• High-cardinality kube-state-metrics labels (all pod labels and annotations) are a top cause of series explosions; allowlist only what you query ($metrics-cardinality).