Kubernetes Debugging skill for Codex

Gives the Codex app a display name, a one-line description and a starter prompt for this skill.

interface:
  display_name: "Kubernetes Debugging"
  short_description: "Root cause from events and exit codes, not restarts"
  default_prompt: "$kubernetes-debugging My pod is in CrashLoopBackOff, figure out why"

# Kubernetes failure playbook

Decision trees for when the triage table in SKILL.md does not settle the cause. Every
command here is read-only unless marked (ask first).

## Pending

Read the `FailedScheduling` message in `kubectl describe pod`. It lists, per reason, how
many nodes were rejected, for example `0/6 nodes are available: 3 Insufficient memory,
3 node(s) had untolerated taint {dedicated: gpu}`.

| Message fragment | Check | Fix direction |
|---|---|---|
| `Insufficient cpu` / `Insufficient memory` | `kubectl describe node` Allocated resources; sum of requests, not usage | Lower inflated requests, add nodes, or let the cluster autoscaler or Karpenter scale (check its logs/events if it did not) |
| `untolerated taint` | `kubectl get nodes -o custom-columns=N:.metadata.name,T:.spec.taints` | Add a toleration only if the pod belongs on those nodes |
| `didn't match Pod's node affinity/selector` | `kubectl get nodes --show-labels` vs pod `nodeSelector`/affinity | Fix the label or the selector |
| `pod has unbound immediate PersistentVolumeClaims` | `kubectl get pvc`, `kubectl describe pvc` | StorageClass missing or wrong, provisioner failing, zone mismatch |
| `volume node affinity conflict` | PV `nodeAffinity` zone vs available nodes | Zonal disk in a zone with no capacity; use `volumeBindingMode: WaitForFirstConsumer` |
| `didn't match pod topology spread constraints` | `whenUnsatisfiable: DoNotSchedule` with too few zones | Use `ScheduleAnyway` or add capacity in the missing zone |
| `Too many pods` | node `pods` capacity | Max pods per node reached (CNI IP limits on EKS) |
| No events at all | `kubectl get pod -o jsonpath='{.spec.schedulerName}'` | Custom scheduler not running, or a ResourceQuota/LimitRange rejected the ReplicaSet (check `kubectl describe rs`) |

Quota problems show on the ReplicaSet, not the pod: `kubectl describe rs <rs>` shows
`exceeded quota` and no pod is ever created.

## ImagePullBackOff

1. Read the exact error in Events.
2. `manifest unknown` / `not found`: tag typo or image never pushed. Check the registry.
3. `unauthorized` / `denied`: pod `imagePullSecrets` or the ServiceAccount's secrets;
   on cloud registries, node IAM role or workload identity.
4. `no matching manifest for linux/amd64`: image built for one architecture only. Build
   multi-arch with `docker buildx build --platform linux/amd64,linux/arm64`.
5. `toomanyrequests`: Docker Hub rate limit. Mirror the image or authenticate pulls.

## CrashLoopBackOff

1. `kubectl logs --previous`. Empty logs with exit 1 often means the process failed
   before logging: wrong `command`/`args`, missing binary, or permissions.
2. Exit 127 or `exec format error`: entrypoint not found or wrong architecture.
3. `permission denied` writing a file: `readOnlyRootFilesystem: true` or non-root user
   without a writable `emptyDir` at that path.
4. Config errors: compare env with `kubectl exec <pod> -- env` on a healthy sibling (ask
   first in production), or render the ConfigMap/Secret keys:
   `kubectl get cm <name> -o jsonpath='{.data}'`. Never print Secret values into chat or
   logs; check key names only: `kubectl get secret <name> -o jsonpath='{.data}' | jq 'keys'`.
5. Exit 137 or 143 with `Liveness probe failed` events: the probe killed it. The app may
   be fine but slow to start; add a startup probe instead of raising liveness delays.
6. Backoff grows to 5 minutes between restarts; after a fix, the next attempt can lag.

## OOMKilled

- Container-level OOM shows `Reason: OOMKilled` in Last State. Node-level pressure shows
  `Evicted` with a memory message instead.
- Compare `kubectl top pod --containers` against `resources.limits.memory`.
- Runtime-specific checks:
  - JVM: `-XX:MaxRAMPercentage` (default 25 percent); heap plus metaspace plus threads plus direct buffers must fit.
  - Node: `--max-old-space-size` defaults are not tied to the cgroup limit on older versions.
  - Go: set `GOMEMLIMIT` to about 90 percent of the limit.
  - Python: no heap cap; look for unbounded caches and large request bodies.
- Raising the limit is valid when the new working set is understood. If memory grows
  monotonically over hours, it is a leak: raising the limit only delays the crash.

## Readiness failing

- `describe` shows `Readiness probe failed: HTTP probe failed with statuscode: 503` or
  `connection refused`.
- `connection refused`: app listens on `127.0.0.1` instead of `0.0.0.0`, or on another port.
- Timeouts: probe `timeoutSeconds` defaults to 1 s; a probe that queries dependencies
  will time out under load.
- Readiness depending on a shared downstream takes every pod out at once; consider
  degrading instead.

## Stuck rollout

```bash
kubectl rollout status deploy/<name> -n <ns>
kubectl get rs -n <ns> -l app.kubernetes.io/name=<name>     # new RS desired vs ready
kubectl describe deploy/<name> -n <ns> | sed -n '/Conditions/,/Events/p'
```

- `ProgressDeadlineExceeded`: new pods never became ready within `progressDeadlineSeconds`.
  Debug a new pod as above.
- New ReplicaSet at 0 pods: quota or admission webhook rejection; `describe rs`.
- Argo CD or Flux reverting your manual change: the live edit is being self-healed;
  fix in Git.

## DNS

```bash
kubectl run -it --rm dnstest -n <ns> --image=busybox:1.37 --restart=Never -- \
  sh -c 'cat /etc/resolv.conf; nslookup kubernetes.default; nslookup <svc>.<ns>.svc.cluster.local'
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
```

(`kubectl run` creates a pod; in shared clusters ask before running it.)

- Short names resolve only within the same namespace; across namespaces use `<svc>.<ns>`.
- `ndots:5` makes external names try every search domain first; slow external lookups
  can be fixed with a trailing dot (`api.example.com.`) or `dnsConfig.options`.
- Alpine/musl resolvers handle search domains and TCP fallback differently; intermittent
  failures on Alpine images are a known pattern.

## NetworkPolicy

- Once any policy selects a pod for a direction (ingress or egress), everything not
  explicitly allowed in that direction is denied.
- Egress policies must allow DNS (UDP and TCP 53 to kube-dns) or every name lookup fails.
- Policies match pod and namespace labels; check `kubectl get ns --show-labels` for the
  `kubernetes.io/metadata.name` label used in namespace selectors.
- Enforcement depends on the CNI (Calico, Cilium); some managed defaults do not enforce
  policies at all. Cilium users can inspect drops with `hubble observe --verdict DROPPED`.

## Terminating forever

```bash
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.finalizers}'; echo
kubectl get node <node> -o wide                                 # NotReady?
```

- Node `NotReady`: pods stay Terminating until the node returns or is removed. For
  StatefulSets, force deletion risks two writers; get the user's explicit confirmation
  and confirm the node is really down first.
- Finalizer whose controller is gone: removing it by patch is a last resort, needs user
  confirmation, and can orphan external resources (load balancers, volumes).