Kubernetes Security skill for Codex

Done well means: every workload runs under the restricted Pod Security profile or has a written exception, no identity has more RBAC than it uses, each namespace starts from default-deny networking, only signed or digest-pinned images from approved registries run, and every policy was rolled out in warn/audit mode before enforce.

Inspect before changing

kubectl version; kubectl get nodes -o wide
kubectl get ns --show-labels | grep pod-security          # PSA labels per namespace
kubectl get clusterrolebindings -o wide | grep -E 'cluster-admin|system:masters'
kubectl get networkpolicy -A
kubectl get validatingadmissionpolicies,validatingwebhookconfigurations
kubectl get pods -A -o json > pods.json && python3 scripts/audit_pods.py pods.json --skip-ns kube-system

scripts/audit_pods.py (standard library) reads kubectl ... -o json for Pods or workload controllers and flags privileged containers, host namespaces, dangerous hostPath mounts, root users, missing drop: [ALL], missing seccomp, mutable image tags, plain-text secret env vars and missing limits. It exits 1 when a finding at or above --fail-on (default high) exists. Run it on rendered manifests in CI too: helm template . | kubectl apply --dry-run=client -o json -f - | python3 scripts/audit_pods.py -.

Also read AGENTS.md and the GitOps repo: changes go through the same Helm/Kustomize path as everything else, not kubectl edit on the cluster.

Priority order

#	Area	Why first
1	Privileged pods, hostPath to /, docker or containerd socket	Each is a node takeover
2	RBAC: cluster-admin bindings, wildcard rules, secret read access	One stolen token becomes the cluster
3	Pod Security Admission per namespace	Stops new risky pods
4	NetworkPolicy default deny	Limits lateral movement after a compromise
5	Image provenance policy	Stops unknown or tampered images
6	Secrets: encryption at rest, external store, no env-var plain text	Limits blast radius of leaks
7	Audit logging and runtime detection	Lets you see the next incident

RBAC

kubectl auth can-i --list --as=system:serviceaccount:shop:api -n shop
kubectl auth can-i get secrets -A --as=system:serviceaccount:shop:api
kubectl get rolebindings,clusterrolebindings -A -o json \
  | jq -r '.items[] | select(.subjects[]?.name=="api") | "\(.kind) \(.metadata.namespace)/\(.metadata.name) -> \(.roleRef.name)"'

• Namespaced Role plus RoleBinding by default; ClusterRole only for cluster-scoped resources or reuse across namespaces (still bound with a RoleBinding).
• No * in verbs, resources or apiGroups for app identities.
• Treat these as admin-equivalent: get/list/watch on secrets, create on pods or pods/exec, escalate, bind, impersonate, write access to rolebindings, mutatingwebhookconfigurations, nodes/proxy.
• One ServiceAccount per workload. Set automountServiceAccountToken: false on pods that never call the Kubernetes API.
• Humans get access through the cloud IAM/OIDC group mapping, not static kubeconfigs or long-lived token Secrets.

Removing a binding can break running controllers. Show the diff and the subjects affected, and confirm with the user before applying RBAC removals on a shared cluster.

Pod Security Admission

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

Rollout: warn/audit at restricted first, read the warnings (kubectl apply prints them; audit annotations land in the API audit log), fix workloads, then raise enforce. Dry-run a level against existing pods without changing anything:

kubectl label --dry-run=server --overwrite ns shop pod-security.kubernetes.io/enforce=restricted

A securityContext that passes restricted:

spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: api
      image: ghcr.io/acme/api@sha256:<digest>
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ["ALL"] }
      volumeMounts:
        - { name: tmp, mountPath: /tmp }
  volumes:
    - name: tmp
      emptyDir: {}

readOnlyRootFilesystem is not required by restricted but blocks most dropped-payload attacks; give the app emptyDir mounts for the paths it writes. Images whose USER is a name rather than a numeric UID fail runAsNonRoot checks; set runAsUser.

System namespaces (CNI, CSI drivers, node agents) often need privileged. Label those namespaces explicitly and keep app workloads out of them.

NetworkPolicy

Policies only work if the CNI enforces them (Calico, Cilium, most managed CNIs with policy enabled). Flannel alone ignores them silently. Check before relying on them.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-from-frontend, namespace: shop }
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: frontend } }
      ports:
        - { protocol: TCP, port: 8080 }

Apply default-deny last, after the allow rules, and in one namespace at a time. Forgetting DNS egress is the classic outage. Allow the ingress controller namespace and monitoring scrapers explicitly. Block egress to the cloud metadata endpoint (169.254.169.254/32) from app namespaces unless workload identity needs it.

Image policy

Native first: ValidatingAdmissionPolicy (GA since Kubernetes 1.30) with CEL.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: images-pinned-and-approved }
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        object.spec.containers.all(c, c.image.startsWith('ghcr.io/acme/') && c.image.contains('@sha256:'))
      message: "Images must come from ghcr.io/acme and be pinned by digest"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: images-pinned-and-approved }
spec:
  policyName: images-pinned-and-approved
  validationActions: [Warn, Audit]      # switch to [Deny] after a clean week
  matchResources:
    namespaceSelector:
      matchLabels: { policy.acme.io/images: enforced }

Extend the expression to initContainers and ephemeralContainers once the basic rule holds. Signature verification (cosign keyless identities, attestations) needs Kyverno verifyImages or the Sigstore policy-controller; see $supply-chain-security for signing.

Secrets

• Enable encryption at rest for Secrets (managed clusters: KMS envelope encryption option).
• Prefer an external store via External Secrets Operator or the CSI secrets driver; see $secrets-management.
• Mount secrets as files over env vars where the app allows; env vars leak into crash dumps, /proc and child processes.
• A leaked ServiceAccount token: delete the token Secret (legacy) or rotate by recreating the ServiceAccount, and review audit logs for its use. Confirm with the user first.

Audit tooling

Tool	Use
kube-bench	CIS benchmark checks for nodes and control plane (self-managed clusters)
kubescape, trivy k8s	Workload and configuration scans in cluster or CI
Kyverno / Gatekeeper policy reports	Continuous admission and background scans
Falco or Tetragon	Runtime detection (shell in container, sensitive file reads)

Installing these pulls images or binaries over the network; ask before installing into a shared cluster.

Verification

• audit_pods.py shows no critical or high findings outside documented exceptions.
• kubectl label --dry-run=server at restricted returns no warnings for target namespaces.
• From a debug pod in the namespace, blocked traffic actually fails (kubectl run np-test --rm -it --image=busybox:1.36 -n shop -- wget -qO- -T 3 http://other.svc:8080) and allowed traffic plus DNS still work.
• A deliberately bad pod (:latest image, privileged) is warned or denied by policy.
• kubectl auth can-i --list --as=<sa> matches what the app needs and nothing more.

Report template

**Scope:** cluster prod-eu (EKS 1.33), namespaces shop, payments
| Finding | Severity | Status |
|---|---|---|
| ds/log-agent mounts /var/run/docker.sock | critical | moved to logging ns, exception documented |
| sa/ci bound to cluster-admin | critical | replaced with Role deploy-shop (confirmed by user) |
| shop: no NetworkPolicies | high | default-deny + 4 allow rules applied |
| 11 containers without drop ALL | medium | fixed in Helm values |
**PSA:** shop enforce=restricted; payments warn=restricted (2 workloads pending)
**Policy:** images-pinned-and-approved in Warn mode until 2025-07-01
**Checks:** audit_pods.py 0 critical/high; NP tests pass; can-i lists reviewed