Kubernetes Manifests skill for Codex

A manifest is production-ready when a rollout or node drain causes zero failed requests, the scheduler knows what the pod needs, a hung process gets restarted but a slow one does not, and the container cannot escalate privileges. Every field below exists for one of those reasons.

Inspect first

1. Read AGENTS.md and find how manifests are delivered: raw YAML, Kustomize (kustomization.yaml), Helm (Chart.yaml), or Argo CD. Edit the source of truth, not rendered output.
2. Check the cluster version (kubectl version) and any policy engine (Kyverno, Gatekeeper, Pod Security Admission labels on the namespace). Policies reject manifests at apply time.
3. Learn the app: port, health endpoints, startup time, memory at steady state, whether it handles SIGTERM, and whether it writes to disk.
4. Reuse existing labels and naming. Changing spec.selector on an existing Deployment is rejected (immutable); it requires delete and recreate, which is an outage.

kubectl against a real cluster needs credentials and network. Offline, validate with kubeconform schemas if present; otherwise ask the user to run the server-side checks.

Reference Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  labels: &labels
    app.kubernetes.io/name: orders
    app.kubernetes.io/part-of: shop
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/name: orders
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels: *labels
    spec:
      serviceAccountName: orders
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 40
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app.kubernetes.io/name: orders }
      containers:
        - name: orders
          image: registry.example.com/shop/orders:1.14.2
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: GOMEMLIMIT
              valueFrom:
                resourceFieldRef: { resource: limits.memory }
          envFrom:
            - configMapRef: { name: orders-config }
            - secretRef: { name: orders-secrets }
          resources:
            requests: { cpu: 250m, memory: 384Mi }
            limits: { memory: 384Mi }
          startupProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 2
            failureThreshold: 60          # up to 120 s to start
          readinessProbe:
            httpGet: { path: /ready, port: http }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              sleep: { seconds: 10 }      # 1.30+; older clusters: exec ["sleep","10"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: tmp
          emptyDir: { sizeLimit: 256Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector: { app.kubernetes.io/name: orders }
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels: { app.kubernetes.io/name: orders }

Probes

Probe	Answers	Endpoint should check	Never
startup	Has the process finished booting?	Same as liveness	Omit it and compensate with a huge initialDelaySeconds on liveness
readiness	Should this pod get traffic right now?	Can serve: warmed caches, required downstreams reachable	Fail it forever; a never-ready pod blocks the rollout
liveness	Is the process wedged beyond recovery?	Event loop / main thread responds	Check the database; a DB blip then restarts every pod at once

Liveness and readiness are disabled until the startup probe succeeds. Use gRPC probes (grpc: { port: 9090 }) for gRPC servers rather than shelling out to a health binary.

Resources

• Always set requests: they drive scheduling, HPA utilisation and eviction order.
• Memory: limit equals request. Memory is not compressible; overcommit means OOM kills under node pressure. Size from observed p99 working set plus 20-30 percent.
• CPU: set a request; prefer no CPU limit for latency-sensitive services because limits cause CFS throttling even on idle nodes. Set a CPU limit only when policy requires it or the workload is batch and noisy.
• Tell runtimes about their limits: GOMEMLIMIT, -XX:MaxRAMPercentage=75 for the JVM, --max-old-space-size for Node (in MB, about 75 percent of the limit).
• Guaranteed QoS (requests equal limits for CPU and memory) only for workloads that need it.

Zero-downtime rollouts and drains

1. maxUnavailable: 0 with a surge keeps capacity during rollout.
2. Readiness gates traffic to new pods.
3. On termination, endpoints are removed asynchronously. The preStop sleep keeps the old pod serving while kube-proxy, ingress controllers and load balancers catch up; then the app receives SIGTERM and must finish in-flight requests.
4. terminationGracePeriodSeconds must exceed preStop sleep plus the app’s drain time.
5. A PDB lets kubectl drain and cluster upgrades evict pods one at a time. A PDB with minAvailable equal to replicas (or maxUnavailable: 0) blocks drains forever; do not.
6. With fewer than 2 replicas a PDB cannot help. Production services run at least 2, spread across zones.

Hard rules

• Pin images by immutable tag or digest, never :latest; with latest a restart can pull different code than its siblings run. Set imagePullPolicy only when you mean it.
• Secrets come from Secret objects or an external secrets operator, never literal env values in the Deployment. Base64 in a Secret manifest is encoding, not encryption: do not commit Secret YAML with real values (see $secrets-management).
• Name ports and reference them by name in probes and Services.
• One Deployment per independently scaled process. Sidecars for proxies and log shippers only; use native sidecars (initContainers with restartPolicy: Always, 1.29+ beta, GA in 1.33) when a sidecar must start before and stop after the app.
• Use apiVersion values that the target cluster still serves. kubectl api-resources lists them; policy/v1beta1, extensions/v1beta1 and autoscaling/v2beta2 are gone.
• Applying to a production cluster is a deploy: show the diff and get the user’s confirmation before kubectl apply against anything other than a dev cluster.

Workflow

- [ ] 1. Find the delivery mechanism and edit its source files
- [ ] 2. Gather port, health endpoints, startup time, memory profile
- [ ] 3. Write Deployment, Service, PDB (plus ServiceAccount, ConfigMap as needed)
- [ ] 4. Validate offline with kubeconform
- [ ] 5. kubectl apply --dry-run=server and kubectl diff against the target namespace
- [ ] 6. With user confirmation, apply and watch the rollout
- [ ] 7. Prove a rollout causes no errors (load test or synthetic traffic during restart)

Verification

kubeconform -strict -summary -kubernetes-version 1.33.0 k8s/*.yaml
kubectl apply --dry-run=server -f k8s/ -n shop        # admission webhooks and policies run
kubectl diff -f k8s/ -n shop
kubectl rollout status deploy/orders -n shop --timeout=5m
kubectl get pdb orders -n shop                         # ALLOWED DISRUPTIONS should be >= 1
kubectl rollout restart deploy/orders -n shop          # while sending traffic; expect 0 errors

kubeconform downloads schemas on first use (network); pass -schema-location for a local mirror in air-gapped setups. CRDs need -ignore-missing-schemas or their schemas.

Output

**Objects:** Deployment, Service, PDB, ServiceAccount (namespace shop)
**Sizing:** 3 replicas, 250m / 384Mi request, 384Mi memory limit, no CPU limit
**Probes:** startup /healthz (120 s budget), readiness /ready, liveness /healthz
**Shutdown:** preStop sleep 10 s, grace 40 s; app drains on SIGTERM (verified in logs)
**Checks:** kubeconform clean, server dry-run accepted, rollout restart under load: 0 errors
**Open:** memory request based on staging; revisit after a week of production metrics