Google Kubernetes Engine skill for Codex

Done well means: the cluster is Autopilot on a release channel unless there is a written reason for Standard, pods reach GCP APIs through Workload Identity Federation with their own principals, every workload declares requests and has a PodDisruptionBudget, and upgrades happen inside a maintenance window after deprecated APIs are cleared.

Before touching a cluster

1. Read AGENTS.md and find the source of truth: Terraform, Config Connector, gcloud scripts. Change the cluster there.
2. Confirm the context. kubectl against the wrong cluster is the classic GKE incident:

gcloud container clusters list --project="$PROJECT"
gcloud container clusters get-credentials prod --region="$REGION" --project="$PROJECT"
kubectl config current-context
gcloud container clusters describe prod --region="$REGION" \
  --format='value(autopilot.enabled,releaseChannel.channel,currentMasterVersion,workloadIdentityConfig.workloadPool)'

get-credentials needs gke-gcloud-auth-plugin installed and network access. Any change to a production cluster (apply, delete, upgrade, drain) needs explicit user confirmation.

Autopilot or Standard

Requirement	Choice
Default web services, workers, jobs	Autopilot
Privileged pods, custom kernel modules, node SSH	Standard
Exact node machine types for licensing	Standard, or Autopilot with a custom ComputeClass
GPU / TPU workloads	Either; Autopilot supports GPUs via node selectors
Team wants zero node operations and pay-per-pod	Autopilot

gcloud container clusters create-auto prod --region="$REGION" --project="$PROJECT" \
  --release-channel=regular --network=prod-vpc --subnetwork=gke-nodes \
  --enable-private-nodes

Autopilot enables Workload Identity Federation, Shielded nodes and node auto-upgrade by default. Choose regular for production unless the user has a reason for stable.

Requests are the bill on Autopilot

Autopilot bills on pod requests and mutates pods without them (default is roughly 0.5 vCPU and 2 GiB). Always set requests explicitly:

spec:
  serviceAccountName: api
  nodeSelector:
    cloud.google.com/compute-class: Balanced        # or Scale-Out, Performance
  containers:
    - name: api
      image: europe-docker.pkg.dev/my-project/apps/api@sha256:...
      resources:
        requests: { cpu: 250m, memory: 512Mi, ephemeral-storage: 1Gi }
        limits:   { memory: 512Mi }

• Autopilot enforces CPU-to-memory ratios and minimums per compute class; pods outside them are adjusted, and the warning appears in kubectl describe pod.
• Fault-tolerant batch and stateless replicas can use Spot pods: nodeSelector: {cloud.google.com/gke-spot: "true"} plus a terminationGracePeriodSeconds of 25 or less (Spot preemption gives about 30 seconds).
• Rejected pods: read the admission webhook message. Common causes are hostPath, privileged: true, NET_ADMIN and host networking, all blocked by Autopilot.

Workload Identity Federation for GKE

Grant IAM roles directly to the Kubernetes service account principal. No GCP SA, no key, no node SA permissions:

kubectl create serviceaccount api -n shop

gcloud storage buckets add-iam-policy-binding gs://shop-assets \
  --role=roles/storage.objectViewer \
  --member="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT.svc.id.goog/subject/ns/shop/sa/api"

When a GCP API does not accept the direct principal, or the team already has a GCP SA, use impersonation:

gcloud iam service-accounts add-iam-policy-binding api-gsa@"$PROJECT".iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:$PROJECT.svc.id.goog[shop/api]"
kubectl annotate serviceaccount api -n shop \
  iam.gke.io/gcp-service-account=api-gsa@"$PROJECT".iam.gserviceaccount.com

• The principal identifier uses the project number; the workload pool uses the project ID.
• Every pod in the same namespace using the same KSA name shares the identity, across all clusters in the project’s fleet pool. Keep namespaces meaningful.
• Remove the old node service account grants (often Editor on the default compute SA) after workloads migrate.

Test from a pod:

kubectl run wi-test -n shop --rm -it --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"api"}}' \
  --image=google/cloud-sdk:slim -- gcloud auth print-access-token --quiet

Disruption budgets and probes

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api, namespace: shop }
spec:
  minAvailable: 2          # or maxUnavailable: 1; never a value that blocks all evictions
  selector: { matchLabels: { app: api } }

GKE respects PDBs during node upgrades only for a limited time (about an hour), then evicts anyway. A PDB that allows zero disruptions does not stop an upgrade; it only makes it slower and less predictable. Pair PDBs with readiness probes and at least 2 replicas spread across zones (topologySpreadConstraints on topology.kubernetes.io/zone).

Upgrades

GKE upgrades the control plane and nodes automatically on the release channel. Your job is to control when, and to make sure nothing breaks.

gcloud container get-server-config --region="$REGION" --flatten=channels \
  --format='table(channels.channel,channels.defaultVersion,channels.validVersions.list())'

gcloud container clusters update prod --region="$REGION" \
  --maintenance-window-start=2026-01-03T02:00:00Z --maintenance-window-end=2026-01-03T06:00:00Z \
  --maintenance-window-recurrence='FREQ=WEEKLY;BYDAY=SA,SU'

gcloud container clusters update prod --region="$REGION" \
  --add-maintenance-exclusion-name=black-friday \
  --add-maintenance-exclusion-start=2026-11-20T00:00:00Z \
  --add-maintenance-exclusion-end=2026-12-02T00:00:00Z \
  --add-maintenance-exclusion-scope=no_minor_or_node_upgrades

Pre-upgrade checklist:

- [ ] Read GKE release notes for the target minor version
- [ ] Clear deprecated API usage (GKE deprecation insights; kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis)
- [ ] Upgrade Helm charts and operators that pin removed APIs
- [ ] Upgrade a staging cluster on the same channel first
- [ ] Confirm PDBs, replicas >= 2 and readiness probes on critical workloads
- [ ] Get user confirmation for any manual upgrade of production

Manual control-plane upgrade (Standard node pools follow separately; Autopilot nodes follow automatically):

gcloud container clusters upgrade prod --region="$REGION" --master --cluster-version=1.33

GKE blocks automatic minor upgrades when it detects usage of APIs removed in the target version; it does not block manual ones.

Gotchas

• kubectl delete namespace deletes everything inside, including PVC-backed data with a Delete reclaim policy. Always confirm.
• Regional clusters spread nodes across three zones; zonal clusters have a single control plane and a maintenance outage for it.
• LoadBalancer Services create external load balancers by default. Use networking.gke.io/load-balancer-type: "Internal" or Gateway API for internal traffic.
• Images from other projects need the node or workload identity to have roles/artifactregistry.reader on that repository.
• Exclusions with no_upgrades scope are limited to about 30 days and still allow security patches to be forced at end of support.

Verification

• kubectl get pods -n shop -o wide shows Running pods across zones; kubectl describe shows no Autopilot resource adjustment warnings.
• The workload identity test pod obtains a token and can read only the intended bucket.
• gcloud container clusters describe shows the release channel, maintenance window and exclusions you set.
• After an upgrade, nodes report the new version and error rates are flat.

Report template

**Cluster:** prod (Autopilot, europe-west1, regular channel, 1.33.x)
**Identity:** shop/api -> roles/storage.objectViewer on gs://shop-assets (direct principal)
**Workloads:** api 3 replicas, requests 250m/512Mi, PDB minAvailable 2, zone spread
**Upgrades:** window Sat-Sun 02:00-06:00 UTC, exclusion black-friday 20 Nov-2 Dec
**Deprecated APIs:** none in last 30 days (deprecation insights)