Deployment Strategies skill for Codex

A safe deploy lets old and new versions run side by side, exposes the new version to a small, measurable slice of traffic first, decides promotion from metrics with thresholds written down beforehand, and rolls back in under a minute without a new build. The strategy is chosen from the service’s risk and infrastructure, not from fashion.

Inspect first

1. Read AGENTS.md, the deploy pipeline and manifests (k8s/, Helm charts, ECS task definitions, Terraform for load balancers).
2. Answer these before choosing:
   • Can two versions run at once? (DB schema, message formats, session state, caches)
   • What is the rollback today, and how long does it take?
   • Which metrics show user harm within minutes? (5xx rate, p99 latency, error logs, business events like checkouts per minute)
   • How much traffic? Canary statistics need enough requests per step.
3. Check health endpoints and readiness probes. Without a readiness signal, no strategy is safe.

Choose

Situation	Strategy
Stateless service, good tests, low risk	Rolling update
Need instant full rollback, can afford 2x capacity briefly	Blue-green
High traffic, user-facing, metrics available	Canary with automated analysis
Risky behaviour change in otherwise safe code	Feature flag, deploy dark, release gradually
Low traffic (under ~1 req/s)	Blue-green or flag; canary statistics will be noise
Stateful singleton, schema-locked app	Maintenance window or recreate, announced in advance

Deploying code and releasing behaviour are separate decisions; flags let you do the second without the first.

Rolling update (Kubernetes default)

spec:
  replicas: 6
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0          # never drop below desired capacity
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 3
          lifecycle:
            preStop:
              exec: { command: ["sleep", "10"] }   # let endpoints drain before SIGTERM handling

• Readiness must check what the pod needs to serve (config loaded, DB pool up), not downstream services that would make every pod unready at once.
• The app must handle SIGTERM: stop accepting, finish in-flight requests, exit within the grace period.
• Pair with a PodDisruptionBudget so node drains do not stack with a rollout.
• Rollback: kubectl rollout undo deployment/api and kubectl rollout status to watch.

Blue-green

• Two full environments; the router (Service selector, load balancer target group, DNS weight) points at one. Deploy to idle, smoke test it directly, switch, keep the old one warm for the rollback window (typically 30 to 60 minutes), then scale it down.
• Kubernetes: two Deployments labelled version: blue|green; switch the Service selector.
• AWS: CodeDeploy blue/green for ECS, or ALB listener rules with weighted target groups.
• DNS-based switches are slow and uneven because resolvers cache; prefer load-balancer switches.
• Background workers and cron must only run in the live colour, or jobs run twice.

Canary with automated analysis

Argo Rollouts example:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      canaryService: api-canary
      stableService: api-stable
      trafficRouting:
        nginx: { stableIngress: api }
      analysis:
        templates: [{ templateName: success-rate }]
        startingStep: 1
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      interval: 1m
      failureLimit: 2
      successCondition: result[0] >= 0.995
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="api-canary",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="api-canary"}[2m]))

• Write the gates before the deploy: for example, canary 5xx rate no more than 0.5 percentage points above stable, p99 latency no more than 20% above stable, zero new crash loops. Compare canary to stable over the same window rather than to a fixed number when traffic varies by time of day.
• Each step needs enough requests to detect the regression you care about; at 5% of 50 req/s, 10 minutes is 1,500 requests, enough to see a 1% error rate, not 0.1%.
• Flagger offers the same loop with a Canary resource for Istio, Linkerd, NGINX or Gateway API.
• Sticky sessions or per-user hashing keep a user on one version; otherwise users bounce between UIs.

Feature-flag releases

• Ship code dark behind a flag defaulted off, enable for staff, then 1%, 10%, 50%, 100%.
• Use percentage rollout keyed on a stable user or account ID, not random per request.
• Every flag gets an owner and a removal date; stale flags become untested branches.
• Hand flag implementation details to feature-flags.

Database changes: expand and contract

Two versions run at once during every non-recreate deploy, so the schema must serve both.

1. Expand: add nullable column or new table; deploy code that writes both old and new.
2. Backfill in batches.
3. Switch reads to the new shape; deploy.
4. Contract: stop writing the old shape, then drop it in a later release.

Never rename or drop a column in the same release that stops using it. Detailed patterns live in zero-downtime-migrations.

Rollback rules

• Rollback must redeploy a previous artifact, never rebuild from an older commit.
• Define triggers in the runbook: any gate failure, error budget burn over 10x for 5 minutes, or on-call judgement. Rollback first, investigate after.
• Rolling back code does not roll back data. If the new version wrote data the old one cannot read, the expand-and-contract steps above were skipped.

Confirmation

Production deploys, traffic shifts, promotions and aborts change what users see. Prepare the manifests and commands, show them, and wait for the user’s explicit go-ahead before running kubectl apply, kubectl argo rollouts promote, or any pipeline deploy against production.

Verification

• Staging rollout with a deliberately failing version (bad readiness or injected 500s) stops or auto-rolls back without user-facing errors
• Load test during a rollout shows no dropped requests (check for 502/503 at the edge)
• Rollback timed end to end and recorded
• Gates and thresholds written in the runbook

Output template

## Deployment plan: <service>
Strategy: canary (5 -> 25 -> 50 -> 100%, 10 min steps)
Gates: 5xx delta <= 0.5 pp vs stable; p99 <= 1.2x stable; no restarts
Rollback: `kubectl argo rollouts abort api` (auto on 2 failed analyses), measured 40 s
Schema: expand-only this release; contract scheduled for vX.Y+1
Prerequisites: readiness probe on /healthz/ready, preStop sleep, PDB minAvailable 4