Kamal Deploys skill for Codex

Done well means: kamal deploy builds, pushes and swaps containers with zero downtime behind kamal-proxy, the health check reflects real readiness, secrets never enter git, stateful accessories have backups, and rollback to the previous version is a tested one-liner.

Every kamal setup, kamal deploy, kamal rollback, kamal remove or accessory reboot against a real server is a production change. Show the command and target destination, and get the user’s confirmation before running it.

Inspect first

cat config/deploy.yml config/deploy.*.yml 2>/dev/null
cat .kamal/secrets* 2>/dev/null | sed 's/=.*/=<redacted>/'   # see keys, never print values
ls .kamal/hooks/ 2>/dev/null
kamal version
kamal config -d staging          # fully resolved config for a destination
grep -n EXPOSE Dockerfile; grep -rn "listen\|PORT" config/ 2>/dev/null | head

Read AGENTS.md for the deploy process (who may deploy, which destination is default). Kamal needs SSH access to the servers, a container registry, and Docker locally or a remote builder: all network operations, so ask for approval in a Codex sandbox.

Minimal Kamal 2 config

# config/deploy.yml
service: shop
image: acme/shop

servers:
  web:
    - 203.0.113.10
  job:
    hosts:
      - 203.0.113.10
    cmd: bin/jobs

proxy:
  ssl: true
  host: shop.example.com
  app_port: 3000              # port the app listens on inside the container (default 80)
  healthcheck:
    path: /up
    interval: 2
    timeout: 3

registry:
  server: ghcr.io
  username: acme-bot
  password:
    - KAMAL_REGISTRY_PASSWORD

builder:
  arch: amd64

env:
  clear:
    RAILS_LOG_TO_STDOUT: "1"
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_URL

volumes:
  - "shop_storage:/rails/storage"

accessories:
  db:
    image: postgres:17
    host: 203.0.113.10
    port: "127.0.0.1:5432:5432"
    env:
      clear: { POSTGRES_USER: shop, POSTGRES_DB: shop_production }
      secret: [POSTGRES_PASSWORD]
    directories:
      - data:/var/lib/postgresql/data

aliases:
  console: app exec --interactive --reuse "bin/rails console"
  logs: app logs -f

.kamal/secrets (gitignored) maps names to values, pulled at deploy time:

KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD
RAILS_MASTER_KEY=$(cat config/master.key)
SECRETS=$(kamal secrets fetch --adapter 1password --account acme --from Deploy/shop POSTGRES_PASSWORD DATABASE_URL)
POSTGRES_PASSWORD=$(kamal secrets extract POSTGRES_PASSWORD $SECRETS)
DATABASE_URL=$(kamal secrets extract DATABASE_URL $SECRETS)

Adapters exist for several password managers and cloud secret stores; check kamal secrets fetch --help for the installed version. Never write literal secret values into deploy.yml or commit .kamal/secrets with values in it.

First deploy

- [ ] 1. App serves 200 on the health path over plain HTTP inside the container
- [ ] 2. DNS A/AAAA for proxy.host points at the server (needed for Let's Encrypt)
- [ ] 3. Ports 80 and 443 open on the server; SSH key access as the configured user
- [ ] 4. Registry credentials work: docker login locally with the same token
- [ ] 5. kamal config shows the expected resolved values
- [ ] 6. Confirm with the user, then: kamal setup (installs Docker if missing, boots proxy, accessories, app)
- [ ] 7. curl -I https://<host>/up returns 200 with a valid certificate

Subsequent releases: kamal deploy (or kamal deploy -d staging for a destination file config/deploy.staging.yml, which overrides the base config).

How a deploy swaps traffic

Kamal builds and pushes the image, pulls it on each server, starts the new container, waits for kamal-proxy’s health check to pass on app_port + healthcheck.path, switches traffic, then stops the old container. If the new container never becomes healthy, the old one keeps serving and the deploy fails.

“target failed to become healthy”

Check in this order:

kamal app logs --lines 200                 # did the app boot? crash on missing env?
kamal app details                          # running containers and versions
kamal proxy logs --lines 100               # health check requests and their status
kamal app exec --reuse "curl -sv http://localhost:3000/up"   # if curl exists in the image

Cause	Fix
App listens on a different port than app_port	Align app_port, Dockerfile EXPOSE, and the server’s listen port
App binds 127.0.0.1	Bind 0.0.0.0
Health path redirects to HTTPS (Rails force_ssl)	Exclude the health path from SSL redirect (config.ssl_options = { redirect: { exclude: ->(r) { r.path == "/up" } } })
Health path needs DB/auth and those are not ready	Make /up a cheap liveness check; boot migrations separately
Boot takes longer than the deploy timeout	Raise deploy_timeout, or speed up boot (precompile assets at build time)
Missing secret causes crash on boot	Check kamal app logs; verify the key is in env.secret and .kamal/secrets
Host header mismatch	Health checks go to the container directly; app-level host authorization must allow it

Day-two operations

kamal rollback <version>         # version = image tag (git SHA); list with `kamal app containers`
kamal app exec -i --reuse "bin/rails db:migrate:status"
kamal accessory logs db
kamal accessory reboot db        # restarts the database: downtime, confirm first
kamal lock status
kamal lock release               # only after confirming no deploy is running
kamal prune all                  # remove old images and containers on servers

• Rollback only swaps the app container; it does not undo database migrations. Write migrations backward-compatible (expand, deploy, contract) so the previous version runs against the new schema.
• A deploy interrupted mid-way leaves the deploy lock; check who is deploying before releasing it.
• Several apps on one server: each has its own service and proxy.host; kamal-proxy routes by host name. Accessories with published ports must not collide.
• proxy.ssl: true with Let’s Encrypt works per server. With multiple web servers, put a load balancer (with its own certificate) in front and terminate TLS there, or use a provider certificate; each server cannot obtain the same certificate independently.

Hooks

Executable scripts in .kamal/hooks/ (for example pre-build, pre-deploy, post-deploy, pre-connect) run locally with KAMAL_* environment variables. Use pre-build to refuse deploying a dirty git tree, and post-deploy to notify chat (after the user confirms the webhook destination). A hook exiting non-zero aborts the deploy.

#!/usr/bin/env bash
# .kamal/hooks/pre-build
set -euo pipefail
if [ -n "$(git status --porcelain)" ]; then
  echo "Refusing to deploy with uncommitted changes" >&2
  exit 1
fi

Gotchas

• Kamal 1 to 2: Traefik was replaced by kamal-proxy, .env by .kamal/secrets, and the default health check port changed. Follow the official upgrade guide and run kamal upgrade per destination only after confirming with the user.
• Building amd64 images on Apple Silicon uses emulation and is slow; configure a remote builder (builder.remote: ssh://...) or build in CI.
• Accessories are not managed databases: no automatic backups, failover or upgrades. Schedule dumps off-server and test a restore, or use a managed database.
• Docker-published accessory ports bypass the host firewall; bind them to 127.0.0.1 as above (see $linux-server-hardening).
• Persistent files need volumes:; anything written inside the container disappears on the next deploy.

Verification

• curl -sI https://<host>/up returns 200 and the certificate is valid (openssl s_client -connect <host>:443 -servername <host> </dev/null | openssl x509 -noout -dates).
• kamal app details shows the new version on every server and role.
• A deliberately failing health check on staging leaves the old version serving.
• kamal rollback <previous> on staging restores the previous version within a minute.

Report template

**Destination:** production (2 web hosts behind Hetzner LB, 1 job host)
**Version:** 3f9c2e1 deployed 14:05 UTC; previous a71b0d4 (rollback target)
**Health:** /up 200 via kamal-proxy; deploy took 74 s
**Changes:** app_port 3000 (was default 80), /up excluded from force_ssl
**Secrets:** fetched from 1Password via .kamal/secrets; nothing committed
**Open items:** nightly pg_dump of db accessory to object storage, restore test