AWS ECS Fargate skill for Codex

Done well means: a bad image rolls back automatically within minutes, tasks drain connections before they stop, secrets never appear in the task definition, each task role grants only what the app calls, and scaling follows a metric that reflects real load.

Before changing anything

1. Read AGENTS.md, the Dockerfile and the IaC that defines the service (CDK, Terraform, Copilot leftovers, raw JSON). Change the IaC, not the console.
2. Inspect the live service when credentials and network are available:

aws ecs describe-services --cluster prod --services orders \
  --query 'services[0].{desired:desiredCount,running:runningCount,td:taskDefinition,deployments:deployments[].{status:status,rollout:rolloutState,running:runningCount,failed:failedTasks},events:events[:8].message}'
aws ecs describe-task-definition --task-definition orders --query taskDefinition
aws ecs list-tasks --cluster prod --service-name orders --desired-status STOPPED
aws ecs describe-tasks --cluster prod --tasks <arn> --query 'tasks[].{reason:stoppedReason,containers:containers[].{name:name,exit:exitCode,reason:reason}}'

Service events and a stopped task’s stoppedReason explain most failures.

Task definition

{
  "family": "orders",
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "512",
  "memory": "1024",
  "runtimePlatform": { "cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX" },
  "executionRoleArn": "arn:aws:iam::111111111111:role/orders-execution",
  "taskRoleArn": "arn:aws:iam::111111111111:role/orders-task",
  "containerDefinitions": [{
    "name": "app",
    "image": "111111111111.dkr.ecr.eu-west-1.amazonaws.com/orders@sha256:<digest>",
    "essential": true,
    "portMappings": [{ "containerPort": 8080, "protocol": "tcp" }],
    "environment": [{ "name": "PORT", "value": "8080" }],
    "secrets": [
      { "name": "DATABASE_URL", "valueFrom": "arn:aws:secretsmanager:eu-west-1:111111111111:secret:orders/db-AbCdEf" },
      { "name": "API_KEY", "valueFrom": "arn:aws:ssm:eu-west-1:111111111111:parameter/orders/api-key" }
    ],
    "healthCheck": {
      "command": ["CMD-SHELL", "wget -qO- http://localhost:8080/healthz || exit 1"],
      "interval": 15, "timeout": 5, "retries": 3, "startPeriod": 30
    },
    "stopTimeout": 60,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": { "awslogs-group": "/ecs/orders", "awslogs-region": "eu-west-1", "awslogs-stream-prefix": "app" }
    }
  }]
}

Field	Rule
cpu / memory	Must be a valid Fargate pair: 256 (512-2048), 512 (1-4 GB), 1024 (2-8 GB), 2048 (4-16 GB), 4096 (8-30 GB), larger sizes up to 16 vCPU / 120 GB
executionRoleArn	Used by the ECS agent: pull from ECR, write logs, fetch secrets. Needs secretsmanager:GetSecretValue / ssm:GetParameters (and kms:Decrypt for CMKs) on those ARNs only
taskRoleArn	Used by your code. Grants only what the app calls. Never reuse the execution role
image	Pin by digest or immutable tag. :latest makes rollbacks meaningless
healthCheck	The command must exist in the image; distroless images have no shell or wget, so add a tiny health binary or rely on the ALB check
stopTimeout	Up to 120 s on Fargate. The app must handle SIGTERM and finish in-flight work in this window
runtimePlatform	ARM64 is cheaper; build a multi-arch or arm64 image first

secrets values resolve at task start only; rotating a secret needs a new deployment (aws ecs update-service --force-new-deployment) to take effect.

Service and load balancer

• Run tasks in private subnets with assignPublicIp: DISABLED. They need a NAT gateway or VPC endpoints (ECR API, ECR DKR, S3 gateway, CloudWatch Logs, Secrets Manager/SSM) to pull images and secrets.
• Target group: type ip, health check path that tests the app but not its dependencies (a database blip should not kill every task). Set the deregistration delay to slightly more than your longest request, often 30 s instead of the 300 s default, which otherwise slows every deploy.
• healthCheckGracePeriodSeconds longer than app start time, or slow starters get killed in a loop.

Deployments

Need	Use
Default	Rolling update with deployment circuit breaker and rollback
Instant cutover with test traffic and fast rollback	ECS blue/green (built into the ECS deployment controller) or CodeDeploy blue/green
Zero extra capacity allowed	minimumHealthyPercent: 50, maximumPercent: 100, slower

"deploymentConfiguration": {
  "minimumHealthyPercent": 100,
  "maximumPercent": 200,
  "deploymentCircuitBreaker": { "enable": true, "rollback": true },
  "alarms": { "alarmNames": ["orders-5xx-high"], "enable": true, "rollback": true }
}

Deploy workflow:

• Image built, scanned and pushed; digest recorded
• New task definition revision registered from IaC
• User confirmed deploy target (production deploys always need a clear yes)
• aws ecs update-service --cluster prod --service orders --task-definition orders:42
• aws ecs wait services-stable --cluster prod --services orders
• Check rolloutState is COMPLETED, error rate and latency normal

Scaling and cost

• Target tracking on ECSServiceAverageCPUUtilization (60-70%) for CPU-bound services; ALB RequestCountPerTarget for request-bound ones; queue depth per task (custom metric) for workers.
• Set a minimum of 2 tasks across 2 AZs for anything user-facing.
• Fargate Spot via capacity provider strategy for interruptible work, keeping a base on on-demand:

"capacityProviderStrategy": [
  { "capacityProvider": "FARGATE", "base": 2, "weight": 1 },
  { "capacityProvider": "FARGATE_SPOT", "weight": 3 }
]

Spot tasks get a 2-minute warning: ECS emits a task state change event and sends SIGTERM. Handle SIGTERM properly or interrupted tasks drop requests.

• Compute Savings Plans cover Fargate. Right-size from Container Insights CPU/memory utilization before buying commitments.

Debugging

Symptom	Likely cause
CannotPullContainerError	No route to ECR (missing NAT/endpoints), wrong image tag/arch, execution role lacks ECR permissions
ResourceInitializationError: unable to pull secrets	Execution role lacks GetSecretValue/KMS, or no route to Secrets Manager/SSM
Essential container in task exited with exit 137	Out of memory (check reason: OutOfMemoryError) or killed after stopTimeout
Exit 1 right after start	App crash; read logs in /ecs/orders
Tasks healthy in ECS, unhealthy in target group	Health check path/port mismatch, security group blocks ALB to task port
Deployment stuck IN_PROGRESS	New tasks never pass health checks; circuit breaker off
exec format error	arm64 image on X86_64 platform or the reverse

ECS Exec into a running task (requires enableExecuteCommand: true on the service, SSM permissions on the task role, and the Session Manager plugin locally):

aws ecs execute-command --cluster prod --task <task-id> --container app \
  --interactive --command "/bin/sh"

Treat exec sessions in production as privileged access; ask before running commands that change state.

Gotchas

• The awslogs log group must exist or the execution role needs logs:CreateLogGroup and awslogs-create-group: "true". Set retention on it.
• Environment variables are visible in the task definition to anyone with describe access. Anything secret goes in secrets.
• Fargate ephemeral storage is 20 GiB by default (configurable up to 200 GiB) and is lost when the task stops. Use EFS or S3 for state.
• A service with desiredCount managed by autoscaling gets reset by IaC on every deploy unless the IaC ignores that field.
• Scaling to zero or deleting a service in production needs explicit user confirmation.

Output

Report: task definition revision and image digest, roles and the permissions changed, deployment settings, the verification results (services-stable, rollout state, health metrics), and any cost effect (task size, Spot share).