AWS CloudWatch skill for Codex

Done well means: investigations start with a targeted Logs Insights query rather than scrolling streams, every log group has a retention policy, custom metrics are emitted cheaply and with bounded dimensions, and alarms fire on sustained symptoms with deliberate missing-data behaviour and an action that reaches a human.

Inspect first

1. Read AGENTS.md; find IaC for alarms and log groups (rg -n "AWS::CloudWatch|aws_cloudwatch|cloudwatch.Alarm|LogGroup").
2. Confirm account, region and identity before running anything: aws sts get-caller-identity and aws configure get region. CLI calls need network and credentials from the user’s environment.
3. Read-only exploration is safe; anything that creates, changes or deletes alarms, log groups, retention or subscriptions in a shared account needs the user’s confirmation and should go through IaC where the repo uses it.

Logs: find things fast

aws logs describe-log-groups --log-group-name-prefix /aws/lambda/checkout \
  --query 'logGroups[].{name:logGroupName,retention:retentionInDays,storedBytes:storedBytes}'
aws logs tail /aws/lambda/checkout-api --since 15m --follow --format short
aws logs tail /ecs/checkout --since 1h --filter-pattern '"ERROR"'

Logs Insights, for anything more than tailing:

fields @timestamp, @message, @logStream
| filter @message like /(?i)error|exception/
| sort @timestamp desc
| limit 50

# JSON logs: fields are auto-discovered
fields @timestamp, level, route, status, duration_ms, trace_id
| filter status >= 500
| stats count(*) as errors by route, bin(5m)
| sort errors desc

# Lambda cold starts and duration percentiles
filter @type = "REPORT"
| stats count(*) as invocations,
        sum(strcontains(@message, "Init Duration")) as cold_starts,
        pct(@duration, 50) as p50, pct(@duration, 99) as p99,
        max(@maxMemoryUsed / 1000 / 1000) as max_mem_mb
  by bin(15m)

# Unstructured lines: extract with parse
parse @message "* * * *ms" as method, path, status, latency
| filter status like /5\d\d/
| stats avg(latency), count(*) by path

CLI form (asynchronous):

qid=$(aws logs start-query --log-group-names /ecs/checkout \
  --start-time $(( $(date +%s) - 3600 )) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | limit 20' \
  --output text --query queryId)
aws logs get-query-results --query-id "$qid"

Logs Insights bills per GB scanned; restrict log groups and time range, and filter early.

Logs: retention and cost

• New log groups default to never expire. Set retention on every group, in IaC: aws logs put-retention-policy --log-group-name /ecs/checkout --retention-in-days 30 (confirm before applying; shortening retention deletes older data).
• Ingestion is usually the largest CloudWatch line item. Reduce at the source: drop debug logs in production, skip health check access logs, sample verbose success logs.
• The Infrequent Access log class costs less to ingest but supports fewer features; choose it at creation for audit-style logs you rarely query.
• Ship to S3 (subscription to Firehose) for long retention instead of keeping years in CloudWatch.
• Find the biggest groups: sort describe-log-groups by storedBytes, and check IncomingBytes per log group in the AWS/Logs namespace.

Custom metrics

Method	When	Notes
Embedded Metric Format (EMF)	Lambda, ECS, anything already logging JSON	Metrics extracted from log lines asynchronously; no API calls in the hot path
PutMetricData	Low-volume metrics from scripts or batch jobs	Batch values; one call per datapoint is slow and costly
Metric filters on logs	Counting patterns in existing logs	Cheap for simple counts; no percentiles from text
CloudWatch agent / ADOT	Host metrics, StatsD, Prometheus scraping, OTLP	Use for EC2 and container workloads

EMF example (one JSON log line produces two metrics):

{
  "_aws": {
    "Timestamp": 1767225600000,
    "CloudWatchMetrics": [{
      "Namespace": "Shop/Checkout",
      "Dimensions": [["Service", "Route"]],
      "Metrics": [{ "Name": "Latency", "Unit": "Milliseconds" }, { "Name": "Errors", "Unit": "Count" }]
    }]
  },
  "Service": "checkout", "Route": "/orders", "Latency": 182, "Errors": 0, "trace_id": "..."
}

Each unique combination of namespace, metric name and dimension values is a separately billed custom metric. Never use request IDs, user IDs or raw paths as dimensions; keep those as plain JSON fields so they are searchable in Logs Insights but not metrics. Powertools for AWS Lambda (Python, TypeScript, Java, .NET) has an EMF metrics utility.

Alarms

aws cloudwatch put-metric-alarm \
  --alarm-name checkout-api-5xx-high \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/checkout/abc123 \
  --statistic Sum --period 60 \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 25 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page \
  --ok-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

• “M out of N” (--datapoints-to-alarm 3 of --evaluation-periods 5) avoids flapping on single spikes.
• Prefer ratios via metric math (m1/m2 with --metrics JSON) over raw counts so traffic changes do not move the threshold.
• treat-missing-data: notBreaching for error counts (no data = no errors), breaching for heartbeats and “should always report” metrics, missing to hold state.
• Composite alarms combine conditions (ALARM(errors) AND ALARM(latency)) and can suppress children during a known parent outage.
• Anomaly detection alarms suit metrics with strong daily patterns; give them two weeks of history before trusting them.
• Alarm actions go to SNS, then to the paging tool. Test with aws cloudwatch set-alarm-state --alarm-name ... --state-value ALARM --state-reason test, which really notifies; warn the on-call person before doing it.

Verification

• Every new Logs Insights query is run over a known incident window and returns the expected lines; save useful ones as saved queries in IaC or the console.
• describe-log-groups shows no group without retention.
• EMF metrics appear in the namespace within a few minutes with the expected dimensions (aws cloudwatch list-metrics --namespace Shop/Checkout).
• Each alarm transitions in a controlled test and the notification reaches the target.

Output

A list of alarms (metric, statistic, period, M of N, threshold, missing-data choice, action), retention per log group, custom metrics with dimensions, saved queries, and expected monthly cost changes.

Gotchas

• Metrics have resolution-dependent retention (1-minute data kept 15 days, then aggregated); long-range graphs show 5-minute or 1-hour averages.
• Alarms evaluate on the period’s statistic; Average latency hides tail problems, use p99 extended statistics.
• Lambda log groups are created on first invocation unless declared in IaC; declare them so retention is set.
• Logs Insights field auto-discovery works for JSON; for other formats you must parse.