ELK Stack and OpenSearch skill for Codex

Done well means: logs land in data streams with an explicit, ECS-aligned mapping; indices roll over at a sensible shard size and age out automatically; the cluster stays green with headroom on disk; and investigators can query by fields rather than grepping message text.

Inspect first

1. Read AGENTS.md; find shipper and pipeline configs (Elastic Agent or Filebeat, Logstash *.conf, Fluent Bit, OTel Collector elasticsearch exporter) and any index templates or ILM/ISM policies in the repo.
2. Confirm product and version: Elasticsearch 8.x/9.x or OpenSearch 2.x/3.x. APIs differ (ILM vs ISM, ES|QL vs PPL). curl -s $ES/ shows it.
3. Cluster health snapshot (read-only, needs network and credentials from the user’s environment, never hard-coded):

curl -s "$ES/_cluster/health?pretty"
curl -s "$ES/_cat/nodes?v&h=name,node.role,heap.percent,disk.used_percent,cpu"
curl -s "$ES/_cat/indices?v&s=store.size:desc&h=index,health,pri,rep,docs.count,store.size" | head -20
curl -s "$ES/_cat/shards?v&s=store:desc" | head -20
curl -s "$ES/_cat/allocation?v"

Index layout

Use data streams for logs (append-only, time-based), named logs-<dataset>-<namespace>, for example logs-checkout.app-prod. The Elastic Agent and recent Beats create these with built-in templates; for custom apps, define your own.

PUT _index_template/logs-checkout
{
  "index_patterns": ["logs-checkout.*-*"],
  "data_stream": {},
  "priority": 200,
  "composed_of": ["logs-mappings", "logs-settings"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-30d",
      "index.mapping.total_fields.limit": 2000
    },
    "mappings": {
      "dynamic": "false",
      "properties": {
        "@timestamp":   { "type": "date" },
        "message":      { "type": "match_only_text" },
        "log.level":    { "type": "keyword" },
        "service.name": { "type": "keyword" },
        "trace.id":     { "type": "keyword" },
        "http.response.status_code": { "type": "long" },
        "event.duration": { "type": "long" },
        "labels":       { "type": "flattened" }
      }
    }
  }
}

Mapping rules:

• Follow ECS names (service.name, log.level, trace.id, http.response.status_code) so Kibana apps and correlations work.
• keyword for anything you filter or aggregate on; match_only_text or text only for free text you search by words.
• dynamic: false (unknown fields stored in _source, not indexed) or runtime stops mapping explosions from arbitrary JSON keys. Put unpredictable key-value bags in a flattened field.
• One field, one type across all services sharing the index pattern. status as string in one app and number in another causes rejected documents or conflicts.

Lifecycle and shard sizing

PUT _ilm/policy/logs-30d
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
      "warm":   { "min_age": "3d",  "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

• Target primary shards of 10-50 GB; tiny daily shards for small services waste heap and slow the cluster. Before Elasticsearch 8.3 the rule of thumb was at most 20 shards per GB of heap; newer versions are more efficient but shard count still costs.
• Rollover by size and age, never by date-named indices written directly.
• OpenSearch uses Index State Management (_plugins/_ism/policies/<name>) with states and transitions instead of ILM phases; the concepts map one to one.
• Long retention at low cost: snapshots to object storage, searchable snapshots or a frozen tier (license-dependent), or archive raw logs to S3 and keep 7-30 days hot.

Changing retention or deleting indices removes data permanently; show the exact indices or policy diff and get explicit confirmation before any DELETE or policy change that shortens retention.

Parsing

Prefer structured JSON logs from the app (see $structured-logging); parsing text in the pipeline is fragile and CPU-hungry. When you must parse:

Ingest pipeline (runs in Elasticsearch, no extra service):

PUT _ingest/pipeline/nginx-access
{
  "processors": [
    { "grok": { "field": "message", "patterns": ["%{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:ts}\\] \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code:int} %{NUMBER:http.response.body.bytes:long}"] } },
    { "date": { "field": "ts", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"] } },
    { "remove": { "field": "ts" } }
  ],
  "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } } ]
}

Test before attaching: POST _ingest/pipeline/nginx-access/_simulate with sample docs. Logstash is worth running only for heavy enrichment, buffering or fan-out; its elasticsearch output should use data_stream => true.

Querying

KQL in Kibana Discover:

service.name : "checkout" and log.level : "error" and not url.original : *healthz*
http.response.status_code >= 500 and trace.id : *

ES|QL (Elasticsearch 8.11+) for aggregations without building a visualization:

FROM logs-checkout.*
| WHERE @timestamp > NOW() - 1 hour AND http.response.status_code >= 500
| STATS errors = COUNT(*) BY url.original
| SORT errors DESC
| LIMIT 20

OpenSearch equivalents: DQL in Dashboards and PPL (source=logs-* | where ... | stats count() by ...).

Troubleshooting

Symptom	Check	Typical fix
Cluster yellow	_cluster/allocation/explain	Replicas cannot allocate (single node: set replicas 0 for dev only)
Cluster red	Same, for primaries	Restore from snapshot or recover node; do not delete blindly
Writes blocked, flood_stage	Disk > 95%	Free space or add nodes, then the read-only block clears automatically in recent versions
Limit of total fields exceeded	Mapping size	dynamic: false or flattened; fix the source app
mapper_parsing_exception	Type conflict	Align field types; reindex or wait for rollover
Slow queries	Leading wildcards, huge time ranges, text aggregations	Use keyword fields, narrow range

Verification

• Index a sample through the real pipeline and GET <data-stream>/_search?size=1 shows parsed fields with correct types; GET <data-stream>/_mapping has no surprise fields.
• GET _data_stream/<name> shows the expected ILM/ISM policy; GET <index>/_ilm/explain shows the phase progressing.
• After a week, shard sizes are in range and cluster disk has at least 20-25% free.

Output

Report the index template, lifecycle policy, pipeline, expected daily volume and shard math (GB/day / target shard size), retention and its cost implication, and the queries or dashboards delivered.

Gotchas

• Templates apply only to new indices; a fix requires rollover (POST <stream>/_rollover).
• Kibana data views on logs-* with conflicting field types show “conflict” and break visualizations.
• Running ingest without @timestamp from the event uses ingest time, which skews every timeline after a backlog.
• Security: never expose Elasticsearch or OpenSearch ports publicly; keep auth and TLS on.