Elasticsearch Mappings skill for Codex

Done well means: every field has an explicit type chosen for how it is queried, the app reads and writes through an alias so mappings can change without downtime, analyzers are proven with _analyze before use, and queries put non-scoring conditions in filter context.

Orient first

1. Read AGENTS.md, index templates or mapping files in the repo, and the client setup.
2. Check the cluster version and distribution (needs network access to the cluster):

curl -s -u "$ES_USER:$ES_PASS" "$ES_URL"            # version.number, distribution (opensearch)
curl -s -u "$ES_USER:$ES_PASS" "$ES_URL/_cat/indices?v&s=index"
curl -s -u "$ES_USER:$ES_PASS" "$ES_URL/_cat/aliases?v"
curl -s -u "$ES_USER:$ES_PASS" "$ES_URL/products/_mapping?pretty"

Elasticsearch 8.x and 9.x enable TLS and auth by default. OpenSearch shares most mapping and Query DSL syntax but differs in vectors (knn_vector), security and some APIs. 3. Collect real queries: what users filter on, sort on, aggregate on and search full-text.

Field type decisions

Field is used for	Type	Notes
Full-text search (titles, descriptions)	text with an analyzer	Not sortable or aggregatable
Exact match, filter, sort, aggregation (status, SKU, tags, ids)	keyword	ignore_above to skip huge values
Both	text plus a keyword multi-field	name for search, name.raw for sort and facets
Numbers used in ranges or math	integer, long, scaled_float, double	Ids you only match exactly are better as keyword
Dates	date with explicit format	Store UTC
Arrays of objects queried per object	nested	Each nested object is a hidden document: costly
Arbitrary user key/values	flattened	One field in the mapping; keyword semantics only
Stored but never queried	"index": false (and doc_values: false if never sorted)	Or exclude from mapping with enabled: false on an object
Embeddings	dense_vector (dims, similarity) or semantic_text (8.15+)	OpenSearch: knn_vector

Mapping template

PUT _index_template/products
{
  "index_patterns": ["products-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "analysis": {
        "analyzer": {
          "folded": { "type": "custom", "tokenizer": "standard",
                      "filter": ["lowercase", "asciifolding"] },
          "autocomplete": { "type": "custom", "tokenizer": "standard",
                            "filter": ["lowercase", "asciifolding", "edge_3_15"] }
        },
        "filter": { "edge_3_15": { "type": "edge_ngram", "min_gram": 3, "max_gram": 15 } },
        "normalizer": { "lower": { "type": "custom", "filter": ["lowercase", "asciifolding"] } }
      }
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "sku":        { "type": "keyword" },
        "name":       { "type": "text", "analyzer": "folded",
                        "fields": {
                          "raw":     { "type": "keyword", "normalizer": "lower", "ignore_above": 256 },
                          "suggest": { "type": "text", "analyzer": "autocomplete",
                                       "search_analyzer": "folded" } } },
        "description":{ "type": "text", "analyzer": "folded" },
        "brand":      { "type": "keyword" },
        "price":      { "type": "scaled_float", "scaling_factor": 100 },
        "in_stock":   { "type": "boolean" },
        "created_at": { "type": "date", "format": "strict_date_optional_time||epoch_millis" },
        "attributes": { "type": "flattened" },
        "variants":   { "type": "nested",
                        "properties": { "color": { "type": "keyword" },
                                        "size":  { "type": "keyword" } } }
      }
    }
  }
}

• dynamic: "strict" rejects unknown fields so typos and payload drift fail loudly. Use "dynamic": false when extra fields should be kept in _source but not indexed.
• Never let dynamic mapping guess types in production: the first document decides, and a "42" string makes the field text + keyword forever.
• Edge n-grams belong at index time only; search_analyzer must not n-gram the query.
• Size shards at roughly 10-50 GB each; one primary shard is right for most indexes under that.

Test analyzers before indexing

POST products-v2/_analyze
{ "analyzer": "autocomplete", "text": "Crème Brûlée Torch" }

Check the tokens are what a user would type. For a field, use {"field": "name.suggest", "text": "..."} so the actual mapping is exercised.

Synonyms: prefer synonym_graph in a search-time analyzer so updates do not need a reindex. Elasticsearch 8.10+ has a synonyms API (PUT _synonyms/<set>) that reloads sets without closing the index.

Changing a mapping

You can add new fields and multi-fields to an existing index. You cannot change the type or analyzer of an existing field. That needs a new index and a reindex.

Change	Path
New field	PUT products/_mapping with the new property
New multi-field on existing field	Add it, then POST products/_update_by_query?conflicts=proceed&wait_for_completion=false to populate
Type, analyzer or nested change	New index plus reindex plus alias swap
Urgent fix without reindex	Runtime field in the query or mapping as a stopgap

Zero-downtime reindex

The app must already read and write through an alias (products), not a concrete index.

- [ ] 1. Create products-v2 from the updated template; set refresh_interval -1, replicas 0
- [ ] 2. POST _reindex?wait_for_completion=false&slices=auto (source products-v1)
- [ ] 3. Track: GET _tasks/<task_id>; on completion compare doc counts
- [ ] 4. Catch up writes made during the reindex (dual-write or reindex with a range query on updated_at)
- [ ] 5. Restore refresh_interval and replicas; wait for green health
- [ ] 6. Atomic alias swap
- [ ] 7. Keep v1 until verified; delete only after the user confirms

POST _reindex?wait_for_completion=false&slices=auto
{ "source": { "index": "products-v1" }, "dest": { "index": "products-v2" } }

POST _aliases
{ "actions": [
  { "remove": { "index": "products-v1", "alias": "products" } },
  { "add":    { "index": "products-v2", "alias": "products", "is_write_index": true } }
] }

Deleting the old index is irreversible; confirm with the user and make sure a snapshot exists before DELETE products-v1.

Query DSL patterns

GET products/_search
{
  "query": {
    "bool": {
      "must":   [ { "multi_match": { "query": "creme torch",
                    "fields": ["name^3", "description"], "type": "best_fields" } } ],
      "filter": [ { "term":  { "brand": "acme" } },
                  { "range": { "price": { "gte": 10, "lte": 50 } } },
                  { "term":  { "in_stock": true } } ],
      "should": [ { "term": { "name.raw": "creme brulee torch" } } ]
    }
  },
  "sort": [ "_score", { "created_at": "desc" }, { "sku": "asc" } ],
  "size": 20,
  "aggs": { "brands": { "terms": { "field": "brand", "size": 20 } } }
}

• Exact matching uses term on keyword fields. term on a text field compares against analyzed tokens and usually finds nothing; this is the most common “search is broken” bug.
• Filters (filter, must_not) do not score and are cached; put every yes/no condition there.
• Nested fields need a nested query; a plain query across nested fields matches values from different objects.
• Deep pagination: from + size is capped by index.max_result_window (10,000). Use search_after with a point in time (POST products/_pit?keep_alive=1m) and a tiebreaker sort field.
• Debug relevance with "explain": true on one hit; debug speed with "profile": true.

Verification

• GET <index>/_mapping matches the intended template (no dynamic surprises)
• _analyze output reviewed for each custom analyzer with real sample text
• Reindex: doc counts equal (GET _cat/indices/products-v*?v&h=index,docs.count after a refresh), sample docs identical
• Golden queries (10-20 real searches with expected top results) return the same or better results after the change
• _cluster/health green; slow log (index.search.slowlog.threshold.query.warn) quiet

Gotchas

• index.mapping.total_fields.limit (default 1000) exists to stop mapping explosion from dynamic keys. Raising it treats the symptom; switch those keys to flattened.
• Updating a document reindexes the whole document; frequently updated counters on large docs hurt indexing throughput.
• keyword fields are case-sensitive unless they have a normalizer.
• Aggregating on a text field needs fielddata, which uses heap heavily; aggregate on the keyword multi-field instead.
• Index templates apply only at index creation; editing a template does not change existing indexes.

Output template

**Index:** products (alias) -> products-v2 (was v1)
**Mapping changes:** name gains .suggest (edge n-gram 3-15); price long -> scaled_float; attributes -> flattened
**Reindex:** 4.2M docs, slices=auto, 18 min; counts equal; 20 golden queries unchanged or better
**Alias swap:** done at 14:05 UTC; v1 kept until user confirms deletion
**Query fixes:** term on name -> term on name.raw; filters moved out of must