Data
Elasticsearch — indexing, querying, and debugging
Fast-lookup reference for the Elasticsearch data model, mappings (text vs keyword), the Query DSL, aggregations, _cat ops APIs, and diagnosing yellow/red clusters.
Docs-checkedelasticsearchsearchindexingquery-dslcluster
Quick reference — reach for these first
Cluster health
GET _cat/health?vOne line: status green/yellow/red + unassigned shard count.Index a doc
PUT /books/_doc/1
{ "title": "Dune", "author": "Herbert", "year": 1965 }Auto-creates the index and dynamically maps new fields on first write.bool query
GET /books/_search
{ "query": { "bool": {
"must": [ { "match": { "title": "dune" } } ],
"filter": [ { "term": { "author.keyword": "Herbert" } },
{ "range": { "year": { "gte": 1960 } } } ] } } }must scores; filter is a cached yes/no — put exact + range clauses there.List indices
GET _cat/indices?v&s=index&h=health,status,index,docs.count,store.sizePer-index health, doc counts, and size; s= sorts, h= picks columns.Profile a slow search
GET /books/_search
{ "profile": true, "query": { "match": { "title": "dune" } } }Returns per-shard query timing; use _validate/query?explain to debug syntax.Index → shards (primary + replica) spread across nodes
Text equivalent: An Elasticsearch cluster of three nodes holding the index 'books', which is split into two primary shards P0 and P1, each with one replica R0 and R1. Node A holds primary P0, Node B holds primary P1 and replica R0, and Node C holds replica R1. A primary and its replica never sit on the same node, so one node can be lost without data loss. A legend explains cluster health: green means all primaries and replicas assigned, yellow means all primaries assigned but some replicas are not, and red means at least one primary is unassigned.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Check overall cluster health before anything else | GET _cat/health?v | status column reads green (yellow/red means unassigned shards — see troubleshooting). |
| Create an index with explicit mappings (avoid the text/keyword trap) | PUT /books
{ "mappings": { "properties": {
"title": { "type": "text" },
"author": { "type": "keyword" },
"year": { "type": "integer" } } } } | GET /books/_mapping shows title=text, author=keyword, year=integer. |
| Add a keyword sub-field to a text field (search AND aggregate) | PUT /articles
{ "mappings": { "properties": { "tag": { "type": "text",
"fields": { "raw": { "type": "keyword" } } } } } } | match on tag works for search; term/agg on tag.raw works for exact + facets. |
| Index a single document | PUT /books/_doc/1
{ "title": "Dune", "author": "Herbert", "year": 1965 } | Response result is "created" (first write) or "updated" (re-write of same _id). |
| Bulk index many docs in one request | POST /books/_bulk
{"index":{"_id":"2"}}
{"title":"Hyperion","author":"Simmons","year":1989}
{"index":{"_id":"3"}}
{"title":"Neuromancer","author":"Gibson","year":1984} | errors: false in the response; each item status is 200/201. |
| Full-text search with match (analyzed) | GET /books/_search
{ "query": { "match": { "title": "dune" } } } | Lowercase/partial query still hits — match runs the field analyzer on the terms. |
| Exact-value lookup with term (NOT analyzed) | GET /books/_search
{ "query": { "term": { "author": "Herbert" } } } | Hits only on the exact keyword value; term on a text field usually returns 0 (see troubleshooting). |
| Combine clauses with bool (must / filter / range) | GET /books/_search
{ "query": { "bool": {
"must": [ { "match": { "title": "dune" } } ],
"filter": [ { "range": { "year": { "gte": 1960, "lte": 1990 } } } ] } } } | Returns scored matches inside the year window; filter clauses add no score. |
| Aggregate with a terms bucket (needs keyword/numeric) | GET /books/_search
{ "size": 0, "aggs": { "by_author": {
"terms": { "field": "author" } } } } | aggregations.by_author.buckets lists doc_count per author; fails on an un-fielddata text field. |
| Inspect shard placement across nodes | GET _cat/shards/books?v&h=index,shard,prirep,state,node,unassigned.reason | Every row state=STARTED; any UNASSIGNED row names the reason to chase. |
| Ask the cluster WHY a shard is unassigned | GET _cluster/allocation/explain
{ "index": "books", "shard": 0, "primary": true } | explanation / deciders text states the cause (disk watermark, node left, allocation filter). |
| Reduce replicas to clear yellow on a single-node dev cluster | PUT /books/_settings
{ "index": { "number_of_replicas": 0 } } | GET _cat/health?v returns green; do NOT set 0 replicas in production. |
| [DESTRUCTIVE] Delete an index (irreversible data loss) | DELETE /books # removes the index and every document in it — confirm first | GET _cat/indices shows books gone; there is no undo — restore from a snapshot if needed. |
Cluster went YELLOW then RED after a node dropped — find and fix the unassigned shards
cluster-ops
1
Confirm the problem at a glance — read the one-line health summary.
GET _cat/health?v
# status node.total unassigned.shards
# red 2 32
Find which shards are unassigned and whether any are primaries (p = primary, r = replica).
GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason | grep UNASSIGNED
# orders 0 p UNASSIGNED NODE_LEFT
# orders 0 r UNASSIGNED NODE_LEFT
# books 1 r UNASSIGNED NODE_LEFT3
Ask the cluster exactly why the primary cannot be allocated — this is the decisive step.
GET _cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": true }
# "explanation": "cannot allocate because all found copies are
# stale or the node holding the in-sync copy has left the cluster"4
Root cause: the node holding orders/0's in-sync primary left. Bring that node back so its shard copy rejoins (preferred — no data loss).
# restart / re-add the departed data node, then re-check:
GET _cat/recovery?v&active_only=true
# orders 0 ... translog ... 62.3% (shard recovering onto the returned node)5
If the node is gone for good and only replicas remain unassigned (yellow, not red), free the cluster to place replicas — e.g. drop replicas on a shrunk dev cluster.
PUT /books/_settings
{ "index": { "number_of_replicas": 0 } }
# { "acknowledged": true }6
Verify: health returns to green and no shards remain unassigned.
GET _cat/health?v
# status unassigned.shards
# green 0Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Cluster status is RED | GET _cat/shards?v | grep -i unassigned (look for prirep = p) | At least one PRIMARY shard is unassigned, so part of the index is unqueryable and writes to it fail. | GET _cluster/allocation/explain for that primary; a lost node or corrupt shard may need the node back, a snapshot restore, or (last resort, accepting data loss) reroute allocate_empty_primary. |
| Cluster status is YELLOW | GET _cat/shards?v | grep UNASSIGNED (prirep = r) | All primaries are assigned (data is safe and searchable) but some REPLICAS are not — commonly a single-node cluster that cannot place a replica on a second node. | Add a data node, or on a dev box set number_of_replicas to 0; allocation/explain confirms it is only replicas. |
| term query (or exact match) returns 0 hits though the value is clearly there | GET /INDEX/_mapping — is the field type text or keyword? | A text field is analyzed (lowercased/tokenized) so term looks for an exact token and misses; keyword stores the whole value verbatim. | Use match on the text field, or term on the keyword sub-field (e.g. field.keyword); add a keyword sub-field if none exists. |
| Writes rejected / index flipped to read-only with a FORBIDDEN/12 disk error | GET _cat/allocation?v and GET _cluster/settings?include_defaults=true&flat_settings=true | grep watermark | A node crossed the flood-stage disk watermark (default 95%), so Elasticsearch set index.blocks.read_only_allow_delete to protect the node. | Free disk or add capacity, then clear the block: PUT /_all/_settings {"index.blocks.read_only_allow_delete": null}. |
| Indexing fails with 'Limit of total fields [1000] has been exceeded' (mapping explosion) | GET /INDEX/_mapping | wc -c and review whether unbounded/dynamic keys are being mapped as fields | Dynamic mapping is creating a field per unique key (e.g. user-supplied JSON keys), bloating the mapping toward the field limit and heap pressure. | Set dynamic: false or strict on the offending object, use flattened type for arbitrary key/value blobs, and raise index.mapping.total_fields.limit only as a stopgap. |
| Reindex/write fails with 'mapper_parsing_exception: failed to parse field … of type [X]' | GET /INDEX/_mapping for that field, then compare to the value being sent | A field's type was locked on first write (e.g. integer) and a conflicting value (string, or a different structure) is now arriving — mappings are immutable per field. | Fix the source value, or create a new index with the correct mapping and reindex into it (POST _reindex); you cannot change an existing field's type in place. |