What Is the _id Field in Elasticsearch?
The one value that uniquely names every document you'll ever store — and quietly decides where it lives, how fast you can fetch it, and how safe your updates are.
Every document you index into Elasticsearch needs an address — some way to say “this exact one, and not any other.” That address is the _id field. It’s a small piece of metadata, but it touches almost everything: where a document is physically stored, how fast you can retrieve it, how updates and deletes are targeted, and how Elasticsearch avoids accidentally overwriting the wrong record. This article walks through what the _id field really is, how it works internally, and how to use it well in production.
1Core Concepts
The _id field is a special metadata field that uniquely identifies a document within its index. No two documents in the same index can share the same _id. You can either supply your own _id when indexing a document (for example, using your product’s SKU or a user’s account number), or let Elasticsearch generate a random, unique one for you automatically.
Think of a hospital’s patient records. Every patient is assigned a unique medical record number the moment they’re admitted — no two patients ever share one, and every prescription, test result, or note is filed under that exact number. The _id field plays the same role for a document: it’s the one value the entire system uses to say, unambiguously, “this specific record.”
What it is: a unique string identifier for a document within a given index, stored as metadata alongside the document’s actual fields.
Why it exists: without a guaranteed-unique identifier, Elasticsearch would have no reliable way to fetch a specific document directly, apply an update to the correct record, or prevent two different documents from silently colliding into one.
Where it’s used: the Get API, Update API, and Delete API all target a document by its _id; the _id also feeds directly into the routing calculation that decides which shard a document lives on.
The _id field is the Elasticsearch equivalent of a primary key in a relational database — with one important difference: it is unique only within its own index, not across the whole cluster. Two different indices can each have a document with the same _id value without any conflict.
Auto-Generated vs. Custom _id
When you don’t specify an _id, Elasticsearch generates one automatically — a short, URL-safe, random string (technically a base64-encoded UUID variant) designed to be generated quickly and spread evenly across shards. When you do specify your own _id, you gain a predictable, meaningful identifier — often mirroring a natural key from your source system, like an order number or a username — at the cost of taking on responsibility for making sure it’s actually unique.
2Architecture & Components
The _id field isn’t just a label — it’s an active ingredient in Elasticsearch’s architecture. By default, Elasticsearch feeds the _id value into a hashing function to decide which shard a document belongs to. This means the exact same _id will always route to the exact same shard, every single time, which is what makes direct document lookups by _id so fast: Elasticsearch doesn’t need to search every shard, it can calculate exactly where to look.
graph TD
A["Document with _id = 'order-58210'"] --> B["Routing Function
hash(_id) mod num_primary_shards"]
B --> C["Result: Shard 2"]
C --> D["Primary Shard 2"]
D --> E["Replica Shard 2"]
F["GET /orders/_doc/order-58210"] --> B
_id
The unique identifier itself — always a string, even if it looks numeric.
_index
Together with _id, forms the full “address” of a document: which index, and which document within it.
_routing
An optional override that lets you control shard placement independently of _id — useful for co-locating related documents on the same shard.
_version
Tracks how many times the document identified by this _id has been changed, used for concurrency control.
Unlike almost every other field in a document, _id is not part of the mapping and does not need a type declared — it is always treated as a keyword-like exact string, searchable via a dedicated “ids” query, but not analyzed, tokenized, or broken into words the way a “text” field would be.
3Internal Working
Assignment
If you supplied an _id, Elasticsearch uses it as-is. If not, it generates a new random, unique string before the document is processed any further.
Routing Calculation
The _id (or an explicit _routing value, if provided) is hashed and combined with the number of primary shards to determine exactly which shard will store this document.
Existence Check
On that target shard, Elasticsearch checks whether a document with this _id already exists. If it does, and the operation is a plain “index,” the old document is versioned out and replaced; if the operation is explicitly “create,” the write is rejected to avoid overwriting existing data.
Storage in Lucene
Internally, Lucene (the search library Elasticsearch is built on) stores the _id as a special, dedicated field on every document, enabling fast existence checks and lookups even before the document becomes fully searchable through the normal refresh cycle.
Think of a coat-check counter at a large event. The numbered ticket you receive is the _id. Instead of searching through every rack to find your coat, the attendant reads your ticket number and walks directly to the correct rack and hook — this is exactly how a GET-by-_id request skips searching entirely and jumps straight to the right shard and document.
A GET request by _id does not use the search/relevance-scoring machinery at all — it’s a direct, near-instant lookup. A search query that filters on _id (using the “ids” query) is technically still a search operation and slightly different in cost and behavior from a plain GET.
4Data Flow & Lifecycle
Create & Index
- Index (upsert-style): if a document with this _id already exists, it’s replaced with a new version; if not, it’s created fresh.
- Create (strict): explicitly fails if a document with this _id already exists, useful for guaranteeing you never silently overwrite data.
- Bulk operations: each action in a bulk request carries its own _id, allowing thousands of targeted index/update/delete operations in one call.
Read, Update & Delete
- Get: retrieves exactly one document by its _id, bypassing search scoring entirely.
- Update: loads the current document at that _id, merges in your changes, and writes it back as a new version under the same _id.
- Delete: marks the document at that _id as deleted; its disk space is reclaimed later during a segment merge.
Because the _id is permanent for the life of a document (you cannot “rename” a document’s _id — you can only delete it and create a new one under a different _id), it’s worth choosing custom _id values carefully. Many production systems mirror a stable external identifier — an order number, a user account ID — specifically so that repeated writes for the “same” real-world entity always land on the same document, naturally supporting idempotent, safe-to-retry indexing.
5Advantages, Disadvantages & Trade-offs
Auto-Generated _id — Advantages
- Guaranteed unique with zero effort on your part.
- Designed to distribute evenly across shards, avoiding “hot shard” problems from sequential or predictable IDs.
- Fast to generate, with no coordination needed across nodes.
Custom _id — Trade-offs
- Gives you predictable, meaningful identifiers and natural idempotent writes (re-indexing the “same” record just updates it).
- But you must guarantee uniqueness yourself — a collision silently overwrites an existing, unrelated document.
- Poorly chosen custom IDs (like strictly incrementing numbers) can create uneven shard load in some scenarios.
The fundamental trade-off is control versus convenience: auto-generated IDs are effortless and safe but meaningless outside Elasticsearch, while custom IDs are meaningful and enable natural upserts but place the burden of uniqueness and good distribution squarely on you.
6Security
Document-Level Security
Role-based filters can restrict which documents a user can retrieve even by direct _id lookup — knowing an _id doesn’t bypass document-level security rules.
Avoid Embedding Secrets in _id
Because _id values often appear in URLs, logs, and API responses, sensitive information (like a full account number or an email address) should generally not be used directly as an _id if it must stay confidential.
Predictable IDs
Sequential or guessable custom IDs can make it easier for an unauthorized actor to enumerate documents by simply incrementing values, unless access controls are properly enforced independently of the ID itself.
A hotel room key card works like an _id — but the door still checks whether you’re authorized to enter, rather than trusting that anyone holding a card automatically deserves access. Similarly, knowing a document’s _id should never be treated as proof of authorization on its own; real access control still applies at the door.
7Monitoring, Logging & Metrics
Because _id sits at the center of both indexing and retrieval, several important operational signals are directly tied to it:
- Version conflict rate: a rising number of “document already exists” or optimistic-concurrency failures on the same _id often signals a race condition between multiple writers.
- Get API latency: since GET-by-_id skips search scoring, it should be consistently fast; sudden slowdowns often point to shard imbalance or resource pressure on a specific node.
- Shard size distribution: monitoring whether documents are spread evenly across shards helps catch cases where a poorly chosen custom _id pattern (like a shared prefix) is clustering documents unevenly.
- Bulk rejection rate: tracking failed bulk operations by _id can reveal systemic issues, like duplicate-key collisions from an upstream data source.
8Design Patterns & Anti-patterns
Pattern: Natural Keys as _id
Using a stable external identifier (order number, SKU, user ID) as the _id makes indexing naturally idempotent — reprocessing the same event or record simply updates the existing document instead of creating a duplicate.
Pattern: Composite _id
When a document is uniquely identified by more than one attribute (for example, a user ID plus a date), combining them into a single deterministic string (like “user123_2026-09-18”) produces a reliable, collision-resistant custom _id.
Pattern: Let Elasticsearch Generate It
When there’s no natural external key and documents are purely additive (like raw event logs), auto-generated IDs are simplest — no design effort needed, and shard distribution is handled automatically.
Sequential, Monotonically Increasing Custom IDs
Manually assigning strictly incrementing numeric IDs can, in some indexing patterns, lead to uneven shard write pressure and larger Lucene segment merge overhead compared to well-distributed IDs.
Treating _id as Sensitive or Secret Data
Relying on an _id being “hard to guess” as a security boundary, instead of enforcing real document-level and field-level access control, leaves data exposed to anyone who can guess or intercept an _id value.
9Best Practices & Common Mistakes
Best Practices
- Use a stable external identifier as _id whenever one naturally exists, to get idempotent writes for free.
- Prefer “create” operations over plain “index” when you specifically want to prevent accidental overwrites.
- Let Elasticsearch auto-generate IDs for purely additive, event-style data with no natural key.
- Combine multiple attributes into a deterministic composite _id when uniqueness depends on more than one field.
Common Mistakes
- Assuming _id is globally unique across the whole cluster, rather than unique only within a single index.
- Embedding personally identifiable or sensitive information directly as the _id value.
- Ignoring version conflicts on retries instead of handling them explicitly with retry logic.
- Trying to “rename” a document’s _id by editing it in place — this isn’t possible; it must be deleted and reindexed under the new _id.
10Real-World & Industry Examples
Order & Product IDs
Retail platforms commonly use the order number or SKU directly as the document’s _id, so reprocessing the same order (say, after a payment retry) safely updates the existing document instead of duplicating it.
Auto-Generated Log IDs
Log analytics pipelines typically let Elasticsearch auto-generate the _id for every log line, since log events are purely additive and have no meaningful natural key.
Content & Post IDs
Platforms handling user-generated posts often reuse the same post ID from their primary database as the Elasticsearch _id, keeping the search index and system of record perfectly aligned.
Composite Device+Time IDs
Sensor and device telemetry systems sometimes combine a device ID and timestamp into a composite _id, ensuring reprocessed readings for the same device and moment overwrite cleanly rather than duplicating.
11Frequently Asked Questions
12Summary and Key Takeaways
Key Takeaways
- The _id field is the unique identifier for a document, guaranteed unique only within its own index.
- You can supply your own _id or let Elasticsearch auto-generate a random, well-distributed one.
- _id directly feeds the routing hash that determines which shard a document lives on, making GET-by-_id extremely fast.
- _id is permanent — it cannot be edited in place; changing it means deleting and re-creating the document.
- Using a stable, natural external key as _id enables safe, idempotent re-indexing of the same real-world record.
- _id is never a substitute for real access control — document- and field-level security still apply regardless of who knows an _id.
- Thoughtful _id design prevents duplicate data, uneven shard load, and accidental overwrites in production systems.