What Are Metadata Fields in Elasticsearch?
A beginner-friendly, no-code walkthrough of the hidden, automatically-managed fields that travel with every document — what they are, what each one does, and why they matter even though you rarely set them yourself.
Imagine receiving a parcel in the mail. Beyond the actual gift inside, the parcel itself carries a shipping label, a tracking number, a barcode, and a postmark stamped by the courier. You didn’t write any of that information — the postal system generated it automatically to track, identify, and manage the parcel as it moved through the system. Every document you store in Elasticsearch works the same way. Alongside the actual content you send, Elasticsearch automatically attaches a small set of built-in labels called metadata fields, which describe the document itself rather than the information inside it. This post explains what these fields are, what each one is responsible for, and why they quietly matter.
1What Exactly Are Metadata Fields?
Let’s start with a clear, plain-language definition before looking at specific fields.
The simple definition
Metadata fields are special, built-in fields that Elasticsearch automatically attaches to every document, describing facts about the document itself — such as which index it lives in, its unique identifier, and how many times it has been updated — rather than describing the actual business data the document represents. They’re easy to spot because their names all begin with an underscore, like _index or _id.
Think of metadata fields like the stamps and labels on a passport — the passport number, the issuing country, the expiry date. None of that is “about you” as a person the way your name or photo is, but all of it is essential for identifying and managing that specific passport as an object.
You don’t need to manually create or manage most metadata fields — Elasticsearch generates and maintains them automatically as part of storing and retrieving your documents.
2Metadata Fields vs. Regular Fields
Understanding the distinction between “about the document” and “inside the document” clears up a lot of confusion.
| Aspect | Regular Field | Metadata Field |
|---|---|---|
| Describes | The actual business data, like a product name or price | The document itself, like where it lives or its version |
| Who defines it? | You, the application developer | Elasticsearch, automatically |
| Naming convention | Any name you choose | Always begins with an underscore |
| Appears in your JSON body? | Yes, as part of the document content | Usually shown alongside, not inside, the document body |
Metadata fields are not just “extra info you can ignore.” Several of them — especially the document’s identifier and version — are actively used by Elasticsearch to manage storage, updates, and consistency behind the scenes.
3The Most Important Metadata Fields
A quick tour of the handful of metadata fields you’ll encounter most often.
_index
Records which index the document belongs to — useful when searching across multiple indices at once.
_id
The document’s unique identifier within its index, used to retrieve, update, or delete that exact document.
_source
Holds the original JSON document exactly as it was submitted, so it can be returned in full when retrieved.
_version
Tracks how many times a document has been updated, helping manage concurrent changes safely.
_score
Appears only in search results, representing how relevant a document is judged to be for a particular query.
_routing
Controls which shard a document is stored on, allowing related documents to be grouped together for efficiency.
4Deep Dive: _index and _id
Together, these two fields form a document’s complete address inside a cluster.
Every document has a home — the index it was stored in — and a unique identifier within that index. Together, the _index and _id values act like a full mailing address: the index is the city, and the ID is the street address within it. This combination is exactly what Elasticsearch uses whenever you ask it to fetch, update, or delete one specific document directly, rather than searching for it.
If you don’t provide your own ID when creating a document, Elasticsearch will automatically generate a unique one for you.
When to choose your own ID
Providing your own meaningful ID — like an existing order number or user ID from another system — makes it simple to update or overwrite the exact same document later, rather than accidentally creating duplicates.
5Deep Dive: _source
The field that quietly makes it possible to get your original document back exactly as you sent it.
When Elasticsearch indexes a document, it doesn’t just break it apart into searchable pieces and throw away the original — it also keeps a complete, untouched copy of the original JSON you submitted, stored under _source. This is what gets returned whenever you retrieve or search for a document, letting you see the full, original data rather than just the internal search structures.
_source is like keeping the original signed copy of a contract in a filing cabinet, even after you’ve created a searchable summary card for it. If anyone ever needs the full, exact original, it’s right there.
Storing the full original document does use additional disk space. In some very storage-sensitive setups, teams choose to disable _source for certain indices, though this means you lose the ability to retrieve the original document or reindex it later.
6Deep Dive: _version and Handling Concurrent Changes
A quiet counter that prevents two updates from silently overwriting each other.
Every time a document is created or updated, its _version number increases by one. This might seem like a minor detail, but it becomes essential when two different processes try to update the same document around the same time — the version number lets Elasticsearch detect that a conflict may have occurred and respond accordingly, rather than silently letting one update overwrite the other without anyone noticing.
sequenceDiagram
participant AppA as Application A
participant ES as Elasticsearch
participant AppB as Application B
AppA->>ES: Read document (version 3)
AppB->>ES: Read document (version 3)
AppA->>ES: Update document (expects version 3)
ES-->>AppA: Success, now version 4
AppB->>ES: Update document (still expects version 3)
ES-->>AppB: Conflict detected — version is now 4
7Deep Dive: _score and Search Relevance
This one only shows up during searches, and it answers a very specific question: how good is this match?
Unlike _index or _id, which exist for every stored document, _score only appears in search results. It’s a number representing how relevant Elasticsearch judged that particular document to be for the specific search query you ran — higher generally means a stronger match. This is exactly what lets Elasticsearch sort results by relevance, showing the most useful matches first.
Why Relevance Isn’t Just “Exact Match or Not”
A search for “comfortable running shoes” might reasonably rank a listing titled “comfortable jogging sneakers” quite highly, even without an exact word match, because the underlying relevance calculation considers how closely related the words and their frequency are, not just literal equality.
8How Metadata Fields Travel Through a Document’s Life
Following one document from creation to search to see when each metadata field comes into play.
Document Submitted
An application sends a new JSON document to a specific index, with or without a chosen ID.
_index and _id Are Assigned
Elasticsearch records which index the document belongs to and finalizes its unique identifier.
_source Is Stored
A full copy of the original submitted JSON is retained for future retrieval.
_version Starts Tracking
The document’s version counter is initialized, ready to increment with any future update.
A Search Is Run
When someone later searches and this document matches, Elasticsearch calculates a _score just for that result.
9Advantages and Trade-offs of Built-in Metadata
Automatic metadata saves effort, but it’s worth understanding what you’re trading away.
Advantages
- No manual effort required to identify, version, or locate documents — Elasticsearch handles it automatically
- Enables safe concurrent updates through automatic version tracking
- Guarantees you can always retrieve the exact original document via
_source - Provides a consistent, ready-made way to rank and sort search results by relevance
Disadvantages / Trade-offs
- Storing full original documents in
_sourceuses extra disk space - Relying too heavily on auto-generated IDs can make it harder to prevent accidental duplicate documents
_scorevalues are only meaningful within a single query, not comparable across different searches
10Security Considerations Around Metadata Fields
A couple of these fields deserve careful thought when sensitive data is involved.
_source
Because _source returns the complete original document, an overly broad search response can accidentally expose sensitive fields to a client that only needed a small piece of information.
Limiting Returned Fields
Search requests can specify exactly which fields from _source should actually be returned, avoiding unnecessary exposure of sensitive data that happens to live in the same document.
Predictable, Non-Sensitive IDs
Choosing document IDs that don’t themselves leak sensitive information — avoiding something like using a person’s national ID number directly as the document’s _id — reduces exposure if IDs are logged or shared.
Access control rules can be applied at the field level, letting you restrict which users or applications are allowed to see certain sensitive fields within _source, even if they can search the index generally.
11Monitoring Metadata-Related Health Signals
A few things worth watching as a reflection of how metadata fields are behaving at scale.
| Signal | Why It Matters |
|---|---|
_source storage footprint | Large original documents stored repeatedly can meaningfully increase total disk usage |
| Version conflict errors | A rising rate may indicate multiple systems are racing to update the same documents |
| Duplicate-looking documents | Often traced back to relying on auto-generated IDs where a stable, chosen ID would have prevented duplication |
12Design Patterns and Anti-patterns for Metadata Fields
A few lessons worth learning from before they cause problems in production.
Problem
Always letting Elasticsearch auto-generate the _id for documents that clearly already have a natural, unique identifier from another system, like an order number.
Why It’s Harmful
Re-sending the same logical record — say, after retrying a failed request — creates a brand-new duplicate document instead of updating the existing one.
Correct Approach
Use the existing natural identifier as the document’s _id whenever one is available, so repeated submissions safely update rather than duplicate.
Problem
Returning the entire _source to end-user-facing applications by default, regardless of what the interface actually needs to display.
Why It’s Harmful
Sensitive or irrelevant fields can be exposed to the client unnecessarily, increasing both security risk and network payload size.
Correct Approach
Explicitly request only the specific fields a given feature actually needs from _source.
13Best Practices for Working With Metadata Fields
Simple habits that avoid the most common metadata-related mistakes.
Whenever a natural unique identifier already exists in your data, use it as the document’s _id instead of relying on auto-generation.
Request only the specific fields you need from _source in performance-sensitive or security-sensitive queries, rather than the entire document by default.
Comparing _score values across two completely different search queries, expecting them to mean the same thing — relevance scores are only meaningful within a single query’s context.
14Real-World Examples
Metadata fields quietly support everyday systems people interact with constantly.
Order Management Systems
Using an existing order number as the document’s _id ensures that reprocessing an order update always corrects the same record, instead of creating confusing duplicates.
Search Result Ranking
Product and content search features rely directly on _score to decide the order in which results are shown to users, surfacing the most relevant matches first.
Inventory Systems With Concurrent Updates
Warehouse systems processing simultaneous stock updates from multiple locations rely on _version to detect and resolve conflicting changes safely.
Multi-Tenant Applications
Applications serving many customers from a shared cluster use _index to separate and manage each customer’s data cleanly, even while running searches across the whole cluster.
15Frequently Asked Questions
The underscore-prefixed metadata fields are reserved and managed by Elasticsearch itself. If you want to store your own descriptive information about a document, it should be added as a regular field within the document body instead.
_score?No. A score only exists in the context of a search result — a document sitting in storage, not part of any active search, doesn’t carry a score.
_source?You save on storage space, but you lose the ability to retrieve the original document as it was submitted, and certain operations that rely on reading the original data, like reindexing, become unavailable.
_id required to be unique across the entire cluster?No — an _id only needs to be unique within its own index. The same ID value can exist in two different indices without conflict, since _index and _id together form the true unique address.
_version matter if I’m not worried about conflicts?Even in simple applications, version tracking is a useful safety net, and some features — like certain safe update operations — rely on it working correctly behind the scenes, whether or not you ever check it directly.
16Summary and Key Takeaways
Metadata fields are the quiet, automatically-managed labels Elasticsearch attaches to every document — describing where it lives, what identifies it, what its original content was, how many times it has changed, and, during a search, how relevant it is. You rarely need to set them yourself, but understanding what each one does — and the trade-offs tied to fields like _source and _id — helps you design applications that avoid duplicate data, protect sensitive information, and handle concurrent updates safely.
Key Takeaways
- Metadata fields describe the document itself, not the business data inside it, and always begin with an underscore.
_indexand_idtogether form a document’s unique address within a cluster._sourcepreserves the original submitted JSON, at the cost of some extra storage._versiontracks changes and helps safely handle concurrent updates._scoreonly exists during search and reflects relevance within that specific query alone.- Choosing a natural, meaningful
_idwhere possible prevents accidental duplicate documents. - Limiting what’s returned from
_sourceprotects both performance and sensitive data.