What Are Metadata Fields in Elasticsearch?

What Are Metadata Fields in Elasticsearch?

A beginner-friendly, no-code walkthrough of the hidden, automatically-managed fields that travel with every document — what they are, what each one does, and why they matter even though you rarely set them yourself.

Imagine receiving a parcel in the mail. Beyond the actual gift inside, the parcel itself carries a shipping label, a tracking number, a barcode, and a postmark stamped by the courier. You didn’t write any of that information — the postal system generated it automatically to track, identify, and manage the parcel as it moved through the system. Every document you store in Elasticsearch works the same way. Alongside the actual content you send, Elasticsearch automatically attaches a small set of built-in labels called metadata fields, which describe the document itself rather than the information inside it. This post explains what these fields are, what each one is responsible for, and why they quietly matter.

1What Exactly Are Metadata Fields?

Let’s start with a clear, plain-language definition before looking at specific fields.

The simple definition

Metadata fields are special, built-in fields that Elasticsearch automatically attaches to every document, describing facts about the document itself — such as which index it lives in, its unique identifier, and how many times it has been updated — rather than describing the actual business data the document represents. They’re easy to spot because their names all begin with an underscore, like _index or _id.

Simple Analogy

Think of metadata fields like the stamps and labels on a passport — the passport number, the issuing country, the expiry date. None of that is “about you” as a person the way your name or photo is, but all of it is essential for identifying and managing that specific passport as an object.

i
Good to Know

You don’t need to manually create or manage most metadata fields — Elasticsearch generates and maintains them automatically as part of storing and retrieving your documents.

2Metadata Fields vs. Regular Fields

Understanding the distinction between “about the document” and “inside the document” clears up a lot of confusion.

AspectRegular FieldMetadata Field
DescribesThe actual business data, like a product name or priceThe document itself, like where it lives or its version
Who defines it?You, the application developerElasticsearch, automatically
Naming conventionAny name you chooseAlways begins with an underscore
Appears in your JSON body?Yes, as part of the document contentUsually shown alongside, not inside, the document body
!
Common Misconception

Metadata fields are not just “extra info you can ignore.” Several of them — especially the document’s identifier and version — are actively used by Elasticsearch to manage storage, updates, and consistency behind the scenes.

3The Most Important Metadata Fields

A quick tour of the handful of metadata fields you’ll encounter most often.

Metadata Field

_index

Records which index the document belongs to — useful when searching across multiple indices at once.

Metadata Field

_id

The document’s unique identifier within its index, used to retrieve, update, or delete that exact document.

Metadata Field

_source

Holds the original JSON document exactly as it was submitted, so it can be returned in full when retrieved.

Metadata Field

_version

Tracks how many times a document has been updated, helping manage concurrent changes safely.

Metadata Field

_score

Appears only in search results, representing how relevant a document is judged to be for a particular query.

Metadata Field

_routing

Controls which shard a document is stored on, allowing related documents to be grouped together for efficiency.

4Deep Dive: _index and _id

Together, these two fields form a document’s complete address inside a cluster.

Every document has a home — the index it was stored in — and a unique identifier within that index. Together, the _index and _id values act like a full mailing address: the index is the city, and the ID is the street address within it. This combination is exactly what Elasticsearch uses whenever you ask it to fetch, update, or delete one specific document directly, rather than searching for it.

i
Good to Know

If you don’t provide your own ID when creating a document, Elasticsearch will automatically generate a unique one for you.

When to choose your own ID

Providing your own meaningful ID — like an existing order number or user ID from another system — makes it simple to update or overwrite the exact same document later, rather than accidentally creating duplicates.

5Deep Dive: _source

The field that quietly makes it possible to get your original document back exactly as you sent it.

When Elasticsearch indexes a document, it doesn’t just break it apart into searchable pieces and throw away the original — it also keeps a complete, untouched copy of the original JSON you submitted, stored under _source. This is what gets returned whenever you retrieve or search for a document, letting you see the full, original data rather than just the internal search structures.

Simple Analogy

_source is like keeping the original signed copy of a contract in a filing cabinet, even after you’ve created a searchable summary card for it. If anyone ever needs the full, exact original, it’s right there.

!
Trade-off to Know

Storing the full original document does use additional disk space. In some very storage-sensitive setups, teams choose to disable _source for certain indices, though this means you lose the ability to retrieve the original document or reindex it later.

6Deep Dive: _version and Handling Concurrent Changes

A quiet counter that prevents two updates from silently overwriting each other.

Every time a document is created or updated, its _version number increases by one. This might seem like a minor detail, but it becomes essential when two different processes try to update the same document around the same time — the version number lets Elasticsearch detect that a conflict may have occurred and respond accordingly, rather than silently letting one update overwrite the other without anyone noticing.

sequenceDiagram
    participant AppA as Application A
    participant ES as Elasticsearch
    participant AppB as Application B
    AppA->>ES: Read document (version 3)
    AppB->>ES: Read document (version 3)
    AppA->>ES: Update document (expects version 3)
    ES-->>AppA: Success, now version 4
    AppB->>ES: Update document (still expects version 3)
    ES-->>AppB: Conflict detected — version is now 4
        
FIG 1 — Version numbers help detect when two updates collide.

7Deep Dive: _score and Search Relevance

This one only shows up during searches, and it answers a very specific question: how good is this match?

Unlike _index or _id, which exist for every stored document, _score only appears in search results. It’s a number representing how relevant Elasticsearch judged that particular document to be for the specific search query you ran — higher generally means a stronger match. This is exactly what lets Elasticsearch sort results by relevance, showing the most useful matches first.

Why Relevance Isn’t Just “Exact Match or Not”

A search for “comfortable running shoes” might reasonably rank a listing titled “comfortable jogging sneakers” quite highly, even without an exact word match, because the underlying relevance calculation considers how closely related the words and their frequency are, not just literal equality.

8How Metadata Fields Travel Through a Document’s Life

Following one document from creation to search to see when each metadata field comes into play.

1

Document Submitted

An application sends a new JSON document to a specific index, with or without a chosen ID.

2

_index and _id Are Assigned

Elasticsearch records which index the document belongs to and finalizes its unique identifier.

3

_source Is Stored

A full copy of the original submitted JSON is retained for future retrieval.

4

_version Starts Tracking

The document’s version counter is initialized, ready to increment with any future update.

5

A Search Is Run

When someone later searches and this document matches, Elasticsearch calculates a _score just for that result.

9Advantages and Trade-offs of Built-in Metadata

Automatic metadata saves effort, but it’s worth understanding what you’re trading away.

Advantages

  • No manual effort required to identify, version, or locate documents — Elasticsearch handles it automatically
  • Enables safe concurrent updates through automatic version tracking
  • Guarantees you can always retrieve the exact original document via _source
  • Provides a consistent, ready-made way to rank and sort search results by relevance

Disadvantages / Trade-offs

  • Storing full original documents in _source uses extra disk space
  • Relying too heavily on auto-generated IDs can make it harder to prevent accidental duplicate documents
  • _score values are only meaningful within a single query, not comparable across different searches
“Metadata fields are the paperwork Elasticsearch fills out for you, so you never have to.”

10Security Considerations Around Metadata Fields

A couple of these fields deserve careful thought when sensitive data is involved.

!
Risk: Full Document Exposure via _source

Because _source returns the complete original document, an overly broad search response can accidentally expose sensitive fields to a client that only needed a small piece of information.

Limiting Returned Fields

Search requests can specify exactly which fields from _source should actually be returned, avoiding unnecessary exposure of sensitive data that happens to live in the same document.

Predictable, Non-Sensitive IDs

Choosing document IDs that don’t themselves leak sensitive information — avoiding something like using a person’s national ID number directly as the document’s _id — reduces exposure if IDs are logged or shared.

i
Good to Know

Access control rules can be applied at the field level, letting you restrict which users or applications are allowed to see certain sensitive fields within _source, even if they can search the index generally.

11Monitoring Metadata-Related Health Signals

A few things worth watching as a reflection of how metadata fields are behaving at scale.

SignalWhy It Matters
_source storage footprintLarge original documents stored repeatedly can meaningfully increase total disk usage
Version conflict errorsA rising rate may indicate multiple systems are racing to update the same documents
Duplicate-looking documentsOften traced back to relying on auto-generated IDs where a stable, chosen ID would have prevented duplication

12Design Patterns and Anti-patterns for Metadata Fields

A few lessons worth learning from before they cause problems in production.

ANTI-PATTERN-01 Avoid
Problem

Always letting Elasticsearch auto-generate the _id for documents that clearly already have a natural, unique identifier from another system, like an order number.

Why It’s Harmful

Re-sending the same logical record — say, after retrying a failed request — creates a brand-new duplicate document instead of updating the existing one.

Correct Approach

Use the existing natural identifier as the document’s _id whenever one is available, so repeated submissions safely update rather than duplicate.

ANTI-PATTERN-02 Avoid
Problem

Returning the entire _source to end-user-facing applications by default, regardless of what the interface actually needs to display.

Why It’s Harmful

Sensitive or irrelevant fields can be exposed to the client unnecessarily, increasing both security risk and network payload size.

Correct Approach

Explicitly request only the specific fields a given feature actually needs from _source.

13Best Practices for Working With Metadata Fields

Simple habits that avoid the most common metadata-related mistakes.

i
Best Practice

Whenever a natural unique identifier already exists in your data, use it as the document’s _id instead of relying on auto-generation.

i
Best Practice

Request only the specific fields you need from _source in performance-sensitive or security-sensitive queries, rather than the entire document by default.

!
Common Mistake

Comparing _score values across two completely different search queries, expecting them to mean the same thing — relevance scores are only meaningful within a single query’s context.

14Real-World Examples

Metadata fields quietly support everyday systems people interact with constantly.

Order Management Systems

Using an existing order number as the document’s _id ensures that reprocessing an order update always corrects the same record, instead of creating confusing duplicates.

Search Result Ranking

Product and content search features rely directly on _score to decide the order in which results are shown to users, surfacing the most relevant matches first.

Inventory Systems With Concurrent Updates

Warehouse systems processing simultaneous stock updates from multiple locations rely on _version to detect and resolve conflicting changes safely.

Multi-Tenant Applications

Applications serving many customers from a shared cluster use _index to separate and manage each customer’s data cleanly, even while running searches across the whole cluster.

15Frequently Asked Questions

Q1Can I add my own custom metadata fields?

The underscore-prefixed metadata fields are reserved and managed by Elasticsearch itself. If you want to store your own descriptive information about a document, it should be added as a regular field within the document body instead.

Q2Does every document always have a _score?

No. A score only exists in the context of a search result — a document sitting in storage, not part of any active search, doesn’t carry a score.

Q3What happens if I disable _source?

You save on storage space, but you lose the ability to retrieve the original document as it was submitted, and certain operations that rely on reading the original data, like reindexing, become unavailable.

Q4Is _id required to be unique across the entire cluster?

No — an _id only needs to be unique within its own index. The same ID value can exist in two different indices without conflict, since _index and _id together form the true unique address.

Q5Why does _version matter if I’m not worried about conflicts?

Even in simple applications, version tracking is a useful safety net, and some features — like certain safe update operations — rely on it working correctly behind the scenes, whether or not you ever check it directly.

16Summary and Key Takeaways

Metadata fields are the quiet, automatically-managed labels Elasticsearch attaches to every document — describing where it lives, what identifies it, what its original content was, how many times it has changed, and, during a search, how relevant it is. You rarely need to set them yourself, but understanding what each one does — and the trade-offs tied to fields like _source and _id — helps you design applications that avoid duplicate data, protect sensitive information, and handle concurrent updates safely.

Key Takeaways

  • Metadata fields describe the document itself, not the business data inside it, and always begin with an underscore.
  • _index and _id together form a document’s unique address within a cluster.
  • _source preserves the original submitted JSON, at the cost of some extra storage.
  • _version tracks changes and helps safely handle concurrent updates.
  • _score only exists during search and reflects relevance within that specific query alone.
  • Choosing a natural, meaningful _id where possible prevents accidental duplicate documents.
  • Limiting what’s returned from _source protects both performance and sensitive data.