To understand analyzers, it helps to first understand the specific problem they exist to solve.

The problem: computers don’t understand text the way humans do

To a human, the sentence “The Quick Brown Foxes Are Jumping!” clearly contains the concept “fox,” even in a different capitalization, plural form, and punctuation context. To a computer performing a naive, literal comparison, “Foxes,” “fox,” and “FOX!” are simply three completely different strings of characters. An analyzer is the component that bridges this gap, transforming raw, human-written text into a normalized, standardized set of searchable terms that captures meaning rather than exact literal characters.

What an analyzer actually produces: tokens

The output of an analyzer is a stream of tokens — individual, cleaned-up units of text that Elasticsearch actually stores and searches against. The sentence “The Quick Brown Foxes” might be transformed into the tokens quick, brown, and fox (with “the” often removed and “foxes” reduced to its base form “fox”). These tokens, not the original raw sentence, are what a search query is ultimately compared against.

An analyzer isn’t a single indivisible unit — it’s built from three distinct, ordered building blocks working together in sequence.

1. Character filters — cleaning up before splitting even begins

The first stage, which is optional, operates on the raw text before it’s even broken into pieces. Character filters can strip out unwanted characters (like HTML tags accidentally included in text) or replace certain characters or patterns with others, cleaning up the text before the more meaningful splitting work begins.

2. Tokenizer — breaking text into pieces

Every analyzer has exactly one tokenizer, and this is the core step that actually splits the (now-cleaned) text into individual tokens — most commonly by breaking on whitespace and punctuation, so “quick brown fox” becomes three separate pieces rather than one long string.

3. Token filters — refining the individual pieces

After splitting, each resulting token can pass through one or more token filters, which perform further refinements: converting everything to lowercase, removing extremely common “stop words” like “the” or “is” that carry little search value, or reducing words to a common base form (so “running,” “runs,” and “ran” might all become “run”).

flowchart LR
    A["Raw Text:\n'The Quick Foxes!'"] --> B["Character Filters\n(optional cleanup)"]
    B --> C["Tokenizer\n(splits into pieces)"]
    C --> D["Token Filters\n(lowercase, remove stop words, etc.)"]
    D --> E["Final Tokens:\nquick, fox"]
        

Built-in analyzers versus custom analyzers

Elasticsearch ships with several ready-to-use, built-in analyzers designed for common needs — a general-purpose “standard” analyzer suitable for most everyday text, along with others tuned for specific behaviors like preserving exact whitespace-separated tokens without further processing. For more specialized needs, you can also build a fully custom analyzer by explicitly choosing your own combination of character filters, a tokenizer, and token filters.

Understanding exactly when analysis occurs clarifies a lot of otherwise-confusing search behavior.

Analysis at index time

When a document containing a text field is indexed, the value of that field is passed through the field’s configured analyzer, and the resulting tokens — not the original raw string — are what actually get stored in the underlying search structures. This is why searching typically feels forgiving about capitalization and minor word variations: the stored data has already been normalized.

Analysis at search time

When you submit a search query containing text, that query text is generally passed through an analyzer as well (often the same one used at index time, though this can be configured differently) before being compared against the already-stored tokens. This is what allows a search for “Running” to match a document that was indexed containing “runs” — both terms get normalized down to a shared, comparable form before any comparison happens.

sequenceDiagram
    participant Doc as Document Text
    participant IdxAnalyzer as Index-Time Analyzer
    participant Store as Stored Tokens
    participant Query as Search Query Text
    participant QAnalyzer as Search-Time Analyzer
    Doc->>IdxAnalyzer: "Foxes are running"
    IdxAnalyzer->>Store: fox, run
    Query->>QAnalyzer: "Running fox"
    QAnalyzer->>QAnalyzer: run, fox
    QAnalyzer->>Store: Compare tokens
    Store-->>Query: Match found
        

Analyzed fields versus non-analyzed fields

Not every field goes through this process. Fields intended for exact matching, sorting, or aggregations (often the “keyword” type) are deliberately left completely unanalyzed, storing the value exactly as given, with no splitting, lowercasing, or other transformation applied. This is why a text field commonly has both an analyzed version (for flexible full-text search) and a keyword version (for precise, exact-value operations) available side by side.

The choice of analyzer for a given field is largely a decision made once, early on, but its effects ripple through everything that happens to that field afterward.

Choosing an analyzer when a field is first defined

An analyzer is typically assigned to a text field as part of the field’s mapping, either explicitly by the person designing the index, or implicitly through the default analyzer Elasticsearch applies to text fields when dynamic mapping creates them automatically.

The analyzer stays fixed for the life of that field

Once documents have been indexed using a particular analyzer, changing that analyzer later doesn’t retroactively re-analyze data that’s already stored — the tokens already written into existing segments remain exactly as they were originally produced. To apply a new analyzer’s behavior to existing data, the standard approach is to create a new index with the updated analyzer configuration and reindex the existing documents into it.

Testing an analyzer before committing to it

Because an analyzer’s behavior is so consequential and difficult to change retroactively, Elasticsearch provides a way to test exactly how a given analyzer will process a sample piece of text before ever indexing real data with it — letting you preview the resulting tokens and catch surprising or undesired behavior early, while it’s still easy to adjust.

Analyzers make search remarkably flexible, but that flexibility involves real design decisions and trade-offs.

Analyzers process raw, often user-supplied text, which raises a few considerations worth being deliberate about.

Character filters and untrusted content

Text ingested from external or user-generated sources may contain unexpected content — such as embedded markup or formatting — that a character filter is often specifically used to strip out during analysis. Configuring appropriate character filters for fields receiving untrusted input helps ensure that unwanted content doesn’t end up preserved and searchable in ways that weren’t intended.

Analyzer configuration and information exposure

Custom analyzers and their exact configuration are generally considered part of an index’s design, not sensitive information on their own, but administrators should still apply the same general access controls to mapping and index-configuration details as they do to other cluster metadata, since a detailed understanding of exact analysis behavior can sometimes help someone craft especially effective (or evasive) search input.

Analyzer-related issues tend to surface as confusing search results rather than obvious errors, which makes deliberate diagnosis especially valuable.

Investigating unexpected search results

When a search unexpectedly returns too few, too many, or seemingly irrelevant results, checking how the relevant analyzer actually tokenized both the indexed text and the search query is often the fastest way to identify the root cause — frequently revealing that the two sides weren’t producing the comparable tokens one might assume.

Reviewing mapping and analyzer configuration over time

As an index’s mapping grows and evolves, periodically reviewing which analyzer is applied to which field helps catch situations where a field ended up with a default analyzer that doesn’t actually suit its content — such as a field holding product codes accidentally using a general-purpose text analyzer that splits codes in unhelpful ways.

A handful of well-established approaches to analyzer usage consistently lead to better, more predictable search behavior.

These habits consistently separate search experiences that feel intuitive from ones that quietly frustrate users.

Analyzers quietly shape the search experience behind many products people use every day.

No. Only text-type fields go through analysis by default. Fields intended for exact matching, such as keyword-type fields, are stored exactly as given, without any analysis applied.

Yes, Elasticsearch supports configuring separate index-time and search-time analyzers for a field, though this is typically done deliberately for specific, well-understood reasons rather than by default.

This often happens when the analyzer used for the search query produces different tokens than the analyzer used when the document was indexed — checking each side’s actual output tokens is the fastest way to find the mismatch.

A tokenizer is just one component within an analyzer — specifically, the part responsible for splitting text into pieces. An analyzer is the complete pipeline, which may also include character filters before the tokenizer and token filters afterward.

Yes. You can assemble a custom analyzer by explicitly choosing your own combination of character filters, a tokenizer, and token filters, tailored to your specific data and search requirements.

No, not automatically. Documents already indexed keep the tokens produced by whatever analyzer was in effect at the time. To apply a new analyzer to existing data, you need to reindex it into a new index configured with the updated analyzer.

An analyzer is the quiet, essential process that transforms raw, messy, human-written text into clean, consistent, comparable tokens — and it’s this transformation that makes Elasticsearch’s full-text search feel natural and forgiving rather than rigidly literal. By understanding its three building blocks (character filters, a tokenizer, and token filters), and by recognizing that the same analysis process happens both when data is indexed and when a search is submitted, you gain real insight into why searches sometimes behave unexpectedly — and how to fix that behavior deliberately, rather than through trial and error. Choosing the right analyzer for each field, and testing that choice before committing to it at scale, is one of the highest-leverage decisions in designing an effective Elasticsearch index.