We start with the simple idea behind the name, and then look at how Elasticsearch makes that idea real.

Search Normally Works One Way

Elasticsearch is a search engine. It stores items called documents, such as products, articles or listings, inside containers called indices. When someone runs a query, which is a description of what they are looking for, Elasticsearch finds the documents that match it. In this normal direction, the documents are stored and the queries are temporary. A query arrives, gets answered and is forgotten.

The Percolator Flips It

The percolator turns this around. It lets you store queries as if they were documents. Later, when a new document arrives, you hand it to Elasticsearch and ask, “Which of my stored queries match this?” Elasticsearch replies with the list of matching queries. The queries are permanent, and the incoming document is the temporary visitor.

The name comes from the coffee percolator, where hot water passes through a basket of coffee grounds. Here, a document passes through a large collection of stored queries, and the ones that “catch” it are the matches.

What the Percolator Is Not

The percolator does not watch your data by itself. It does not send emails or push alerts. It only answers one question when you ask it: “Which stored queries match this document?” Your own application must ask that question for each new document and then act on the answer. The percolator is also not a machine-learning classifier that guesses. It matches exactly according to the rules in the stored queries. And it is not meant for everyday searching, where ordinary queries against stored documents are the right tool.

To understand why the percolator exists, look at what goes wrong when people try to solve matching problems with normal search.

The Two Naive Solutions

The first naive solution is to run every saved search for every new document. If you have two million saved searches and five thousand new documents an hour, that is ten billion query runs an hour. Even if each run is tiny, the total is enormous, and almost all of the runs will find nothing.

The second naive solution is polling, which means running each saved search on a schedule, say every ten minutes, and checking for anything new. Polling wastes effort, because most runs find no change. It also adds delay, because a person may wait up to ten minutes for an alert. Shorten the schedule to reduce delay and the cost climbs even higher.

Why Not Build It Yourself?

A team could write its own matcher, but real queries are complicated. They involve words that must be broken into pieces and reduced to their roots, numbers within ranges, locations within distances, combinations with “and” and “or”, and exclusions. Elasticsearch already implements all this for ordinary search. The percolator reuses the same query language and the same text analysis, so a saved query means exactly the same thing whether it is used in search or in reverse search.

A handful of words appear in every percolator discussion. Learn them here and the later chapters will feel easy.

The Chain of Ideas

Here is how these words fit in one sentence. You store queries in a percolator field, in an index with a proper mapping. At storage time, Elasticsearch performs query extraction. Later, your application sends a candidate document inside a percolate query. Elasticsearch does candidate selection to narrow the field, then verification to confirm matches, and returns the matching stored queries. Read the chain twice, and the rest of the guide will feel familiar.

Putting the two directions side by side makes the percolator’s role very clear.

flowchart LR
    subgraph FWD["Forward search: ordinary Elasticsearch"]
        Q1["One query arrives"] --> IDX1["Many stored documents"]
        IDX1 --> R1["Matching documents returned"]
    end
    subgraph REV["Reverse search: the percolator"]
        D2["One document arrives"] --> IDX2["Many stored queries"]
        IDX2 --> R2["Matching queries returned"]
    end

Pull and Push

Another way to see the difference is through the words pull and push. Ordinary search is pull: a person goes to the search box and pulls the results they want. The percolator supports push: when something new arrives, it tells you who should be pushed a message about it. Most alerting systems are push systems, which is why the percolator fits them so well.

Let us look at the parts that work together in a percolator setup, from the people who save searches to the people who receive alerts.

The Parts of the System

flowchart TD
    U["People on web and mobile apps"] --> G["API Gateway"]
    G --> LB["Load Balancer"]
    LB --> SUB["Subscription Service: save, edit, delete searches"]
    SUB --> PIDX["Elasticsearch percolator index: stored queries"]
    SRC["New documents from publishers or data pipeline"] --> ING["Ingestion Service"]
    ING --> DIDX["Elasticsearch document index: real documents"]
    ING --> MATCH["Matching Service"]
    MATCH --> PIDX
    PIDX --> MATCH
    MATCH --> Q["Message queue"]
    Q --> NOTIF["Notification Service"]
    NOTIF --> U
    MON["Monitoring: latency, match counts, errors"] --> MATCH
    MON --> PIDX

Reading the Diagram

Look at the top-left first. People use web and mobile apps, and their requests pass through the gateway and load balancer to the subscription service. This service writes each saved search into the percolator index as a stored query. That is the “left side” of the system, which is quiet and changes only when people edit their searches.

Now look at the right side. New documents come from publishers or a data pipeline. The ingestion service stores each one in the ordinary document index, and also passes it to the matching service. The matching service asks the percolator index which stored queries match, receives a list of matches, and places them on a message queue. The notification service picks them up and sends the alerts to the right people. Monitoring watches the matching service and the percolator index, since they carry the real-time part of the system.

Follow one new document from the moment it arrives to the moment the list of matching queries comes back.

sequenceDiagram
    participant PUB as Publisher
    participant MS as Matching Service
    participant CN as Coordinating node
    participant SH as Percolator index shards
    participant ML as Notification queue
    PUB->>MS: New document arrives
    MS->>CN: Percolate query with the document
    CN->>SH: Send the request to every shard
    SH->>SH: Build a tiny in-memory index of the document
    SH->>SH: Candidate selection using extracted terms
    SH->>SH: Verify each candidate query against the document
    SH-->>CN: Matching stored queries
    CN-->>MS: Merged list of matches
    MS->>ML: Queue one alert per match

Why the Tiny Index Is So Clever

The trick of building a small in-memory index for the document lets Elasticsearch reuse its entire ordinary search machinery. A stored query does not know or care that the document is temporary. It runs against the tiny index exactly as it would against a large one. This is why any query that works in normal search can, with a few exceptions, work as a stored query too.

Before any document can be percolated, the queries must be stored in an index that is prepared for them. This chapter explains the preparation.

The Percolator Field Type

Every field in an Elasticsearch index has a type, such as text, number, date or location. The percolator adds one more type, also called percolator. A field of this type holds a whole query. You choose its name, for example “query”, and then each document in the index carries one stored query in that field. In effect, the percolator index is a table of saved searches, where each row is one saved query with some extra columns.

The Index Needs Two Kinds of Fields

The percolator index must contain the percolator field, and it must also describe every field that the stored queries and the candidate documents will use. Suppose stored queries talk about “title”, “price”, “city” and “bedrooms”. Then the index mapping must define those four fields with the correct types, even though the index does not store real listings. The mapping is the shared vocabulary that lets stored queries and incoming documents understand each other.

Extra Fields Beside the Query

You can add ordinary fields next to the percolator field. They describe the stored query and are very useful. Typical examples are the identifier of the person who owns the saved search, the region it applies to, whether it is active, when it was created and the language of the alerts. These extra fields are normal, searchable fields, so you can filter and sort by them.

Fields Must Exist First

The fields used inside a stored query must already be defined in the mapping when the query is stored. Elasticsearch checks the query at that moment, and it will reject a query that refers to unknown fields. This is helpful, because it catches typing mistakes early, but it also means that adding a new searchable field requires updating the mapping first, before anyone can save a query that uses it.

The secret to handling millions of stored queries is that the percolator almost never runs most of them. This chapter explains how.

The Problem It Solves

Imagine one million stored queries and one incoming document. If Elasticsearch had to run every query against the document, it would perform one million query runs for a single document. Even at a tiny cost each, this adds up to a large delay, and it would be hopeless at thousands of documents per minute. Yet in almost every real system, only a handful of the stored queries could possibly match any given document.

Step One: Extract Terms When the Query Is Stored

When a query is saved, Elasticsearch studies it. It looks for the terms that must be present in a document for the query to have any chance of matching. For a simple query that requires the word “garden”, the key term is “garden”. For a query that requires both “leeds” and “balcony”, Elasticsearch picks terms that a matching document is certain to contain. It stores these extracted terms in hidden fields next to the query.

Step Two: Look Up Candidates Fast

When a document arrives, Elasticsearch collects the terms that appear in it and asks one fast question of the hidden fields: “Which stored queries have key terms that this document actually contains?” This uses the same fast lookup structure that ordinary search uses, so it is quick even with millions of entries. Only the queries that pass this test become candidates.

Step Three: Verify the Candidates

The candidate list is a shortlist, not the final answer. A query might have the right key term but fail on another condition, such as a price range. So each candidate is run for real against the tiny in-memory index of the document. Only those that truly match are returned.

flowchart TD
    S["Stored queries: extracted key terms saved at storage time"] --> C1
    D["Incoming document"] --> T["Collect the document's terms"]
    T --> C1["Fast lookup: which stored queries share terms with the document"]
    C1 --> CAND["Shortlist of candidate queries"]
    CAND --> V["Run each candidate query against the in-memory copy of the document"]
    V --> M["Confirmed matches"]
    V --> X["Rejected candidates"]

When Selection Cannot Help

Some queries cannot be narrowed by key terms. A query that matches everything has no term to look up, so it is always a candidate and must always be verified. Certain other shapes, such as some negations or unusual combinations, also give little to extract. A percolator index full of such queries loses the benefit of candidate selection, and performance falls toward the naive “run everything” case.

There are several ways to hand documents to the percolator. Each fits a different situation.

One Document Inline

The simplest way is to include the document inside the percolate query. You name the percolator field and give the document’s contents. Elasticsearch builds the tiny in-memory index and returns matching stored queries. This is what a real-time matching service does for a single new document, for example when a new listing is posted.

Many Documents Together

You can also give a list of documents in the same request. Elasticsearch then builds a small in-memory index holding all of them and matches each stored query against the group. In the response, each hit reports which of your documents matched, using a position number called a slot. Percolating documents in batches is usually much more efficient than one at a time, because the fixed costs of the request are shared. It is a natural fit for systems that receive documents in bursts.

An Already Stored Document

If the document already lives in an ordinary index, you do not need to send its contents again. You can point the percolate query at the index and the document’s identifier, and Elasticsearch fetches the document and runs it against the stored queries. This helps when you index first and percolate afterward, or when you want to re-check old documents against newly created queries.

What Comes Back

The response looks like an ordinary search response. Each hit is a stored query, complete with the extra fields stored beside it, such as the owner identifier and region. When you percolated several documents, each hit also says which documents matched. The score on each hit is not a measure of how good the document is. It reflects how the stored query was matched, and most systems ignore it and simply treat the list of hits as a list of matches.

The percolate query is a normal query, which means it can be combined with other queries. This makes real systems much faster and much more useful.

Filter Before You Percolate

Stored queries come with extra fields such as region, category and status. You can combine the percolate query with ordinary filters on those fields. For example, “only stored queries that are active and belong to the Leeds region.” Elasticsearch applies these cheap filters first, so the expensive matching runs on far fewer stored queries. In systems with regions, languages or customers, this single habit can reduce the work by an order of magnitude.

Combining Several Percolate Queries

A single request can hold more than one percolate query, joined with “and”, “or” or “not” logic. A common use is to percolate several different kinds of documents at once, or to run two different percolator fields against the same document. Each part can be given a name, and the response reports which named part matched, which helps you tell the reasons for a match apart.

Highlighting the Reason

Sometimes it is useful to know not just that a stored query matched, but why. The percolator supports highlighting, which marks the exact words in the incoming document that caused the match. For an alert email, this lets you show “Your search for ‘garden’ matched this listing, which says ‘a private garden and terrace’.” It is a small feature that makes alerts feel far more intelligent and trustworthy.

Almost every query type works, but a few do not. Knowing the limits early avoids surprises.

The Broad Rule

A stored query can use the normal query language: words and phrases, exact values, numeric and date ranges, geographic distances, combinations with “and”, “or” and “not”, and many more. The core rule is that the query must be answerable using only the information inside the incoming document and the mapping. A query that needs to look at some other document, or at the current clock, cannot be prepared ahead of time.

The Trouble With “Now”

People often want alerts such as “listings posted within the last day.” A stored query cannot say “within the last day” using the current time, because the stored query is prepared once, when it is saved, and would keep using that moment forever. The usual solution is to move the time logic into your application. Since the percolator sees each document as it arrives, “just posted” is already true for every document you percolate. If you need age rules, add a field to the document such as “hours since posting” at percolate time, or apply the rule in your own code afterward.

Fields Must Match Between Queries and Documents

A stored query and an incoming document must use the same field names with the same meanings. If a stored query talks about “price” as a number, but the incoming document sends “price” as text like “nine hundred pounds”, the match will fail or be rejected. Agree on one clear document shape, and validate both queries and documents against it.

The most common use of the percolator is the alert: “tell me when something new matches what I am looking for.”

The Alert Pipeline

flowchart LR
    NEW["New listing posted"] --> ING["Ingestion Service"]
    ING --> IDX["Store in the document index"]
    ING --> PERC["Percolate against stored searches"]
    PERC --> HITS["List of matching saved searches"]
    HITS --> DEDUP["Remove duplicates and respect quiet hours"]
    DEDUP --> QUEUE["Alert queue"]
    QUEUE --> EMAIL["Email"]
    QUEUE --> PUSH["Mobile push"]
    QUEUE --> HOOK["Webhook to partner system"]

The flow is simple. A new listing arrives and is stored in the ordinary index so that it appears in normal search. At the same moment, it is percolated against all the saved searches. The list of matches is cleaned up, for example by removing duplicates and respecting quiet hours, and placed on a queue. Separate workers turn queue items into emails, phone notifications or messages to partner systems.

Designing Good Alerts

• Be timely. Percolate as soon as a document is indexed. The value of an alert falls quickly with time, especially for scarce items.
• Be specific. Ask users for enough detail that their queries are precise. Vague queries create noisy alerts and slow matching.
• Avoid duplicates. If a document is edited several times, decide whether each edit should alert again. Keep a record of which alerts were sent.
• Respect limits. Give users a way to control the number and frequency of alerts, and offer a daily digest as an alternative to instant messages.
• Explain the match. Use highlighting to show why the alert was sent, which makes alerts feel useful and not random.

The percolator can also sort and label documents using rules that people write, without any machine learning.

Rules as Stored Queries

Imagine a news site that wants to label each incoming article with topics such as “football”, “elections” or “technology”. Editors write rules for each topic: the words and phrases that signal it. Each rule is stored as a query. When an article arrives, the percolator returns which topic rules matched, and the system attaches those topics as tags. Adding a new topic requires only saving a new query. No new code is needed.

Why Use Rules Instead of Machine Learning?

Machine learning classifiers can be powerful, but they have costs: they need training data, they can be hard to explain, and they can change behavior when retrained. Rules are transparent, quick to change and easy to audit. For classification tasks where the categories are clear and the signals are keywords, or combinations of fields, rules stored in the percolator are often simpler and more trustworthy. Many teams also combine both: rules for the clear cases and a model for the fuzzy ones.

Typical Tagging Tasks

The percolator is also a general “rules on a stream” engine. Any time you have many rules and a flow of incoming events, it can help.

Watching Streams of Events

Operations teams handle streams of logs, messages and events. Each team may care about different patterns: a specific error code, a mention of a certain customer, a payment above a limit in a certain country. Instead of building one big program with all rules hard-wired, the team stores each rule as a query. As events flow past, the percolator reports which rules fired, and the system triggers the right action for each.

Compliance and Safety Checks

Regulated industries must check communications and content for banned or sensitive language, or for patterns of concern. Compliance staff maintain a list of rules. Each new message is percolated, and any that match are flagged for review. Because the rules are ordinary stored queries, compliance staff can add or change them quickly when regulations change, without waiting for a software release.

Routing Work to the Right Place

The same idea helps route work. An insurance company might route each incoming claim to a specialist team, based on rules about the type of loss, the region and the amount. A logistics company might route shipment events to the right customer dashboards. Each routing rule is a stored query, and the percolator says which routes apply to each event.

Several other tools can seem to solve the same problem. Comparing them shows where the percolator fits best.

Alerting Features Versus the Percolator

Elasticsearch and Kibana also have alerting features. They usually run a search on a schedule and fire an action if the results meet a condition, such as “more than a hundred errors in the last five minutes.” That is perfect for operational monitoring. The percolator is different: it reacts to each new document, and it can hold millions of separate rules belonging to millions of users. Think of alerting rules as a smoke detector that watches a room over time, and the percolator as a doorman who checks each guest against a guest list as they enter.

Enrichment Versus Percolation

An enrich feature at ingest time adds information to a document by looking up matching reference data, for example adding a region name based on a postcode. It answers “what extra data belongs to this document?” The percolator answers “which rules does this document satisfy?” They can work together: enrich the document first so that it has the fields that rules need, then percolate it.

Text search depends on how words are broken up and simplified. For the percolator, this process must be identical on both sides.

What Analysis Does

Before text can be searched, Elasticsearch runs it through an analyzer. An analyzer splits text into words, converts them to lower case, and may reduce them to a root form, so that “running”, “runs” and “ran” can be treated as related. It may also remove tiny words such as “the”. This is what lets a search for “garden flat” find a listing that says “Flats with Gardens”.

Both Sides Use the Same Recipe

In the percolator, the stored query and the incoming document are both analyzed using the mapping of the percolator index. Because they share the same recipe, their terms line up. If the recipe changed in the middle, the stored queries would hold terms produced by the old recipe, while new documents would be broken up by the new one. Matches could silently disappear.

Changing Analyzers Later

Extracted terms and prepared queries are created when a query is stored. If you change the analyzer for a field, you must store the queries again so they are prepared with the new recipe. In practice, this means creating a new percolator index with the new mapping, copying the queries into it, and switching your application to the new index. Plan this as a normal migration, with testing and a rollback plan.

The percolator can handle very large collections of stored queries, but only if you understand where the time goes.

Where the Time Goes

Each percolate request has a few parts. First, the request travels to every shard of the percolator index. Second, each shard builds a tiny in-memory index of the incoming document. Third, each shard performs candidate selection over its stored queries. Fourth, each shard verifies the candidates by running them. Finally, the results travel back and are merged. The biggest variable is usually the number of candidates that need verification, and the cost of each one.

Parallel Work Across Shards

Stored queries are spread across shards, and each shard works on its own share at the same time. If one shard holds a quarter of the stored queries, it does about a quarter of the matching. This is why percolator indices scale well as they grow: adding shards and nodes spreads the stored queries thinner. However, the setup cost per shard, such as building the tiny index, is repeated on every shard. Choose a shard count that keeps each shard busy with a meaningful number of queries and no more.

Throughput Versus Latency

Latency is how long one request takes. Throughput is how many requests or documents can be handled per second. A system that percolates each document individually may have low latency for each but limited throughput. A system that batches documents raises throughput but adds a little waiting time for each document while the batch fills. Choose based on your needs: instant alerts need small batches, and overnight backfills can use large ones.

A stored query passes through a series of stages, from the moment a person saves a search to the day it is removed.

flowchart LR
    C["Created by a person"] --> V["Validated by your service"]
    V --> S["Stored in the percolator index"]
    S --> X["Terms extracted and indexed"]
    X --> A["Active: matched against new documents"]
    A --> E["Edited: replaced by a new version"]
    E --> V
    A --> P["Paused: status set to inactive"]
    P --> A
    A --> D["Deleted or expired"]
    P --> D

Making Changes Safely

Stored queries are ordinary documents, so editing one means replacing it with a new version. There is a brief moment during which the old version is still visible, until the change is refreshed and searchable. Elasticsearch makes new documents visible on a short schedule, usually about a second. For alerts, this delay is fine. But be aware that a query saved just now may not match a document percolated in the same instant.

Expiry and Cleanup

Saved searches that nobody uses waste space and processing time. Give each stored query a creation date and a last-used date, and remove those that have been quiet for a long time, after warning the owner. Some systems also let users set an end date, such as “alert me until I move house.” A good cleanup habit keeps the percolator index small and matching fast.

A short story with numbers makes the ideas concrete. Here is an invented example of a job board and its job alerts.

The Setting

A job board has two million saved searches from candidates. About six thousand new jobs are posted each hour, which is under two per second. The team wants alerts sent within a few seconds of a job being posted. All figures in this example are invented to illustrate the reasoning and are not from a real company.

The Design

What Happens to One Job

A company posts a job called “Senior Data Engineer, Remote, Python and Spark” in the United Kingdom. The ingestion service stores it in the job index and sends it to the matching service. The percolate request filters to active saved searches for the United Kingdom, about 240,000 of them. Candidate selection looks at the words in the job and shortlists about 300 stored queries that share key terms. Verification confirms that 42 of them truly match, including candidates who asked for “data engineer” and “python”, and one who asked for “spark remote”. The response carries the 42 owner identifiers and highlights of the matching words.

The matching service removes six candidates who already received an alert for a near-identical job an hour earlier, then queues 36 alerts. Workers send the emails, each one showing the highlighted words that matched. Total time from posting to queue is under a hundred milliseconds, and emails leave within a few seconds.

What Went Well, and What Was Tuned

In the first test, without the country filter, the same request took several times longer, because more than a million stored queries were checked in candidate selection. Adding the filter fixed most of the delay. A second problem was a small number of very vague saved searches such as “any job”, which were candidates for every posting and produced huge, unwanted alert volumes. The team added a rule requiring at least two conditions in every saved search, and the noise stopped.

Good index design makes everything else easier. This chapter covers the main decisions.

One Index or Many

You can keep all stored queries in one index or split them into several. A single index is simplest, and works well when all queries use the same mapping and similar traffic. Separate indices help when different kinds of queries have different mappings, when tenants need strong isolation, or when different parts of the business have different performance needs. Adding an alias that points to several indices lets one percolate request cover them all when needed.

Choosing the Number of Shards

Shards spread stored queries over the cluster. Too few shards means each does too much work and only a few nodes are used. Too many shards means each does little and the fixed setup cost of every request grows. A useful method is to estimate the number of stored queries and the time to match a typical document, then test with two or three shard counts, and pick the smallest one that meets your latency target with room to spare.

Routing by Tenant or Region

Routing lets you decide which shard stores each query, using a value such as a customer identifier or region. When you percolate with the same routing value, Elasticsearch asks only the shards that hold that value, not all of them. For systems where every request concerns one tenant or one region, routing cuts the work sharply, and it also improves isolation between tenants.

The percolator often stores queries written by outsiders, such as users of your website. Treat those queries with care.

User-Written Queries Are Untrusted Input

If people can save their own searches, they can accidentally, or deliberately, write queries that are very expensive. A query with a wildcard on a very common prefix, a huge list of terms, or a complicated regular expression may take a long time to verify, and it would run for every document in which it is a candidate. One bad query can slow down everyone’s alerts.

The defense is to never store raw queries from users. Instead, let users fill in a structured form, such as keywords, price range and location. Your service builds the actual query from those fields, using only safe query types and limits you have chosen. This is the same rule that protects databases from injection: build queries from validated parts, not from raw user text.

Limits Per User

• Number of saved searches. Cap how many each user can have, so one account cannot flood the index.
• Size and shape. Limit the number of conditions and the number of terms in each query.
• Allowed query types. Permit only the types you have tested and know to be cheap.
• Creation rate. Limit how often a user or an automated script can create or change saved searches.

Keep Tenants Apart

In a system with many customers, one customer’s stored queries and alerts must never leak to another. Store a tenant identifier in every stored query, and always apply a filter on it in every percolate request. Consider using Elasticsearch’s security features, such as roles that limit which documents a user or service can see, so the rule is enforced by the cluster and not only by your code.

Protect the Content

Incoming documents may contain private data, such as messages or personal details. Percolate requests carry the whole document, so encrypt connections with TLS, restrict who can call the matching service, and avoid logging full documents in application logs. Alerts sent to users should show only the parts of a document that they are allowed to see.

A missed alert is a silent failure, which makes reliability especially important for percolator systems.

Reliability Basics

The percolator index is an ordinary Elasticsearch index, so it gets the usual protections: replica shards on other nodes, spread across zones, so a node failure does not lose stored queries or stop matching. Keep at least one replica of the percolator index. Since your application also holds the original queries in its own database, the percolator index can always be rebuilt from that source.

Never Lose a Document

The percolator only matches the documents you give it. If your matching service is down, documents that arrive in that time are not matched, unless you have a way to catch up. The safe design puts new documents onto a durable queue first, and lets the matching service consume from it. If the service stops, documents wait in the queue and are matched when it returns. Alerts are late, but none is lost.

Because failures are quiet, monitoring must actively look for the absence of alerts as well as errors.

The Numbers That Matter

Watching for Silence

A percolator system can fail by simply matching nothing. Set an alert if the number of matches per hour drops far below the normal level for that time of day. Also keep a small “canary” query and a canary document that must always match. Percolate the canary every few minutes. If it stops matching, you know that something in the chain has broken, even if there are no error messages.

Finding Expensive Queries

Elasticsearch offers slow logs, which record requests that take longer than a limit you set. Use them on the percolator index to find slow percolate requests. If a few stored queries are the cause, you can identify them by looking at which queries are candidates most often and are slow to verify, then work with their owners to make them more specific.

The percolator is a strong tool for a specific job. A fair look at its strengths and limits helps you decide when to use it.

Key Trade-offs to Decide

Freedom versus safety. Letting users write any query gives flexibility but exposes you to costly queries. A structured form limits freedom and protects performance. Most public systems choose the form.

Speed of alerts versus efficiency. Percolating each document immediately gives the fastest alerts, but small requests carry more overhead. Batching for a second or two improves efficiency with almost no visible delay.

One big index versus several. One index is simple to run. Several indices give isolation and independent tuning at the cost of more to manage.

Rules versus learning. Stored rules are clear and controllable but need human upkeep. Machine learning handles fuzzy cases but is harder to explain. Many systems use both.

Teams that have run percolators in production repeat the same good habits, and the same avoidable mistakes.

Patterns That Work

Anti-Patterns to Avoid

A short list of habits that separate smooth percolator projects from painful ones.

Common Mistakes

• Assuming automatic notifications. Forgetting that your own code must call the percolator and deliver the messages.
• Using time-relative ranges. Trying to store queries with “now” in them and finding that they are refused or behave wrongly.
• Forgetting to map fields. Storing queries that refer to fields the mapping does not define.
• Changing analyzers casually. Altering text analysis without rebuilding the stored queries, causing silent missed matches.
• Ignoring the vague queries. Leaving very broad saved searches in place, which slow every request and flood users with alerts.
• Too many shards. Splitting a modest percolator index into many shards and paying the setup cost of each request many times over.
• No source of truth elsewhere. Having no copy of the saved searches outside the percolator index, so a rebuild is impossible.

Setting up a percolator is a few days of work. Running it for years is the rest. These are the routine tasks that keep it healthy.

Keeping the Index Tidy

Saved searches accumulate. People change their minds, leave the service or forget they ever created an alert. Regularly remove stored queries for closed accounts, expired alerts and searches that have not produced a match in a very long time. Tell owners before removing anything they might miss. A tidy index is faster and cheaper.

Updating the Mapping

When you want to support a new searchable field, add it to the mapping first. New fields can usually be added without a rebuild. But changing the type or analyzer of an existing field cannot be done in place. For those changes, create a new percolator index with the new mapping, copy the stored queries into it from your own source of truth, test it side by side and then switch the alias. Keep the old index for a short while as a rollback.

Upgrading Elasticsearch

Read the release notes for the versions you are upgrading through. The percolator has changed across versions, and older systems used a different, separate percolate interface, which has long since been replaced by the current query. Test your stored queries and your known-match checklist on the new version before upgrading production, and watch for any query types that are no longer accepted.

Reviewing the Heaviest Queries

Every month or so, look at which stored queries are slow or are candidates for a very large share of documents. Talk to their owners, or improve the form that created them, so that they are more specific. A short review often shows one or two patterns that account for most of the load, and fixing them can free up a great deal of capacity.

When the percolator seems to misbehave, an ordered set of checks finds the cause faster than guessing.

Common Symptoms and First Guesses

The stories below are illustrative composites showing how different kinds of organizations use the percolator. They are not accounts of specific companies.

Half-true ideas about the percolator cause wasted effort and bad surprises. Here are the ones that appear most often.

A quick reference for the important words in this guide, in plain language.

Quick answers to the questions people ask most often when they first meet the percolator.