Storing and searching data is only half the value. The other half is knowing, without asking, when the data says something important.

Data that changes all day and night

Modern systems produce a constant stream of facts: web requests, payments, login attempts, temperature readings, error messages, disk usage numbers. Elasticsearch is very good at keeping all of this and answering questions about it. But a question only gets answered when somebody asks it. If nobody is looking at the dashboard when the error rate triples at three in the morning, the problem grows quietly until customers notice.

People are not built to watch screens all day. They sleep, they take lunch, they attend meetings, and they get tired of looking at charts that are normal ninety-nine percent of the time. The sensible answer is to teach the computer to ask the question on a regular basis and to speak up only when the answer is worrying.

What good alerting gives you

The two ways alerting can fail

Alerting can fail by being too quiet or too loud. A too-quiet system misses a real fire. A too-loud system cries wolf so often that people mute it, and then it also misses the real fire, because nobody is listening anymore. A large part of this guide is about walking the line between those two failures.

Watcher borrows from several ideas you may already know. Getting comfortable with them makes everything else easier.

Searches and aggregations

Elasticsearch stores documents in indices. A search asks for documents that match some conditions, such as “all log lines from the checkout service in the last five minutes that contain the word error”. An aggregation summarizes matching documents, such as “how many per minute” or “the average response time per server”. Most alerts are built from exactly this kind of question, asked again and again.

Kinds of alert conditions

Time windows and schedules

Every alert is based on a window of time. A watch that asks “how many errors ever?” is useless, because the answer only grows. A watch that asks “how many errors in the last five minutes?” answers a fair, repeatable question. The schedule (how often you ask) and the window (how far back you look) are two separate choices, and they work together. If you ask every minute but look back five minutes, each error is counted several times. That is fine for a threshold alert, but it can be confusing if you want to count events exactly once.

Notification channels

An alert is only useful if it reaches a person or a system. Common destinations are email, chat tools such as Slack, paging services such as PagerDuty, ticketing systems such as Jira, and generic web endpoints that let you connect anything else. Watcher includes built-in ways to talk to these, and it can also write the alert itself back into an Elasticsearch index, so you have a record.

Templates

Messages are rarely fixed text. You want the message to say “The checkout service had 87 errors in the last five minutes”, with the real number filled in. Watcher uses a simple templating language, called Mustache, that lets a message pull values out of the search results and the watch itself. You will see later how the same watch produces different messages depending on the data it found.

flowchart LR
  A[Data arrives in Elasticsearch] --> B[Question asked on a schedule]
  B --> C{Is the answer worrying?}
  C -->|No| D[Stay quiet]
  C -->|Yes| E[Send message to a channel]
        

Watcher is the alerting and notification feature built into Elasticsearch itself.

The short definition

Watcher lets you create watches. A watch is a stored job that runs on a schedule inside the Elasticsearch cluster. Each time it runs, it collects some data (usually by running a search), checks a condition on that data, and, if the condition is true, carries out one or more actions such as sending an email or posting to a chat channel. Because it lives inside the cluster, it needs no extra server and it can ask questions of your data directly, with no export step.

Where it came from

Watcher began years ago as a separate plugin for Elasticsearch and later became part of the commercial feature set of the Elastic Stack, sitting alongside features like security and monitoring. Today its code ships inside Elasticsearch, and it is switched on or off by settings and by your license. Because the Elastic Stack has evolved, Kibana now also offers its own alerting framework with rules and connectors. Both exist, and this guide compares them later so you can choose between them with your eyes open.

What makes it different from a script and a timer

You could write your own script that runs every five minutes, queries Elasticsearch, and sends an email. Many people did before Watcher existed. Watcher gives you that same idea with several things already solved: it runs in the cluster and survives node failures, it stores its own history so you can see what happened, it can suppress repeats so people are not flooded, it has a testing tool, and it uses the same security system as the rest of Elasticsearch.

What Watcher is not

Watcher is not a full incident management system. It does not schedule on-call rotations, group related alerts into a single case, or track who fixed what. It sends messages to systems that do those jobs. Watcher is also not a stream processor: it looks at data on a schedule rather than reacting the instant a single document arrives, so it suits problems measured in minutes rather than milliseconds.

Every watch is built from the same five building blocks. Learn them once and you can read any watch.

flowchart LR
  T[Trigger fires] --> I[Input loads data]
  I --> C{Condition}
  C -->|False| X[Stop and record]
  C -->|True| TR[Transform optional]
  TR --> A[Actions run]
  A --> H[History recorded]
  X --> H
        

A watch is just data

A watch is a JSON document stored in a system index. That means you can create, read, update, and delete watches through the same kind of API calls you use for any other Elasticsearch data. It also means you can keep watches in version control, review them like any other configuration, and deploy them with automation. Kibana offers a user interface for creating simple watches and for editing the raw definition of advanced ones.

The watch record and the watch status

Alongside the definition, each watch has a status section that Watcher maintains itself. It records whether the watch is active, when it last ran, when its condition was last met, and the state of each action (for example, whether it has been acknowledged). You do not edit this by hand. It is how Watcher remembers what it has already done between runs, which is the key to avoiding duplicate alerts.

The trigger answers one question: how often should we look?

Schedule triggers

Watcher supports a schedule trigger. It comes in several flavors, from simple to precise.

Choosing an interval

The natural temptation is to make everything run every second, so nothing is missed. This is almost always a mistake. Each run costs a search, and thousands of searches per minute across many watches can load the cluster more than the problem they are watching. Choose an interval that matches how quickly someone must react. If a person will take at least ten minutes to respond, checking every minute is already generous. If the alert protects something that must be noticed within seconds, consider whether a different tool is the right choice.

Scheduled time versus triggered time

Every run carries two timestamps. The scheduled time is when the run was supposed to happen. The triggered time is when it actually started. They usually differ by only a moment, but if the cluster is busy, the difference can grow. Good watches base their search windows on the scheduled time, not on the current clock, so that a delayed run still looks at the correct slice of history and does not skip or double-count data.

Time zones and calendars

Calendar-style schedules such as daily at nine in the morning depend on a time zone. Confirm which time zone your version uses, and remember that the people receiving reports may live in another. Daylight saving changes can also shift which hour a job seems to run in. For anything where a wrong hour matters, test the schedule and check the recorded trigger times after it first runs.

Manual runs

Besides the schedule, you can trigger a watch manually with the execute tool. This is what you use while building and testing. It runs the watch right now, optionally ignoring the condition or simulating the actions, and it returns a full record of what happened without waiting for the clock.

The input gathers the facts the watch will judge. Four kinds exist, and they can be combined.

The search input in depth

The search input lets you name the indices (patterns are allowed), and provide a full search request body. You can filter by time, aggregate, sort, and pick which fields to return. The search body may include templated pieces, so it can refer to the scheduled time of the run. This is how a watch says “the five minutes before this run” without hard-coding any dates.

Because the whole search response becomes the input, it can be large. If your alert only cares about a count, ask for a count and request no hits at all. Returning thousands of documents that the condition never looks at wastes memory and time.

The HTTP input

The HTTP input reaches beyond Elasticsearch. For example, a watch can ask a status page for the health of a partner service, or ask another cluster for its statistics. Because it makes outbound calls, it deserves extra care: allow only the addresses you intend, keep credentials in the secure store rather than in the watch text, and set sensible time limits so a slow service cannot stall the watch.

Chaining inputs

A chain lets you build a small pipeline. The first step might find the busiest server in the last hour. The second step might then ask for the error counts of that particular server. The result of each step is available to later steps by name. This is powerful but easy to overuse. If a chain grows longer than two or three steps, consider whether the logic belongs in a dedicated application instead.

flowchart TD
  S[Scheduled run] --> IN{Input type}
  IN --> A[Simple: fixed data]
  IN --> B[Search: query an index]
  IN --> C[HTTP: call a web service]
  IN --> D[Chain: several in order]
  A --> P[Payload passed to condition]
  B --> P
  C --> P
  D --> P
        

The condition is the judge. The transform is the editor. Together they decide whether to speak and what to say.

Condition types

Thinking about thresholds

Choosing the line is harder than writing it. A threshold that is too low fires at every small bump. One that is too high fires only after the damage is done. Start by looking at a few weeks of historical data to learn what normal looks like, including daily and weekly rhythms. A shop has more traffic at lunchtime than at four in the morning, so a single fixed error count may be a good alert at night and a constant nuisance at noon.

Two common improvements help. One is to compare the current window to a matching window in the past, such as this hour compared with the same hour last week. The other is to require that the bad condition has lasted for several consecutive runs before alerting, which filters out one-off spikes.

Scripts as conditions

The script condition gives you full flexibility, for example checking that at least three of five services are unhealthy, or that a ratio between two aggregations passes a limit. The price is readability and safety. Scripts are harder to review, easier to get subtly wrong, and run inside the cluster, so keep them short and test them well. Prefer the simple condition types when they are enough.

Transforms

A transform sits between the condition and the actions. It changes the payload, the working data, before the actions see it. There are three forms: a search transform that replaces the payload with the result of a new search, a script transform that reshapes the data with code, and a chain that applies several transforms in order.

A very useful pattern is to keep the first search cheap (just a count), and use a search transform only when the condition is true to fetch a handful of example documents to show in the message. That way, the expensive detail is gathered only when it is actually needed. Transforms can also be defined separately for each action, so the email can carry a long explanation while the chat message carries one short line.

Actions are where alerting turns into something a human can read. A single watch can have several actions, each with its own settings.

Building good messages

Messages are templates. They can include the number that triggered the alert, the time window, the name of the watch, and values you stored as metadata. A good message answers four questions in the first lines: what is wrong, how bad is it, since when, and where should I look? A link to the relevant dashboard or the exact search is worth more than a paragraph of explanation.

Configuring the channels

Each channel needs to be set up on the Elasticsearch nodes before a watch can use it. Email needs mail server details. Slack and PagerDuty need account details, such as a webhook address or a service key. These secrets belong in the secure settings store of each node rather than in plain configuration files or in the watch text itself. Once an account has a name, watches refer to it by that name, so you can rotate a secret in one place.

Per-action control

Each action can have its own condition, transform, and throttle. This allows tiered alerting inside one watch: a warning to a chat channel when the error count passes fifty, and a page to the on-call engineer only when it passes two hundred. It is often cleaner than creating separate watches that repeat the same search.

Watcher is not a separate server. It is a set of components inside each Elasticsearch node. Knowing what they are explains most of its behavior.

flowchart TD
  U[Watch stored via API or Kibana] --> W[(Watches index)]
  W --> N1[Node 1: shard copy and scheduler]
  W --> N2[Node 2: shard copy and scheduler]
  N1 --> Q[Execution thread pool]
  N2 --> Q
  Q --> S[Search on data indices]
  Q --> ACT[Actions to email, chat, web]
  Q --> HI[(Watch history index)]
        

How work is spread across the cluster

Each watch document lives in a shard of the watches index. The node that holds a shard copy is responsible for scheduling the watches stored on that shard. This means the work of firing triggers is spread naturally across the nodes that hold the shards, and no single scheduler node is required. The watches index is configured so that copies are available on the nodes that need them, which lets a surviving node pick up the schedules of a failed one.

Which nodes should run Watcher?

Watches execute on data nodes that hold copies of the watches index. Because each run performs searches and calls other systems, heavy watcher workloads compete with the searches and indexing that those data nodes already do. In large clusters, plan capacity with this in mind, and avoid placing the watches index on the weakest nodes in the cluster.

The thread pool

Executions run in a dedicated thread pool with a bounded queue. When many watches fire at the same moment, the extra runs wait in the queue. If the queue fills up, further runs can be rejected and recorded as such. This is a safety valve. It stops a flood of alerts from overwhelming the node, but it also means that a badly overloaded cluster may skip or delay alerts precisely when you need them most.

Let us follow a single run from the tick of the clock to the record left behind in history.

sequenceDiagram
  participant Clock as Trigger engine
  participant Exec as Execution service
  participant Data as Data indices
  participant Chan as Notification channel
  participant Hist as History index
  Clock->>Exec: Trigger fired for watch
  Exec->>Data: Run input search
  Data-->>Exec: Search result
  Exec->>Exec: Evaluate condition
  Exec->>Chan: Send action if condition true
  Exec->>Hist: Write watch record
        

Recovery after a failure

The triggered watches index exists for a good reason. If a node stops in the middle of a run, the pending entry is still there. When Watcher starts again on that shard, it can find these orphaned entries and run them, so an alert that was interrupted by a restart is not simply forgotten. This recovery is best-effort: very old pending entries may be handled differently than fresh ones, so do not treat it as a promise of exactly-once delivery.

At-least-once thinking

Alerting systems are usually designed so that they would rather send an alert twice than not at all. In practice this means occasional duplicates after failures are possible, and receiving systems should tolerate them. Paging services help with this by merging repeated alerts that carry the same key into one incident.

If a problem lasts an hour and the watch runs every minute, it could send sixty identical messages. Two tools prevent that.

Throttling

Throttling sets a minimum gap between two executions of the same action. If an email action has a throttle period of thirty minutes, then after it sends one message, Watcher will skip it for the next thirty minutes even if the condition stays true. The watch keeps running, and the condition keeps being checked, but that action stays silent until the period ends. A throttle can be set on the whole watch or on each action separately, so a chat message can repeat every ten minutes while a page repeats only every hour.

Acknowledgement

Throttling works on time. Acknowledgement works on human attention. When a person knows about an alert and is dealing with it, they can acknowledge it. Watcher then stops sending that action while the condition stays true. Once the condition becomes false, the acknowledgement resets automatically, so the next real problem will notify again. This matches how people actually work: “I know, I am on it, be quiet until it is fixed, and then tell me if it comes back.”

Action states

stateDiagram-v2
  [*] --> AwaitsExecution
  AwaitsExecution --> Ackable: action runs successfully
  Ackable --> Acknowledged: person acknowledges
  Acknowledged --> AwaitsExecution: condition becomes false
  Ackable --> AwaitsExecution: condition becomes false
        

Choosing a strategy

An untested alert is a gamble. Watcher gives you several ways to try a watch before it ever wakes someone at night.

The execute tool

The execute watch API runs a watch on demand. You can run a stored watch by name, or supply a complete definition that is not stored at all, which is ideal for experiments. You can also override parts of the run. For example, you can ignore the condition to force the actions to run, or supply your own trigger time to see how the search window behaves for a moment in the past.

Action modes

A safe workflow

Recording test runs

By default, a manual execution is not saved into the history unless you ask for it. You can choose to record it. Recording is helpful when you are debugging and want to keep a trace, but it also pollutes the history with test entries, so use it deliberately.

Over the years, teams have converged on a handful of reliable alert shapes. Here are the most useful.

flowchart TD
  W[One watch runs each minute] --> C[Count errors and total requests]
  C --> D{Error rate high and enough traffic?}
  D -->|No| Q[Quiet]
  D -->|Yes| L1[Chat message to team]
  D -->|Very high| L2[Page the on-call engineer]
  D -->|Yes| L3[Write alert record to index]
        

Symptoms versus causes

A good rule is to page people on symptoms that affect users, such as failing checkouts or slow searches, and to leave causes, such as high processor use on a single server, for dashboards and low-priority notifications. A busy processor may be perfectly healthy. A failing checkout never is.

A watch can read data and call outside systems. That power needs guard rails.

Whose permissions does a watch use?

When a watch is saved through the API or Kibana, Elasticsearch records the security context of the user who saved it. When the watch later runs on its own, it uses those recorded permissions to run its searches. This means a watch can only read the indices the author was allowed to read at that time. It also means that if the author’s roles change afterward, the watch may keep the older permissions until it is saved again, and if the author’s account is removed, the watch may start failing. Treat the ownership of watches like the ownership of any automated job: give important watches to a dedicated service account, not to a person who may leave the company.

Roles for Watcher

Protecting secrets

Alert channels need credentials: mail server passwords, chat webhook addresses, paging keys, and passwords for web inputs. Keep these in the secure settings store on each node, and refer to accounts by name in watches. If a watch must include a secret inside its own definition, Watcher can encrypt sensitive values when a system key is configured, and it hides them when the watch is read back. Never put a secret in a message template, because messages are stored in history and delivered to third parties.

Outbound calls and network safety

Webhook actions and HTTP inputs make requests from inside your network. Restrict which addresses they may call using the allowed-hosts settings, so a mistake or a malicious edit cannot turn Watcher into a way to probe internal systems. Use encrypted connections for all outbound calls and verify certificates.

What alerts reveal

Alert messages travel through email servers, chat platforms, and paging systems that may be operated by other organizations. Avoid including personal data, secrets, or full log lines with sensitive content in the message. A short description with a link to a secured dashboard is usually safer than pasting raw data.

Audit and change control

Alert definitions are part of your operational safety, so changes to them deserve the same discipline as code. Keep watch definitions in a repository, review changes, and deploy through automation. Elasticsearch audit logging, where available under your subscription, can record who created, changed, or deleted a watch.

Watches are cheap one at a time and expensive in bulk. Planning keeps alerting from becoming a load problem.

What each watch costs

The cost of a watch is mostly the cost of its input search, multiplied by how often it runs. A simple count over a small window on a well-indexed field is very cheap. A wide aggregation across a month of data every minute is expensive. Add the cost of actions, which are usually small unless they call slow external services, and the cost of writing the history record.

Multiply it out: a hundred watches running every minute is a hundred and forty-four thousand searches per day. That is manageable when each is light, and a serious burden when each is heavy.

Ways to make watches lighter

Watch history growth

Every execution writes a record, and records for busy watches accumulate quickly. Watcher stores history in time-based indices, and recent versions manage their retention with an index lifecycle policy. Confirm that the policy exists and matches how long you actually want to keep history, and reduce the amount of data stored by keeping the payload small. A history full of huge search responses is expensive to store and to search.

Cluster health and Watcher

Watcher depends on a healthy cluster: the watches index must be available, and searches must complete. During a serious incident, such as a cluster running out of memory, Watcher can be one of the first casualties, at the very moment you need it. This is why many organizations run alerts about the health of their production cluster from a separate monitoring cluster.

flowchart LR
  P[Production cluster] -->|metrics and logs| M[Separate monitoring cluster]
  M --> W[Watcher on monitoring cluster]
  W --> N[Notifications to team]
        

An alerting system that fails silently is worse than none, because it gives false comfort. So you must watch the watcher.

The stats API

The Watcher stats API reports the state of the service on each node: whether it is started, how many watches are currently loaded, how many executions are running or queued, and details of the queue. A growing queue means watches are arriving faster than they can be executed. A service that has stopped means no alert will fire until it is started again.

The history

The watch history holds one record per execution. Searching it tells you which watches failed, which actions failed, how long each run took, and how often each condition was met. Useful questions to ask of the history are: which watches fail most often, which take the longest, which have never fired, and which fire constantly.

Alerts about alerts

A meta-watch is a watch that searches the watch history for failures and sends a message when many appear. Combine this with a dead-man’s switch that fires if no execution records appear at all for some time, since that would mean Watcher is stopped or the cluster is unhealthy.

Review as a habit

Schedule a regular review, perhaps monthly. Look at each alert that fired, decide whether it was useful, and improve or retire it. An alert that fired thirty times and led to action zero times is a candidate for removal or redesign. Good alerting is not built once. It is tended.

Watcher is not the only way to alert on Elasticsearch data. Choosing well starts with knowing the neighbors.

Watcher and Kibana alerting

Kibana has its own alerting framework built around rules and connectors. A rule checks a condition on a schedule and runs actions through connectors to email, chat, and other services. Rules are created through a friendly interface, come with ready-made rule types for common needs, integrate with Kibana features such as observability, security, and machine learning, and have their own management and history views. Elastic’s own applications have moved toward this framework for their built-in alerts, and it is often the recommended starting point for new alerting work.

Watcher remains supported and is still valuable. Its strengths are its low-level flexibility, its direct API, its ability to express custom multi-step logic with chained inputs and transforms, and the large number of existing setups that already rely on it. Because feature sets and recommendations change between versions, check the current documentation to see which is favored for your case.

Other approaches

An honest look at what you gain and what you accept.

The trade-off in one sentence

Watcher gives you power and proximity to your data in exchange for care: you must design carefully, test thoroughly, and watch its own health, because the flexibility that makes it useful also makes it possible to build noisy, heavy, or fragile alerts.

When Watcher is a strong fit

• You need custom, multi-step alert logic that guided rule types cannot express.
• You manage alerts as code through APIs and automation.
• You already have a working Watcher estate and no strong reason to migrate.
• Your alerts are based on windowed searches and aggregations of data you already keep in Elasticsearch.

When to consider something else

• You need sub-second reactions to single events.
• You want non-technical teams to build and manage their own alerts through a guided interface.
• You need full on-call scheduling, escalation policies, and incident tracking in one place.
• You want the data-source to be something other than Elasticsearch.

These mistakes appear again and again in real alerting systems. Each is easy to avoid once you can name it.

Smaller mistakes worth avoiding

• Alerting on every metric: each alert should map to an action a person can take.
• Never reviewing alerts: stale thresholds drift away from reality.
• Forgetting about time zones: report schedules that surprise the people who read them.
• Deleting a watch to silence it: deactivate it instead, so you can bring it back and keep its history.
• Skipping tests after edits: small changes to a condition can invert its meaning.

These are common, representative uses. They describe patterns seen across industries, not accounts of any named company.

flowchart TD
  A[Orders per minute drop against last week] --> B[Watch condition is true]
  B --> C[Chat message to shop team]
  B --> D[Page to on-call engineer]
  B --> E[Alert stored in alerts index]
  D --> F[Engineer opens runbook and dashboard]
        

Let us design alerting for a made-up online shop, step by step. All numbers are illustrative.

The situation

The shop logs every checkout attempt into Elasticsearch. Each document records the time, the result (success or failure), the payment provider, the response time, and the server. On a normal weekday, there are between two hundred and two thousand attempts per five minutes, depending on the hour. Failures normally stay below two percent.

Step 1: Decide what matters to customers

Customers care that checkout works and is fast. So we choose three symptoms: the failure rate, the slowness of the checkout, and the absence of any checkout activity. Everything else, such as server processor load, will live on dashboards and not wake anyone up.

Step 2: Design each watch

Step 3: Write good messages

The failure-rate message says, for example: “Checkout failure rate is 11 percent (limit 5 percent) over the last 5 minutes, mostly through payment provider B. Started around 14:05. Dashboard and runbook links below.” The reader sees impact, size, cause hint, timing, and next step in one glance.

Step 4: Control noise

The chat action repeats at most every fifteen minutes, and the page repeats at most every hour. Responders acknowledge the alert as soon as they start work. A separate small watch sends “checkout failures are back to normal” when the rate drops below the limit, so the team knows they can stand down.

Step 5: Test and roll out

Symptoms rarely announce their cause. These scenarios link common symptoms to likely explanations.

If you are introducing Watcher to a team, a staged approach reduces risk and builds trust in the alerts.

Templates are what turn a generic watch into a specific, helpful message. Understanding what data they can see is the key.

The execution context

During every run, Watcher builds a bundle of information called the context. Templates and scripts can read from it. The context carries the identity of the watch, the time the run was scheduled, the time it actually started, the metadata you attached to the watch, and the payload produced by the input and transforms. Nearly every useful message is just a few well-chosen values pulled from this bundle.

Why metadata is a quiet hero

Imagine that the threshold appears three times in a watch: once in the search, once in the condition, and once in the message. If you change one and forget the others, the alert says “limit is five percent” while the condition uses ten. Placing the threshold in the metadata, and reading it in all three places, means there is one number to change. The same goes for the dashboard link and the owner name.

Formatting values for humans

Raw values are often ugly: long decimals, timestamps in machine format, and huge counts without separators. Templates can loop over lists, show or hide sections depending on whether a value exists, and format numbers and dates through simple helper functions. A message that lists the top three affected servers, with a friendly time and a rounded percentage, is far easier to read than a dump of raw fields.

Escaping and safety

Values from your data may contain characters that mean something to the destination, such as angle brackets in email or quotation marks in a web request body. A log line that happens to include those characters can break formatting or, worse, be interpreted as markup. Keep message templates simple, and be careful with any field that contains text written by outsiders, such as user input or web addresses in log lines.

As more teams rely on alerting, the way you organize and scale Watcher matters as much as any single watch.

Scaling out

Watcher scales mostly by adding nodes and by keeping each watch light. Because the schedules and executions are spread across the nodes that hold shard copies of the watches index, adding capable data nodes helps to spread the work. But if every watch is heavy, adding nodes only moves the pain around. The first lever is always to make searches cheaper, and the second is to add capacity.

High availability

The watches index keeps copies on several nodes, so the loss of a single node does not lose the definitions of your alerts. Another node can take over the schedules. For the strongest guarantees, keep your cluster spread across zones, keep at least the recommended number of master-eligible nodes, and make sure notification accounts are configured on every node that could run a watch, not just some of them.

flowchart TD
  subgraph Z1[Zone A]
    A1[Data node with watches shard]
  end
  subgraph Z2[Zone B]
    B1[Data node with watches copy]
  end
  subgraph Z3[Zone C]
    C1[Data node with watches copy]
  end
  A1 -.->|fails| B1
  B1 --> N[Notifications continue]
        

Many teams, one cluster

When several teams create watches in the same cluster, problems of ownership and fairness appear. One team’s heavy watch can slow everyone else’s. One team’s mistake can flood a shared channel. Agree on conventions before the estate grows.

Multiple environments

Keep separate copies of alerts for development, testing, and production. Parameterize the destinations and thresholds through metadata or deployment settings, so the same watch definition can be promoted through the environments with only the values changed. This avoids the classic mistake of a test alert paging a real person at night.

You cannot improve what you do not measure. A few simple numbers reveal whether your alerts are helping.

Using the alerts index

If every important watch also writes a record into an alerts index, you can build a dashboard about your alerting. Chart alerts per day, per team, and per watch. Show which alerts were acknowledged, how long they took to clear, and which were never acted upon. This turns discussions about alert fatigue from opinion into evidence.

Many teams find themselves with older watches and newer rules at the same time. A calm plan avoids gaps.

Take an inventory first

List every existing watch: what it checks, how often, where it sends messages, who owns it, and how often it fires. You will usually discover that some watches are obsolete, some duplicate others, and a small number are truly critical. Only the critical and useful ones deserve migration effort.

Decide alert by alert

Run in parallel

When you migrate an alert, run the old and the new versions side by side for a while, with the new one sending to a test destination. Compare when each fires. When the new alert matches or improves on the old one, switch its destination to the real one and deactivate the old watch. Keep the old definition for a while so you can revert.

Avoid double notification

During the overlap, make sure the same problem does not page the same person twice. Only one version should use the real paging destination at a time.

Technology sends the message. People decide what happens next. Alerting works only when the human process around it works too.

Every alert deserves a runbook

A runbook is a short page that answers the question a tired person asks at three in the morning: “What do I do now?” It explains what the alert means, how to confirm it is real, the first three things to check, how to fix the common causes, and who to call if those fail. Link the runbook from the alert message itself, so there is no searching. If you cannot write a runbook for an alert, that is a sign the alert may not be actionable, and it may not deserve to interrupt anyone.

Route alerts to owners

Messages should go to the team that can fix the problem, not to a giant shared channel that everybody has learned to ignore. Use metadata to hold the owner of each alert and build routing from it. Shared channels are fine for awareness, but the thing that wakes a person should have a clear owner and a clear rotation.

Protect the people who respond

Building trust in alerts

Trust is the real currency of alerting. When every alert that arrives is real and useful, people respond quickly and calmly. When many alerts are false, people start to hesitate, delay, or ignore them. Each false alarm spends a little of that trust, and it takes many good alerts to earn it back. Guard it by testing carefully, tuning steadily, and retiring alerts that do not help.

A quick reference for the vocabulary used throughout this guide.

No. Watcher is built into Elasticsearch and defines alerts as stored watches. Kibana alerting is a separate framework in Kibana, built on rules and connectors. They solve the same broad problem in different ways, and many organizations use both.

Yes. Watcher is a subscription feature, not part of the free basic license, though a trial can be used to try it. Licensing details can change, so check the documentation for your version.

Intervals can be very short, but that does not mean they should be. Choose an interval that matches how fast someone can respond, and make sure the watch finishes faster than the interval.

Yes. Count documents in a recent window and fire when the count is zero. This dead-man’s switch pattern is one of the most valuable alerts you can build.

Throttling holds an action back for a set period of time, no matter what. Acknowledgement is a person’s request to be quiet until the condition clears. Both reduce repeated messages, in different ways.

Yes. The search and the condition still run every time. Only the actions are held back. Deactivate watches you no longer need.

It uses the permissions recorded from the user who saved it. If that user’s access changes or the account disappears, the watch may fail. Use dedicated service accounts for important watches.

Yes. The execute tool can run a watch on demand and simulate the actions, showing you the messages without sending them. You can also skip actions, ignore the condition, or supply a custom time.

Pending runs are recorded in a system index, and another node holding a copy of the watches shard can take over the schedule and resume interrupted runs. Recovery is best effort, so occasional duplicates or delays are possible.

In the watch history indices, which hold a record of each execution, including the input result, whether the condition was met, and the outcome of each action. Kibana also shows recent executions for each watch.

Not indefinitely. Failures are recorded in the history, but a message may be lost if the destination is down at that moment. For critical alerts, use a paging service and monitor for failed actions.

Yes. A watch can have several actions, each with its own condition, transform, throttle, and destination. This lets you send mild warnings to a chat channel and severe ones to a paging service.

No. Alert on symptoms that affect users and that a person can act on. Keep the rest on dashboards, where they help investigation without demanding attention.

Check the stats for the service state and queue, search the history for failures, and add a dead-man’s switch that fires if executions stop appearing. For production health, alert from a separate monitoring cluster.

Watcher is Elasticsearch’s built-in alerting engine. A watch is a stored job that wakes on a schedule, loads data with a search or a web call, judges it with a condition, optionally reshapes it, and then sends messages, creates tickets, or records the result. It runs inside the cluster, spreads across nodes, keeps a full history, and offers throttling, acknowledgement, and testing tools. Its power is flexibility and closeness to your data. Its risks are noise, load, and silent failure, which careful design, sensible schedules, secure ownership, and monitoring of Watcher itself can address. Kibana alerting is a friendly alternative for many everyday needs, and the two can live side by side. Above all, a good alert is a small, clear story that reaches the right person at the right time and asks them to do something useful.