Start here. By the end of this chapter you will know what an anomaly is, what a “machine learning job” is, and where this feature came from.

Anomaly detection in Elasticsearch is a built-in machine learning feature that automatically learns what “normal” looks like in your time-based data and then flags anything that does not fit. An anomaly is simply a moment or a pattern that is unusual compared with what came before. A sudden jump in errors is an anomaly. So is a sudden drop in sales, a user who logs in from a country they have never used, or a server that suddenly sends ten times more data than its neighbors.

The feature works on time series data. That is a fancy name for records that each carry a timestamp: web server logs, CPU readings, payment transactions, temperature sensors, security events. If your data has a “when” attached to every row, anomaly detection can watch it.

What does “machine learning job” mean?

Elasticsearch organizes this work into jobs. A job is a saved set of instructions that says, in effect, “watch this data, ask this question, and remember what you learn.” You create one job for each kind of question you want answered. For example, one job might watch how long web pages take to load, and another might watch how many failed logins each user has.

In the Kibana screens you will find them under Machine Learning, then Anomaly Detection, then Jobs. You may also hear the words “ML jobs” used loosely. Elasticsearch has other machine learning tools too, such as data frame analytics jobs that can find odd rows in a static table, but this article is about anomaly detection jobs, the ones that watch data flowing through time.

Why is it called unsupervised?

Many machine learning systems need a teacher. You show them thousands of examples labeled “good” and “bad,” and they learn to copy your labels. Anomaly detection in Elasticsearch needs no teacher at all. Nobody has to tag a single record. The job studies the data on its own, decides what is typical, and measures how far each new moment strays from typical. This is called unsupervised learning, and it is what makes the feature practical: nobody has the time to label a year of server history.

A short history

Elastic, the company behind Elasticsearch, bought a specialist anomaly detection company called Prelert in 2016. Prelert had spent years perfecting statistical techniques for spotting strange behavior in IT data. Elastic folded that engine into its stack, and it first shipped to customers in 2017 as part of a commercial add-on called X-Pack. In the following years the add-on’s code was bundled into the standard download, and the machine learning family grew: new job types were added, data frame analytics arrived for finding outliers in static data sets, and the feature became woven into Elastic’s security and observability products.

Today anomaly detection is a first-class citizen of the platform. It has its own screens in Kibana, its own programming interfaces, its own dedicated processes, and ready-made jobs for common needs such as unusual logins or slow application services. Like most advanced features in the stack, it requires a paid subscription tier (or a trial) rather than the free basic license, so check your license before you plan a project around it.

To understand why anomaly detection exists, first see why the older approach, fixed alert rules, keeps letting people down.

The classic way to watch a system is to pick a number and draw a line. “Alert me if response time goes above two seconds.” “Alert me if errors go above fifty per minute.” These static thresholds are easy to understand, and they are still useful for hard limits such as a disk that is completely full. But as a general watching method they fail in four predictable ways.

Failure one: the world has a rhythm

Real traffic is not flat. A shop that serves 2,000 requests per minute at lunchtime may serve only 40 at three in the morning. If you set the threshold at 1,000 to catch daytime trouble, then a night-time explosion from 40 to 400 requests, ten times more than usual and very possibly an attack, slips right under the line. If you lower the threshold to 100 to catch the night trouble, it fires all day long for perfectly healthy traffic. No single number can be right at every hour.

Failure two: there are too many things to watch

A company with 10,000 servers, each reporting 20 different measurements, has 200,000 separate streams of numbers. Nobody can hand-write and hand-tune 200,000 rules, and every new server or new service adds more. Anomaly detection lets one job watch every server, learning a separate normal for each one.

Failure three: normal keeps changing

Businesses grow. A threshold that was perfect in January is too low by June because customers doubled. Somebody has to remember to update it, and somebody always forgets. Learned models adapt as the data changes, so the definition of normal moves along with the business.

Failure four: you cannot alert on what you never imagined

Every threshold is a guess about what might go wrong. The most expensive incidents are often the ones nobody imagined: a cache that quietly stops working, a partner whose requests suddenly triple, a login pattern that looks like a slow, careful thief. Because anomaly detection compares the data with its own history rather than with your guesses, it can notice trouble you never thought to ask about.

None of this means thresholds are useless. A disk that is 99 percent full deserves an alert no matter what history says. The best monitoring setups use both: thresholds for hard limits, and learned models for everything that has a rhythm.

Eight words unlock the whole feature. Each one is explained with what it is, why it exists, where you meet it, and a picture from everyday life.

The job

A job is the saved question. It records which field holds the timestamp, how often to summarize the data, which measurements to study, and how much memory the learning may use. You meet it in the Jobs list in Kibana, where each job shows its state, its data, and its latest results. Think of a job as a night-shift worker with a written assignment card: “Watch the checkout page and tell me if it acts strangely.”

The datafeed

The datafeed is the delivery route that brings data from your Elasticsearch indices to the job. It runs a search on a schedule, collects the matching documents, and hands them to the learning engine. A datafeed can also apply a filter so that only the data you care about, say, requests to one website, is delivered. If the job is the worker, the datafeed is the conveyor belt carrying items to the worker’s desk.

The bucket span

The job does not judge every single record one by one. It sorts time into equal slices called buckets and judges each slice as a whole. The length of a slice is the bucket span, for example fifteen minutes. In each bucket the job might count events or average a measurement, and then ask “is this bucket normal?” A bucket is like an hourly report card for a school: not every heartbeat of the day is graded, only the summary of each period.

The detector

A detector is one precise question inside a job. It pairs a function (what to calculate, such as count, average, or maximum) with optional fields (what to calculate it over and how to split it). “Count the events” is a detector. “Average the response time, separately for each server” is another. One job can hold several detectors, like a worker who has several checklist items.

The model

The model is everything the job has learned so far: typical levels, daily and weekly cycles, long-term trends, and how much natural wobble to expect. It lives inside a dedicated native process and is saved periodically so it survives restarts. The model is the guard’s memory of how the building behaves.

The anomaly score

For every bucket and record, the job produces an anomaly score from 0 to 100. It tells you how surprising the moment was, on a scale that is adjusted to your own data. A score near 0 means “completely ordinary.” A score near 100 means “among the most unusual things this job has ever seen.” Kibana turns scores into four colored bands: warning, minor, major and critical.

The influencer

Knowing that something is odd is half the story. The other half is who or what was involved. Influencers are fields, such as host name, user name or country, that you list so the job can point at the entities that seem to have contributed most to an anomaly. It is like a detective asking not only “was the milk spilled?” but “which of these children was standing next to it?” Influencers are clues, not proof of guilt.

The results

Results are what the job writes down: a score for every bucket, a detailed record for every anomaly, and influencer rankings. They are stored in special system indices inside Elasticsearch, so you can search them, chart them, and alert on them just like any other data. Results are the guard’s logbook.

Now let us zoom out and see the machinery: which parts exist, where each one runs, and how they talk to each other.

Anomaly detection is not a plugin bolted on the side of Elasticsearch. It is woven into the cluster. Your data stays in your own indices, the learning happens on your own machines, and the answers come back into your own indices. Nothing leaves the cluster. The diagram below shows the full path from raw data to a notification in somebody’s chat window.

flowchart LR
    SRC["Source indices - logs, metrics, events"] --> DF["Datafeed"]
    CFG["Job configuration - system index"] --> AP
    DF --> AP["Native analysis process - one per open job"]
    AP --> RES["Results index - scores, records, influencers"]
    AP --> SNAP["Model snapshots - saved memory of the model"]
    RES --> KB["Kibana - Anomaly Explorer and Single Metric Viewer"]
    RES --> RULE["Alerting rules"]
    RULE --> OUT["Email, chat, tickets, webhooks"]
    subgraph MLNODE["Machine learning node"]
        DF
        AP
    end

Walking through the diagram

On the far left are your source indices, the ordinary Elasticsearch indices that already hold your logs, metrics or business events. The datafeed reads from them. The job configuration, kept safely in an internal system index, tells the analysis process what to compute. The native analysis process is the brain: a separate program written in C++ that runs beside Elasticsearch, one copy for each open job. It receives the summarized data, updates the model, and decides how surprising each bucket was.

It writes two kinds of output. The first is results, which land in a results index. Kibana reads these to draw its charts, and alerting rules read them to decide when to notify a human. The second is model snapshots, periodic saves of everything the model has learned, so that a restart never means starting the education from zero.

The components in detail

Who decides where a job runs?

When you open a job, the cluster’s master node picks a machine learning node with enough spare memory and assigns the job to it. This uses the same “persistent task” idea that Elasticsearch uses for other long-running background work. If that node later disappears, the master notices and reassigns the job elsewhere. Think of a control tower at an airport assigning planes to gates: it always knows which gates are free and moves planes when a gate closes.

This is the engine room. You do not need a mathematics degree to follow it, because every idea comes with a picture.

Under the hood, the analysis process repeats a simple loop for every bucket of time. It receives the summary of the bucket, compares it with what the model expected, works out how unlikely the difference is, turns that unlikeliness into a score, and then learns from the new data so that tomorrow’s expectations are a little smarter.

flowchart TD
    A["New bucket of data arrives"] --> B["Summarize it - for example count or average"]
    B --> C["Model predicts what was expected"]
    C --> D["Compare actual with expected"]
    D --> E["Work out the probability of seeing this"]
    E --> F["Convert probability to a score from 0 to 100"]
    F --> G["Write results for Kibana and alerts"]
    D --> H["Update the model with the new value"]
    H --> C

Step one: summarize

Raw records are far too numerous and noisy to judge one at a time, so the job first squeezes each bucket into a small number that matches the detector’s function. If the detector is “count,” the number is how many events happened in the bucket. If it is “mean of response time,” the number is the average. This is why picking the right bucket span matters: it sets how much detail survives the squeeze.

Step two: predict

The model holds a picture of how this number usually behaves. Elastic describes the machinery as a blend of statistical techniques. One breaks the history into layers: a long-term trend (are we growing?), repeating cycles such as day and week, and leftover random noise. Another describes how likely each possible value is, using Bayesian probability. Others group similar entities together and notice when measurements move together. Combined, they let the model say, “at 10 a.m. on a Tuesday for this server, I expect around 1,900 requests, give or take 150.”

Step three: measure surprise as a probability

Now the job looks at the real number, say 5,400 requests, and asks: “If my model is right, how likely is a value this far away?” The answer is a probability, often a tiny one such as one in a million. The smaller the probability, the stranger the moment. The probability of each bucket is stored in the results as well, so you can inspect it later if you are curious.

Step four: normalize into a score

Tiny probabilities are hard for humans to read, so the job translates them to the friendly 0 to 100 scale. It does this by comparing each surprise with the most surprising moments seen so far in that job. This is called normalization. Because it is relative to your own history, “90” means the same kind of rare event in a quiet system as in a wild one. A side effect is that scores can be revised: if something even stranger happens later, earlier scores may be nudged down, because the yardstick has stretched. The results keep both the first-draft score and the current one.

Step five: learn

Finally the new value is folded into the model. This is online learning: the model updates continuously as data arrives, with no separate training phase and no nightly retraining schedule. It also means the model keeps adapting to change. If your traffic permanently doubles after a marketing campaign, the model gradually accepts the new level as normal.

Anomalies that build slowly

Not every problem is a spike. A slow memory leak might push a measurement up by only a small step every bucket. Each single step is unremarkable, but the sum is alarming. The job therefore also looks at groups of neighboring buckets, roughly a dozen at a time, and can report a multi-bucket anomaly when the combined deviation is unlikely even though no single bucket was. Results carry a field that shows how much the multi-bucket view contributed. A sudden blowout of a tire is easy to spot; a slow leak needs you to compare the pressure now with the pressure an hour ago.

The early learning period

A new job knows nothing, so it starts humble. In its first buckets it has little evidence and its predictions are rough, so early results can be noisy. As a rule of thumb, a few hundred buckets give a decent picture of ordinary behavior, and if your data has weekly rhythms, about three weeks of history helps the model learn them properly. This is why it is common to start a job by feeding it historical data first, a step described in the next chapter.

A worked example with real numbers

Suppose a job counts requests to an online store in five-minute buckets. From weeks of history, the model has learned that on Tuesday mornings around ten o’clock the count is usually near 1,900, and that the natural wobble is about 150 either way. Now consider three possible buckets. In the first, 1,950 requests arrive. That is well inside the usual wobble, its probability is high, and the score stays near zero. In the second, 2,300 arrive. That is a little unusual, perhaps a small campaign, and the score is low but visible. In the third, 5,400 arrive. That is far outside anything the model has seen at this hour, its probability is minuscule, and the score climbs toward the top of the scale. The exact mathematics is richer than a simple bell curve, but this picture gets the intuition right: the further outside your usual range, and the rarer that distance, the higher the score.

Now imagine the same 5,400 arriving on a Saturday night during a big televised event when the model has seen similar surges many times before. The score would be much lower, because the model has learned that such nights are busy. The same number can be alarming on one occasion and ordinary on another. That is precisely the power of learning context rather than comparing against a fixed line.

Memory management inside the process

Every entity the job models, each server, each user, each country, costs a little memory. The job has a configured limit, and it watches its own usage. When usage approaches the limit it first tries to forget the least useful models, such as those for entities that have not been seen for a while. If it truly reaches the limit, it stops creating models for new entities, and some data is dropped rather than analyzed. The job reports which of these states it is in, and you will meet that report again in the monitoring chapter. Consider it a backpack: you can carry many things, but eventually you must decide what stays.

A job is born, opened, fed, watched, stopped and eventually retired. Knowing each stage prevents most beginner surprises.

Two switches, not one

Beginners often assume a job is a single on-off switch. It is actually two things. The job can be open or closed, which decides whether its analysis process is running and holding a model in memory. The datafeed can be started or stopped, which decides whether data is flowing in. A job must be open before its datafeed can deliver anything. Picture a factory: opening the job turns on the machine, starting the datafeed turns on the conveyor belt.

stateDiagram-v2
    [*] --> closed
    closed --> opening : open job
    opening --> opened : process ready
    opened --> closing : close job
    closing --> closed : model saved
    opened --> failed : unrecoverable error
    failed --> closed : force close
    closed --> [*] : delete job

Historical lookback, then real time

You can start a datafeed at a chosen time in the past. The datafeed then races through old data as fast as the machine allows. This is the lookback phase, and it teaches the model about your normal patterns in minutes instead of weeks. Thirty days of history at fifteen-minute buckets is 2,880 buckets, plenty for a solid first impression. When the datafeed catches up with the present and you did not give it an end time, it flips into real-time mode and keeps polling for fresh data forever, or until you stop it.

Frequency and query delay

In real-time mode two timing settings matter. The frequency is how often the datafeed knocks on the door of your indices to ask for fresh data. The query delay is how far behind “now” it looks, and it is deliberately not zero. Real systems are messy: an event that happened at 10:00:02 may only arrive in Elasticsearch at 10:00:40 because of network hops and ingest pipelines. If the datafeed asked for data up to exactly 10:00:00 at exactly 10:00:00, it would miss such late arrivals forever. The default delay is around a minute or two, and you should raise it if your data reaches Elasticsearch slowly.

Summarized datafeeds for huge data

If your source generates millions of documents per minute, dragging every raw document across to the job wastes effort. A datafeed can instead ask Elasticsearch to do the counting and averaging first using aggregations, so only a tiny summary travels. This is called an aggregated datafeed. The trade-off is that you must design the aggregation carefully, because the job now trusts whatever summary it is given.

Where the results go

Every result is written to the results indices as it is produced. Notifications about the job itself, such as “job opened” or “memory limit reached,” are written to a separate notifications index. Both can be searched with the tools you already know.

Before we go deeper, here is what actually happens on the screen when you create a job. Knowing the route makes the later chapters easier to follow.

Creating a job in Kibana feels like filling in a well-designed form with a live preview. You do not write instructions in a special language. You choose, look at a chart, adjust, and confirm. Here is the journey, step by step, for a beginner who wants to watch the response time of a web application.

Before you begin

Two things must be in place. First, your license or trial must include the feature, and your cluster must have at least one machine learning node. Second, your data must already be in Elasticsearch with a timestamp field, and Kibana must have a data view for it. A data view is simply Kibana’s saved description of which indices to read and which field holds the time. If you can already see your data in Discover, you are ready.

Explore the data first

Kibana includes a Data Visualizer, a screen that profiles a data view. It shows every field, how many different values each one has, and what the typical values look like. This is the perfect moment to check the cardinality of a field you plan to split by. Discovering that a field has ten values is good news. Discovering it has ten million saves you from a costly mistake. Think of it as walking around a house before you start decorating.

What you will see in the first minutes

As the historical data flows through, Kibana shows a progress bar and starts drawing the model’s expected band around your line. Anomalies appear as colored dots. In the early stretch the band may look wide and jumpy, because the model is still forming an opinion. As time passes the band tightens and follows your daily rhythm closely. This is the model growing confident. When the run finishes, you can open the Single Metric Viewer or the Anomaly Explorer and start reading results.

Kibana offers several job wizards, each shaped for a different kind of question. Picking the right one saves hours of tuning.

All job types share the same engine. What differs is how much the wizard configures for you and what kind of question it is designed to answer. The decision chart below is a quick way to choose. Start at the top and follow the question that sounds most like yours.

flowchart TD
    Q["What do you want to watch?"] --> A{"One number over time?"}
    A -->|"Yes"| SM["Single metric job"]
    Q --> M{"Several numbers, or one number split by a field?"}
    M -->|"Yes"| MM["Multi-metric job"]
    Q --> B{"Compare many entities with each other?"}
    B -->|"Yes"| POP["Population job"]
    Q --> C{"Free-text log messages?"}
    C -->|"Yes"| CAT["Categorization job"]
    Q --> D{"Unusual values or places?"}
    D -->|"Values seen rarely"| RARE["Rare job"]
    D -->|"Locations on a map"| GEO["Geographic job"]
    Q --> E{"Need every setting under your control?"}
    E -->|"Yes"| ADV["Advanced job"]

Single metric job

The simplest of all. It watches exactly one measurement, for example “the number of orders per five minutes” or “the average CPU use across the fleet.” It is the friendliest way to learn the feature because you can see one line, one shaded band of expected values, and the anomalies sitting on it. Think of it as watching one pulse on one heart monitor.

Multi-metric job

Here you watch several measurements at once, and optionally split them by one field, such as host name. If you ask for “average CPU and maximum memory, split by server,” the job builds a separate model for every server and every measurement. It is a wall of heart monitors, one per patient, all managed by a single nurse. This is the workhorse for infrastructure monitoring.

Population job

A population job compares members of a group with each other instead of with their own past. Imagine a company where 5,000 employees download files. Most download a few megabytes a day, but a few download gigabytes. A population job builds a picture of “what typical employees do” and flags those who stray far from the crowd. It is powerful for security, because a compromised account often behaves differently from its peers even if it has no long history of its own. The comparison is a teacher noticing that one child is doing something none of the classmates are doing.

Advanced job

The advanced wizard exposes every setting: several detectors, all field roles, custom rules, memory limits and more. It is what you reach for when the friendlier wizards cannot express your question, or when you want to save your configuration as a repeatable recipe.

Categorization job

Logs are often free text, and free text is hard to count. A categorization job first sorts messages into groups by their structure, so that “Connection to 10.0.0.5 timed out” and “Connection to 10.0.0.9 timed out” land in the same category because only the address differs. Then it watches how often each category appears. A brand-new category, or a category that suddenly explodes, is a strong signal that something new is going wrong. Picture a librarian who sorts thousands of letters by their type without reading names, and then notices that a new type of letter has started arriving.

Rare job

A rare job looks for values that almost never occur. “This program has never run on this server before.” “This user has never signed in from this country.” Either it compares against the entity’s own history, or against the whole population to say “almost no machine ever does this.” It is the party guest who says, “Wait, who is that person nobody has ever seen?”

Geographic job

Geographic jobs work on latitude and longitude pairs and flag places that are unusual. A user who always logs in from one city and suddenly appears on another continent is the classic example.

Ready-made jobs

You do not always have to design jobs yourself. Elastic ships groups of preconfigured jobs for common data sources, often called modules. Security products include jobs for unusual processes, odd network traffic and suspicious logins. Observability products include jobs for slow services, log message rates and host metrics. They are a superb starting point and a good way to see what well-designed jobs look like.

A detector is where you spell out the exact question. This chapter explains the vocabulary of functions and the three ways to split your data.

Functions: what to calculate

The function is the calculation performed on each bucket. Functions come in families, and most of them have “high” and “low” variants that tell the job which direction of surprise you care about. A plain count flags unusual numbers in either direction. A high count flags only surprisingly large counts, which is handy when a drop is harmless but a surge is not. A low count does the opposite, which suits watching for silence, such as a sensor that suddenly stops reporting.

The “non-zero count” family is worth a special mention. Some data is sparse: a bucket with no events is simply missing rather than being a real zero. Non-zero functions ignore empty buckets, which prevents the job from screaming about “no data” for something that only happens now and then.

Fields: how to split the data

Analyzing “all requests” as one lump hides problems. If one server out of a hundred is sick, the fleet average hides it. Splitting lets the job model each entity properly. There are three ways to split, and they behave differently.

By field versus partition field

These two look alike but differ in an important way. With a partition field, every value is scored on its own scale. A very noisy server cannot use up the “surprise budget” and hide an odd moment on a calm server, because each server’s scores are normalized separately. With a by field, values share a scale, so the most unusual entity in the job earns the highest scores. Use partition when entities are truly independent, such as separate customers; use by when you want a global ranking of oddness.

Over field and population analysis

The over field switches on population analysis. Instead of asking “is this user different from their own history?” the job asks “is this user different from other users right now?” It is the right tool when individual histories are short, such as newly created accounts, or when you specifically care about outliers within a group. The models are also memory-friendlier than giving every entity a full history.

Influencers

After splitting, you list influencers, the fields you would want to see named when an anomaly appears. Usually you include the fields you split by and any others that could explain the story, such as data center, application version or customer. The job ranks the values of these fields by how much they contributed to each anomaly, and you see them highlighted in Kibana. Without influencers, you know that something is strange but not where to look first.

How to choose a function: three questions

Choosing a function feels hard at first, but three questions settle most cases. First, are you interested in how many things happened, or in the value of something that was measured? If you care about volume, such as requests or errors, use a count-style function. If you care about a measured value, such as response time or temperature, use a metric-style function. Second, do you care about both directions of change, or only one? An error spike is bad but an error drop is fine, so a high variant fits. A sensor going silent is bad but a chattier sensor is harmless, so a low variant fits. Third, is your data steady or sparse? If events arrive only now and then, use the non-zero variants so that empty buckets are not treated as real zeros.

When you use counts on data that has already been summarized before it reaches the job, for example by an aggregated datafeed, the job needs to know how many original documents each summary row represents. A dedicated setting called the summary count field carries that number, and forgetting it makes counts come out wrong. It is like receiving a box of letters that is labeled “one bundle” when it actually contains a hundred letters.

Other useful settings

Two smaller options deserve a mention. Exclude frequent tells the job to ignore extremely common values, such as a load balancer’s own address, so they do not drown out real clues among the influencers. Use null decides whether records missing the split field should be analyzed as their own group. These settings rarely matter on day one, but they solve puzzling behavior later.

A job is only useful if you can read its answers. This chapter shows what each result means and how to investigate an alert calmly.

Three levels of result

The job reports at three levels. Bucket results answer the big question: “How unusual was this whole time slice overall?” They carry one overall score, which combines everything that happened inside the slice. Record results give the detail: “Which specific entity and which detector was unusual, what did we expect, and what did we see?” Influencer results answer “who or what was involved?” and rank the entities across the whole time window.

Scores and severity

Every score falls on the 0 to 100 scale. Kibana groups scores into colored bands so you can scan quickly.

Remember that these bands measure unusualness, not badness. A critical anomaly can be wonderful news, such as sales tripling after a successful advert. The job cannot know what is good for your business. It only knows what is rare.

Actual, typical and the shaded band

Each record contains an actual value (what really happened) and a typical value (what the model expected). In the Single Metric Viewer, Kibana draws your data as a line, the model’s expected range as a shaded band around it, and anomalies as colored markers where the line escapes the band. Seeing the band move up and down through the week, following your real rhythm, is often the moment people understand how this feature works.

Anomaly Explorer

The Anomaly Explorer is the control room. It shows swim lanes, colored timelines that show when anomalies clustered, first for the whole job and then for each influencer value. Click a colored block and the lower part of the screen lists the anomalies inside it, with their actual and typical values, their influencers, and a small chart. It is designed for the question “I have got an alert; what happened around that time?”

Extra details inside a record

Each record also carries several helpful fields. A short function description says which detector produced it, so you can tell a “high count” story from an “average latency” story. The probability is the raw rarity behind the score. The multi-bucket impact tells you whether the anomaly was mostly a single-bucket spike or a longer drift across several buckets. And results for a bucket that has not yet finished are flagged as interim: they are early opinions that can be revised once the whole bucket has arrived. If you only want stable facts, look at finalized results; if you want the earliest possible warning, interim results give you a head start with a little less certainty. It is the difference between a referee’s quick signal and the confirmed decision after the video replay.

Annotations, forecasts and searching results

Kibana lets you add annotations, short notes pinned to a moment on the chart, such as “deployed version 4.2.” The system also adds its own notes, for example when the model changes its behavior after a permanent shift. And because results are ordinary documents in indices, you can build dashboards that mix anomaly scores with business data, or write searches that ask “show me every critical anomaly for the payments service last week.”

Once the basics work, these features remove noise, add foresight, and let you steer the model when your business does something unusual on purpose.

Forecasting

The model that learned your rhythm can also look ahead. A forecast extends the learned pattern into the future and draws it with an upper and lower boundary, so you can see where a measurement is likely to be tomorrow or next week. Capacity planning is the classic use: “at this growth rate, our disk fills in eleven days.” A forecast needs the job to be open and is best suited to count and metric style jobs that track a single series or a split series. Some detector types, such as rare, geographic and population styles, cannot be forecast. The weather forecast is the everyday picture: it does not promise, it gives a likely range.

Scheduled events and calendars

Some unusual moments are unusual on purpose: a sale, a planned maintenance window, a holiday. You do not want the job to shout about them, and just as important, you do not want the model to learn from them and get confused. A calendar holds a list of scheduled events, each with a start and an end, and is attached to one or more jobs. During an event the job skips producing results and skips updating the model. It is like telling the night guard, “Tonight is the building party, so ignore the noise and please do not adjust your idea of a normal night.”

Custom rules

Sometimes the model flags something mathematically rare but practically boring. Perhaps traffic dropped from 12 requests to 3, which is a large percentage change but a trivial amount. A custom rule lets you say “ignore results when the actual value is below 20,” or “ignore anomalies for this list of test servers.” Rules can skip the result, skip the model update, or both. They can use conditions on the actual or typical value, and they can point at a filter list, a reusable list of values such as trusted server names, that you can edit without touching the job. It is telling the guard, “The janitor always comes at 3 a.m., so please do not report him.”

Model snapshots and reverting

Every few hours, and when a job closes, the job saves its model snapshot. If the model absorbed a bad period, for example a long outage that it began to treat as normal, you can revert to an earlier snapshot and replay the data from there. It is a save point in a video game: if the last level went badly, you go back to before it. Snapshots are kept for a limited time by default, and older ones are trimmed automatically, so keep any that you specifically care about.

Model plot

Turning on model plot tells the job to write the model’s expected bounds for each bucket into the results. This is what lets Kibana draw the shaded band around your data. It costs extra storage and processing, especially with many entities, so enable it selectively, often only for a chosen few entities.

Latency and out-of-order data

Data does not always arrive in perfect time order. The latency setting gives the job a short grace period to accept slightly late records before closing a bucket for good. It is a teacher who says, “Hand in your homework by Friday, but I will accept it until Monday morning.”

Multivariate analysis by field

Normally each value of a split field is modeled independently. If the values move together, for example the temperatures of several sensors in the same room, an option called multivariate by fields lets the model also learn the correlations, so it can notice when one sensor breaks the usual relationship with its neighbors.

Job groups and custom links

You can label jobs with groups, such as “payments” or “network,” which makes it easy to manage many jobs together and to attach a calendar to a whole group. You can also add custom URLs to a job so that a person looking at an anomaly can click straight through to a prepared dashboard or a filtered log search for the same time and entity. Shortcuts like these turn a chart into a starting point for action.

What happens when you go from ten servers to ten thousand, or from thousands of events per minute to millions? This chapter explains the levers and the safety nets.

What actually costs resources

Three things drive the cost of a job. The first is the number of distinct entities it models, because each one needs its own model in memory. The second is the bucket span: shorter spans mean more buckets to analyze and more results to write. The third is the volume of data the datafeed has to search and transfer. Understanding these three tells you where to look when a job is slow or hungry.

The model memory limit

Each job has a model memory limit, its private backpack. The wizards estimate a sensible value from your data, and there is a cluster-wide ceiling to stop anybody from claiming too much. Setting it too small means the job reaches its limit and starts dropping detail. Setting it too large wastes memory that other jobs could use. Estimate first, add a healthy margin, and check the job’s memory status after a few days of real traffic.

Many small jobs or a few large ones?

A common mistake is creating one job per server. A hundred jobs means a hundred native processes, a hundred datafeeds and a hundred sets of overhead. One job that splits by server does the same analysis far more efficiently. As a general habit, use split fields inside one job instead of many separate jobs, and keep separate jobs for genuinely different questions or different data sources.

flowchart LR
    M["Master node - assigns jobs"] --> N1["ML node 1 - runs jobs A and B"]
    M --> N2["ML node 2 - runs job C"]
    M --> N3["ML node 3 - spare capacity"]
    N1 --> IDX["Results and state indices on data nodes"]
    N2 --> IDX
    N3 --> IDX
    N1 -.->|"if node 1 fails, job A moves"| N3

Ways to scale up

Capacity planning in plain steps

Planning capacity does not need a spreadsheet full of formulas. Follow four simple steps. First, list the jobs you plan to run and, for each, count the distinct entities it will model, for example 3,000 servers or 500 services. Second, look at the memory estimate that Kibana gives for each job, which already takes entities and detectors into account. Third, add the estimates together and add a safety margin so that growth and pruning behavior have room to breathe. Fourth, compare that total with the memory your machine learning nodes are allowed to use for native processes, and add nodes until you have that total plus spare room for the loss of your largest node. Repeat this exercise whenever the business adds a large new source of data.

Think of packing a moving truck. You list the boxes, estimate their sizes, add space for surprises, and only then decide how many trucks to rent. Renting one truck too few is far more painful than renting one too many.

Reliability: what if a node dies?

Machines fail. The good news is that a job’s state is not trapped in one place. The configuration lives in a system index, and the model is saved regularly as snapshots. If a machine learning node vanishes, the master node detects it, chooses another node with enough free memory, and restarts the job there from its latest snapshot. Data that arrived between the last snapshot and the crash is simply replayed from the datafeed, and results resume. It is a relay race where the baton is always saved in a locker before a runner can trip.

The safety net needs room to work. If every machine learning node is nearly full, a failed job has nowhere to go and waits in an unassigned state until capacity returns. The job’s status includes an explanation of why it could not be placed. Keep spare capacity, at least enough to absorb the loss of your biggest node, and use replicas for the system indices so their data survives too.

Other reliability details

Datafeeds can search across several clusters, so a central cluster can analyze data that lives elsewhere, provided the nodes are set up to reach the remote clusters. After a major version upgrade, saved model snapshots from an older version may need an upgrade step before a job can use them, so read the upgrade notes before touching production. Finally, results and job statistics are ordinary data, so you can back them up with the same snapshot repository policy you use for every other index.

No tool is perfect. Knowing where anomaly detection shines and where it struggles helps you use it wisely.

Trade-off one: sensitivity versus noise

Every detection system balances two mistakes. A false positive is a false alarm. A false negative is a missed problem. Raising the score at which you alert reduces false alarms but risks missing real trouble. Lowering it catches more but wakes people needlessly. There is no universally right level, so a good practice is to alert only on major and critical anomalies, review the minor ones during the working day, and move the dial as you learn your data.

Trade-off two: adapting versus forgetting

Because the model keeps learning, a slow and steady drift, such as a response time that creeps up a fraction of a percent every day, may eventually be accepted as normal. This is sometimes called the boiling frog problem. Fast jumps are caught easily, while very slow ones are absorbed. The remedy is to pair learned jobs with a few hard thresholds for values that must never be crossed, and to review long-term charts, not only alerts.

Trade-off three: speed versus certainty

Shorter buckets alert sooner, but each bucket holds less data and is therefore noisier. Longer buckets smooth out the noise but delay the alert. If you need both, run two jobs with different bucket spans on the same data: a short one for fast detection of big problems and a long one for careful detection of subtle ones.

How it compares with neighboring tools

These tools are teammates, not rivals. Anomaly detection tells you something changed; the analysis tools help explain the change; thresholds guard the hard limits.

Anomaly detection reads your data and writes new data about it. Both directions need care, and this chapter shows where the locks are.

Permissions come in two flavors

Elasticsearch security controls two separate things. Cluster privileges decide which actions a person may perform on the system as a whole, such as creating or deleting jobs. Index privileges decide which data a person may read. Anomaly detection needs both. To create a job, a person needs the right to manage machine learning and the right to read the source indices the job will study. It is like a library: one key opens the reading room, another key opens the rare book vault, and you may hold neither, one, or both.

Roles can be combined with your own custom roles. A common design gives analysts the viewer role plus read access to the dashboards they need, and gives only a handful of engineers the admin role.

Whose permissions does the datafeed use?

A datafeed runs in the background, possibly for months. When it searches your indices, it uses the permissions of the person who created it or who last updated it. If that person’s access is later reduced, the datafeed may start failing because it can no longer read the data, and a job that never sees data never finds anything. This is a surprising number of “my job stopped” mysteries. The fix is to create long-lived datafeeds under a well-governed service identity rather than a person who might change teams.

Results contain real values

Results are not anonymous numbers. When a job splits by user name or host name, those names appear in the results indices and in Kibana. So a person who can view results can learn which entities were involved, even if they are not allowed to read the raw logs. Treat the results indices as sensitive data of the same class as the source. If some teams must not see certain customers, use the security features that apply to the results, and use separate jobs and separate Kibana spaces for separate audiences.

Kibana spaces

Jobs can be assigned to spaces, which are like separate rooms inside one Kibana building. A payments team can see its jobs in its own space while a security team sees theirs, and neither is distracted by the other’s work. Spaces organize visibility; they are not a replacement for correct data permissions.

Protecting the machinery

The machine learning nodes talk to the rest of the cluster over the same internal channels as any other node, so those channels should be encrypted and authenticated in the usual way. Keep an audit trail of who creates, changes and deletes jobs, since a changed job can silently blind a monitoring system. A sneaky attacker who edits a rule to ignore his own machine has just tricked the guard.

The watcher also needs watching. A job that quietly stops is worse than no job at all, because it gives false comfort.

Job statistics

Every job keeps a running scoreboard. The counts tell you how many records were processed, how many were skipped for missing fields or bad timestamps, and when the latest record arrived. If the number of processed records stops rising while the datafeed says “started,” something upstream is broken. If the skipped counts jump, your data changed shape, perhaps because someone renamed a field during a release.

Model size statistics and memory status

The scoreboard also shows how big the model is and reports a memory status with three values. “OK” means all is well. “Soft limit” means the job is nearing its limit and is pruning models it thinks are least useful. “Hard limit” means the limit has been hit, new entities are no longer modeled, and data is being dropped. Think of a fuel gauge with a yellow warning light and a red light. You want to act on the yellow one.

Datafeed statistics

The datafeed keeps its own numbers: its state, how many searches it has run, how long they took, and whether it is currently assigned to a node. A datafeed that is stopped, or unassigned, or whose searches are getting slower over time, needs attention. Slow searches usually point to a query that is too heavy or an index that is too large.

Messages and notifications

Kibana’s job management screen has a Job messages tab. Every important event, such as the job opening, the datafeed starting, a memory warning, a delayed data warning or an error, is written there and also to a notifications index. Because they are stored as data, you can search them, and even alert on them.

An alert for the alerter

Kibana includes a special alerting rule type that checks the health of anomaly detection jobs themselves. It can notify you when a datafeed is not running, when a job is not open, when memory status is a concern, when delayed data is detected, or when errors appear in the job messages. Setting this up on day one is one of the best investments you can make.

Logs

The cluster’s own logs record the lifecycle of the native processes and any failures they hit, such as being unable to start because of a memory shortage. When a job fails and the messages tab does not explain it, the logs of the machine learning node usually do. Ship those logs into Elasticsearch too, and then you can build a dashboard that shows the health of your monitoring system next to the health of the things it monitors.

Finding an anomaly is only useful if the right person hears about it. This chapter connects results to notifications and to the products built on top of the feature.

flowchart LR
    J["Anomaly detection job"] --> R["Results index"]
    R --> AR["Anomaly detection alert rule - severity threshold"]
    R --> DASH["Dashboards and swim lane panels"]
    AR --> C1["Chat message"]
    AR --> C2["Email"]
    AR --> C3["Ticket or incident tool"]
    AR --> C4["Webhook to automation"]
    C4 --> ACT["Automatic action - scale out or block"]

The anomaly detection alert rule

Kibana’s alerting framework includes a rule type made for this job. You choose which jobs to watch, which kind of result to look at (bucket, record or influencer), and a minimum severity. The rule checks the results at regular intervals, looking back over a short window so that nothing slips through between checks. When a result crosses your severity level, the rule triggers actions through connectors: a chat post, an email, a ticket in your incident tool, or a web request that starts an automation.

A wise pattern is to send critical anomalies to the on-call person, major ones to a team chat channel, and everything below that to a dashboard that is reviewed during working hours. This keeps the night quiet and the day informed.

Elastic Security

Elastic’s security solution ships ready-made jobs for common threats: unusual processes on a machine, rare network destinations, suspicious logins, odd data volumes leaving a network, and patterns that resemble hidden channels through DNS. A detection rule type can turn an anomaly above a chosen score into a proper security alert, which then enters the same investigation workflow as any other alert. This is powerful because attackers who use stolen credentials do not trip signature-based rules, yet their behavior often differs from that of the real owner.

Elastic Observability

In the observability solution, anomaly detection is woven into the screens engineers use every day. Application performance monitoring can create jobs that watch each service’s latency, throughput and failure rate, and shows the anomaly right on the service map. Log screens use jobs that track log rates and categories, so a sudden flood or a new kind of message is highlighted. Infrastructure screens offer jobs for host memory, network traffic and container behavior. Engineers meet anomaly detection as a helpful colored marker on a chart they were already reading.

Dashboards

Swim lanes and anomaly charts can be embedded into ordinary dashboards. A single screen can then show an order-volume chart, the latest anomaly heat-map for that same period, and a list of the top influencers. Seeing the business number and the anomaly signal next to each other shortens investigation time dramatically.

Closing the loop with automation

Because an alert can call a web address, you can connect anomalies to automation: open an incident, attach a link to the anomaly, add capacity, or temporarily block a suspicious address. Automatic reactions deserve caution. Start with automation that only gathers information, and give it power to change things only after you have watched the detections for a while and trust them.

Patterns are recipes that experienced teams keep repeating because they work. Anti-patterns are the traps those same teams learned to avoid.

Patterns worth copying

Traps to avoid

A short, practical checklist drawn from the way careful teams roll this feature out.

Common mistakes and their cures

Here is how the same engine adapts to very different industries. Named companies below are used as familiar illustrations of the kind of scale and problem involved, not as claims about their internal systems.

A complete walk-through: checkout latency

To make everything concrete, here is how a team might build one job from start to finish.

Short answers to the questions that beginners ask most often.

The whole article in one place. If you remember only this box, you still know the essentials.