Start here. By the end of this chapter you will know what a cluster is, what “cross-cluster” means, and why the idea is useful.

An Elasticsearch cluster is a group of computers, called nodes, that work together to store and search data. Every cluster has a name, its own indices, its own settings and its own security rules. Normally, a search sent to a cluster looks only at the indices inside that cluster.

Cross-cluster search, usually shortened to CCS, changes that. It allows one cluster, called the local cluster, to run a search against indices that live on one or more other clusters, called remote clusters. The local cluster sends the request, the remote clusters do their share of the work, and the local cluster merges everything into one response. The person who asked sees a single answer. The data never moves permanently. It stays exactly where it was.

Searching in place, not copying

This last point is the heart of the idea. There are two big ways to make data in different places available together. You can copy it to one location, or you can ask the places where it already lives. CCS is the second way. No copy is made, no storage is doubled, and no pipeline has to be built and maintained. When you search, the question travels to the data instead of the data traveling to the question.

A short history

Before cross-cluster search existed, people who wanted to search several clusters used a special node type called a tribe node, which joined multiple clusters into one view. It worked, but it was awkward: it had to hold the state of every cluster it joined, it clashed on index names, and it was hard to secure. Cross-cluster search arrived in the 5.x series, in 2017, as a cleaner replacement. It let an ordinary node talk to other clusters through a simple connection, and the tribe node was retired in later major versions.

Over the years the feature has grown. Connections became easier to configure and easier to secure. Newer versions added a modern security model based on special keys, more detailed information in the search response, better handling of clusters that are temporarily unreachable, and the ability to use the same idea in other query languages. Kibana, the machine learning features, and the security and observability solutions all learned to use it. Today CCS is the standard way to build a “federated” Elasticsearch view across regions, teams and environments.

If clusters can hold so much data, why would anyone run several of them? Understanding the reasons explains why CCS is needed.

A newcomer often asks a fair question: “Why not put everything in one giant cluster?” In a perfect world that would be simplest. In the real world there are at least six good reasons to have more than one.

Reason one: distance

Information takes time to cross the ocean. A round trip between Frankfurt and Singapore is measured in hundreds of milliseconds, and that delay is added to every operation. Systems that write data all day, such as log collectors, work much better when the cluster is close to them. So each region often gets its own cluster.

Reason two: laws and promises

Many countries and industries require that certain data stay inside a certain border. European customer data may need to remain in Europe. A hospital may be unable to let patient records leave its own building. Separate clusters make it easy to keep each set of data in its proper place.

Reason three: limits and blast radius

Every cluster has practical limits: how many shards it can manage, how large its cluster state can grow, how long an upgrade takes. Beyond a certain size, one cluster becomes slow to coordinate. Also, if a single huge cluster has a serious problem, everything stops. Several smaller clusters mean that a failure affects only its own part. This is called limiting the blast radius.

Reason four: different owners

The payments team and the marketing team may have different budgets, different security rules, different upgrade schedules and different on-call people. Giving each its own cluster keeps their lives independent.

Reason five: safer upgrades

Upgrading one giant cluster is a big-bang event. With several clusters, you can upgrade a small, low-risk one first, learn from it, and roll the change out gradually.

Reason six: different jobs

Some clusters exist to ingest data quickly, some to serve searches for customers, and some to store cheap long-term archives. Each is tuned differently, and cramming them together forces awkward compromises.

The new problem: scattered answers

Splitting things up solves the problems above but creates a new one. Questions that span the whole company now have to be answered from several places. There are two clumsy ways to handle this. The first is copying everything into one central place. This doubles storage costs, needs pipelines to keep the copies fresh, can break the legal rules that caused the split in the first place, and always shows slightly old data. The second is asking each cluster separately and merging the answers by hand, which is slow, error-prone and impossible to put on a single dashboard.

This does not mean CCS is right for every situation. Later chapters explain when copying, through a feature called cross-cluster replication, is the better tool. But for the very common need to search several live clusters at once, CCS is the natural answer.

A handful of words unlock the whole feature. Each one is explained with what it is, why it exists, where you meet it, and a picture from everyday life.

The local cluster

The local cluster is the cluster that receives your search. It is the one Kibana or your application talks to. It coordinates the whole operation: it decides which clusters to contact, sends the requests, waits for the answers, and merges them. It is the librarian you walked up to.

The remote cluster

A remote cluster is any other cluster that the local cluster has been configured to reach. It does the searching on its own data and sends back results. It is a partner library.

The cluster alias

Every remote cluster is given a short nickname on the local side, called its alias, such as “eu_west” or “us_east”. The alias is chosen by you, and it is only a label used on the local cluster. It does not have to match the real name of the remote cluster. Aliases are how you point a search at a particular remote. Think of them as the names in your phone’s contacts list: you save “Grandma” whatever her real name is.

The remote index expression

To search a remote index, you write the alias, then a colon, then the index name, for example “eu_west:logs-2026”. You can use stars as wildcards, so “eu_west:logs-*” means every log index on that cluster, and “*:logs-*” means every log index on every remote cluster. Indices without a prefix are searched on the local cluster as usual. It is like writing “Grandma: recipes” to say which person’s cookbook you mean.

Seed nodes and gateway nodes

To connect to a remote cluster, the local cluster needs an address to knock on. In the common connection style, you give it a few seed nodes, which are addresses of machines in the remote cluster. It connects to one of them, asks about the other machines, and then keeps connections open to a small number of suitable ones, called gateway nodes. Gateways are the front doors through which requests enter the remote cluster. Seeds are how you find the building; gateways are the doors you actually use.

The connection mode

There are two ways to build the connection. In sniff mode the local cluster discovers and connects to nodes on its own. In proxy mode it connects to a single fixed address, such as a load balancer, and lets that address forward traffic. Chapter 8 compares them in detail.

Minimizing round trips

A search has several phases, and each phase can require a conversation across the network. A setting called minimize round trips decides whether the local cluster hands the whole job to each remote cluster in a single conversation, or whether it manages every phase itself. It matters a great deal for speed, and Chapter 5 explains it fully.

Skipping unavailable clusters

Remote clusters sometimes go offline. A per-cluster setting decides whether such a cluster causes the whole search to fail, or whether it is quietly left out with a note in the response. It is the difference between “the phone call fails if any partner library is closed” and “we will give you what the open libraries found, and tell you which ones were closed.”

Trust and credentials

Clusters should not accept requests from strangers. Cross-cluster search therefore relies on a trust arrangement. In the older style, the clusters trust each other’s security certificates. In the newer style, the remote cluster hands out a special cross-cluster API key, which the local cluster stores and presents with each request. The key is like a visitor badge that says exactly which doors it opens.

Now let us zoom out and look at the machinery: which parts exist, where each one lives, and how they connect.

Cross-cluster search is not a separate product. It is a capability built into every node. There is no special “CCS server” to install. You simply tell a cluster about its remote neighbors, and its nodes learn to talk to them. The diagram below shows a typical arrangement with one local cluster and three remote clusters in different regions.

flowchart LR
    USER["User or application"] --> KB["Kibana or API client"]
    KB --> LB["Load balancer"]
    LB --> LC["Local cluster - coordinating node"]
    LC --> GW1["Gateway node in eu_west cluster"]
    LC --> GW2["Gateway node in us_east cluster"]
    LC --> GW3["Gateway node in ap_south cluster"]
    GW1 --> D1["Data nodes and shards in eu_west"]
    GW2 --> D2["Data nodes and shards in us_east"]
    GW3 --> D3["Data nodes and shards in ap_south"]
    LC --> LD["Local data nodes and shards"]

Walking through the diagram

On the left, a person or an application sends a search, usually through Kibana or a web service. Often a load balancer sits in front so that requests are spread across the local cluster’s nodes. One node in the local cluster picks up the request and becomes its coordinating node, the node in charge of that particular search. It looks at the indices named in the request. Names without a prefix belong to the local cluster and are searched there. Names with an alias prefix belong to a remote cluster, so the coordinator sends a request across the network to a gateway node in that cluster. The gateway node coordinates the work inside its own cluster, talking to the data nodes that hold the relevant shards, and returns results to the local coordinator, which merges everything.

The components in detail

The special node role

Not every node needs to talk to other clusters. Elasticsearch has a dedicated node role for it, called the remote cluster client role. Only nodes with this role open connections to remote clusters. By default, nodes usually have it, but in a carefully designed production cluster you may give the role only to a small set of nodes, for example a few coordinating-only nodes, so that connections, memory and network usage are concentrated in a small, well-understood place. It is like designating one room in the head office as the “international calls” room.

Everything is symmetrical, yet each direction is separate

Setting up CCS from cluster A to cluster B does not automatically let B search A. Each direction is its own configuration, with its own trust and its own permissions. This is a helpful safety property: a central search cluster can be allowed to look at many regional clusters, while the regional ones cannot look at each other or back at the center.

This is the engine room. You do not need to be an expert in distributed systems, because every idea comes with a picture.

First, a quick reminder of how one cluster searches

Inside a single cluster, an index is split into pieces called shards, spread over many nodes. A search has two main phases. In the query phase, every relevant shard finds its best matching documents and sends back a small list of their identifiers and scores. The coordinating node merges those lists into one ranked list. In the fetch phase, the coordinator asks only the shards holding the winning documents to send the full text. It is a talent show: every shard sends its best acts to a central judge (query), and the judge then invites only the finalists to perform in full (fetch).

Two ways to extend that across clusters

When some shards live in other clusters, Elasticsearch can extend this two-phase process in one of two ways, chosen by a setting known as minimize round trips. In current versions the default is to minimize.

Mode one: minimize round trips (the default)

Here the local coordinator behaves like a project manager who delegates entire tasks. For each remote cluster, it sends a single request that says, “Run this whole search over your own indices and send me your final top results.” The remote cluster performs both the query phase and the fetch phase on its own, using its own coordinating node, and returns a finished, ranked list. The local cluster does the same for its own indices. Finally, the local coordinator merges the finished lists, which requires only a light bit of work.

sequenceDiagram
    participant U as User or Kibana
    participant L as Local coordinating node
    participant R1 as Remote cluster eu_west
    participant R2 as Remote cluster us_east
    U->>L: One search request
    L->>R1: One request for eu_west indices
    L->>R2: One request for us_east indices
    R1-->>L: Finished top results from eu_west
    R2-->>L: Finished top results from us_east
    L->>U: One merged response with cluster status

The great advantage is fewer trips across slow networks. Since a trip from one continent to another may cost a hundred milliseconds or more, doing one trip instead of four is a big saving. Another advantage is that a slow or busy remote cluster does not hold the local cluster hostage between phases.

Mode two: do not minimize round trips

In the other mode, the local coordinator behaves like a hands-on manager who wants to supervise every step. It treats every shard, whether local or remote, as if it belonged to one big cluster. It first checks which shards could possibly match, then runs the query phase across all clusters, merges the ranked lists itself, and finally reaches back to the specific shards holding the winning documents for the fetch phase. This requires more conversations across the network, but the local coordinator sees every shard result directly.

flowchart TD
    A["Local coordinator receives the request"] --> B["Ask all shards in all clusters whether they could match"]
    B --> C["Query phase runs on every relevant shard in every cluster"]
    C --> D["Local coordinator merges and ranks all shard results"]
    D --> E["Fetch phase asks only the shards holding the winners"]
    E --> F["Final merged response is returned"]

How results are merged

Whichever mode is used, the local coordinator finishes by combining lists. For sorted or scored results, it takes the top entries from every list and interleaves them into one ranked list. For aggregations, such as counting how many errors each service produced, it adds the partial counts from every cluster together. This is why a single dashboard over three clusters can show one coherent chart. Note that some aggregation results are approximate when data is spread across many shards and clusters, in the same way that a poll of several towns gives an estimate of the whole country.

A worked example with real numbers

Suppose a support engineer searches for the last fifteen minutes of errors for one customer, across a local cluster and two remote clusters. The local cluster needs 30 milliseconds. The cluster eu_west, close by, needs 60 milliseconds plus a 20 millisecond network trip each way, so about 100 milliseconds in total. The cluster ap_south, on the other side of the world, needs 80 milliseconds of its own work and a network trip of 180 milliseconds each way, about 440 milliseconds. Because the requests are sent in parallel, the user waits roughly 440 milliseconds, plus a few milliseconds of merging.

Now compare the talkative mode. If the local coordinator had to talk to ap_south four separate times, once to check which shards could match, once for the query phase, once more for the fetch phase and once to close the search context, the trips alone would cost around four times 360 milliseconds, or more than a second and a half. The numbers are only illustrations, but the lesson is real: on long-distance links, the number of conversations matters more than the amount of work, and that is exactly why the default is to minimize them.

The scoring wrinkle

Search engines rank documents by relevance scores, and those scores depend partly on statistics about the data, such as how common a word is. Each cluster computes those statistics from its own data. So a word that is rare in one region and common in another can yield slightly different scores for otherwise similar documents. For most log and analytics use cases this makes no practical difference, because results are sorted by time or filtered by exact values. For text search, where ranking quality matters, be aware that scores from different clusters are not perfectly comparable unless you choose a search type that collects statistics globally, which requires the more talkative mode.

Where the index name goes

When results come back, each hit records which index it came from, and remote hits carry the alias prefix, for example “eu_west:logs-2026”. This means that two clusters can safely hold indices with exactly the same name, and you can always tell them apart. Without the prefix, the same name on two clusters would be a confusing collision.

A remote connection is configured, established, used, sometimes broken, and repaired. Knowing each stage prevents most surprises.

The life of a connection

When you save a remote cluster’s settings on the local cluster, it immediately tries to connect. If the remote answers, the connection becomes live and stays open in the background, kept alive by small periodic messages so that firewalls do not close it for being idle. If the connection breaks, for example because of a network outage or a remote restart, the local cluster notices and tries again on a schedule until it works. A connection that has been established is like a phone line that stays open: the next search does not have to dial.

stateDiagram-v2
    [*] --> configured
    configured --> connecting : settings saved
    connecting --> connected : remote reachable
    connecting --> disconnected : timeout or refused
    connected --> disconnected : network break
    disconnected --> connecting : scheduled retry
    connected --> [*] : remote removed

The life of one search

Data stays put

At no point is your data stored at the local cluster. Documents travel over the network only as part of a result, they are held briefly in memory to be merged, and then they are sent to the caller and forgotten. If a remote cluster deletes a document a second later, it is gone from the next search. This is why CCS results are always as fresh as the source clusters.

Parallel work, single wait

Because the local cluster contacts all clusters at the same moment, the total waiting time is roughly that of the slowest one, plus the merge. If two clusters answer in 80 milliseconds and one answers in 900 milliseconds, the user waits about 900 milliseconds. This makes it important to understand which cluster is the slow one, and Chapter 10 explains how the response reveals it.

Before going deeper, here is the route you follow to connect two clusters for the first time. Knowing the road makes the later chapters easier to follow.

Setting up cross-cluster search is less about clever commands and more about careful preparation. Most failures come from the network or from permissions, not from Elasticsearch itself. The steps below describe the journey for a beginner who wants a central cluster to search a regional cluster.

Before you begin

Check three things. First, the versions. Elastic publishes a compatibility table that tells you which versions of a local cluster may search which versions of a remote cluster. In general the clusters should be on the same major version, or on close neighbors, and it is best to keep them on similar versions. Second, the subscription. Some parts of the feature, especially the newer key-based security model, can depend on the subscription level, so confirm what your license includes. Third, the network. The local cluster must be able to open a connection to the remote one, which means the right addresses, the right ports and no blocking firewall in between.

The two trust styles in one paragraph each

With the certificate-based style, the clusters talk over their normal internal transport channel. Each cluster is configured to trust the certificate authority of the other, so their encrypted connections are accepted. Permissions are then decided by matching the roles of the searching user on both sides. It works well but gives a broad kind of trust between clusters.

With the newer API key-based style, the remote cluster opens a separate, dedicated network interface just for cross-cluster traffic. An administrator on the remote cluster creates a special key that names exactly which indices may be searched, and hands it to the local cluster, which keeps it in its secure settings store. Every request carries the key. The remote can revoke it at any time, and the access it grants is narrow and explicit. Chapter 14 covers security in more detail.

The local cluster can reach a remote one in two different ways. Choosing the right one depends on how your network is built.

flowchart LR
    subgraph SNIFF["Sniff mode"]
        L1["Local node"] --> S1["Seed node"]
        S1 -.->|"shares list of nodes"| L1
        L1 --> G1["Gateway node A"]
        L1 --> G2["Gateway node B"]
        L1 --> G3["Gateway node C"]
    end
    subgraph PROXY["Proxy mode"]
        L2["Local node"] --> P["Load balancer or proxy address"]
        P --> R1["Remote node 1"]
        P --> R2["Remote node 2"]
    end

Sniff mode

Sniff mode is the traditional style and the usual default. You provide one or more seed nodes. The local cluster connects to a seed, asks for the list of nodes in the remote cluster, and then chooses a small number of them, three by default, that are allowed to accept cross-cluster requests. It keeps those connections open. If a gateway node disappears, the local cluster picks another one from the list. This gives good resilience with no extra equipment.

The catch is that the remote nodes must be directly reachable by the addresses they announce about themselves. If a remote node lives behind a network translation layer or inside a container network and announces a private address that the local cluster cannot reach, the seed connection works but the discovered gateways do not. This produces the confusing symptom “it connects and then immediately fails.”

Proxy mode

In proxy mode you give a single address, typically a load balancer or a gateway service that forwards traffic to remote nodes. The local cluster opens a fixed number of connections to that address and does not try to discover or connect to individual remote nodes. This is ideal when only one entry point is exposed to the outside, for example when the remote cluster runs inside a container platform, behind a cloud load balancer, or behind a strict firewall. Proxy mode can also carry a server name so that a shared entry point knows which cluster the traffic is meant for.

Ports and firewalls

Cross-cluster traffic does not use the normal HTTP port that applications use to search. Under the certificate-based style, it uses the internal transport port that nodes use to talk to each other, which is often 9300 by default. Under the newer key-based style, a dedicated remote cluster interface listens on its own port, which is often 9443 by default. Either way, you must allow the local cluster’s nodes to reach that port on the remote side, and you should allow only those source addresses rather than the whole internet.

Keeping connections alive

Firewalls and load balancers love to close connections that look idle. Cross-cluster links can sit quiet for a while between searches, so Elasticsearch can send small keep-alive pings on a schedule, and you can adjust that schedule if your network closes idle connections quickly. Optional compression of the traffic can also reduce the number of bytes on long-distance links, at the cost of a little processor time on both ends.

Once clusters are connected, using them is pleasantly simple. This chapter shows how to name targets, which tools support them, and how Kibana fits in.

The naming pattern

The whole trick is the prefix. An index written as plain “logs-app” is searched on the local cluster. An index written as “eu_west:logs-app” is searched on the cluster whose alias is eu_west. You can mix as many targets as you like in one request: a local index, one index from eu_west, and a wildcard from us_east. The response contains hits from all of them.

Wildcards

Wildcards make the pattern powerful. A star after the colon selects many indices on one cluster: “eu_west:logs-*”. A star in the alias position selects many clusters: “*:logs-*” means every log index on every configured remote cluster, and “eu_*:logs-*” means every remote whose alias begins with “eu_”. This is convenient for a single Kibana data view that spans the whole company, but it deserves caution. A careless star can send a heavy search to every cluster you own.

Which tools understand remote names

The core search interface accepts remote names, and so do the multi-search interface for sending several searches at once, the field capabilities interface that Kibana uses to learn which fields exist, and the asynchronous search interface that runs long searches in the background and returns partial results as they arrive. Utilities for checking which indices and remote clusters a pattern will actually match also exist in recent versions. Newer releases have extended cross-cluster querying to other query languages as well, though the exact coverage depends on your version, so check the documentation for yours.

Using CCS in Kibana

Kibana works through data views. To make remote data visible, create a data view whose pattern includes the alias, such as “*:logs-*” for all remote logs, or “logs-*,*:logs-*” for local and remote together. After that, Discover, Lens, dashboards, maps and most other tools can use it without knowing anything about clusters. Users simply see one big combined data set, and a field showing the index name reveals which cluster each document came from.

Three everyday search shapes

Most real cross-cluster searches fall into three simple shapes. The first is the needle search: find every event that mentions one customer, one order or one request identifier, anywhere. It returns very few documents, so it is cheap on the network, and it is the classic support-desk request. The second is the count search: how many errors per region in the last hour? It returns only numbers, so almost no data crosses the network, and it is the natural shape for dashboards. The third is the browse search: show me the most recent hundred events across all regions, newest first. It returns real documents, so keep the size small and the time range tight. Choosing the right shape for the question keeps searches fast and kind to every cluster involved.

Consistent field names matter

When several clusters feed one view, they must agree on what fields mean. If one cluster stores the response time as a number and another stores it as text, Kibana shows a conflict and some charts and sorts stop working. If one region calls the field “client_ip” and another “clientIp”, a single filter will not match both. The best protection is to agree on a common schema across all clusters before you start, and Elastic’s own common schema exists for exactly this purpose. It is like making sure every office uses the same form: “date of birth” must mean the same thing, in the same format, everywhere.

A cross-cluster answer can be complete, or it can be missing pieces. The response tells you which, if you know where to look.

The cluster summary

Every cross-cluster response carries a small summary of how the clusters behaved. It counts the total number of clusters involved (including the local one), how many were successful, how many were skipped, and, in versions that support asynchronous searches, how many are still running or returned only partial results. Newer versions can also include a detailed entry for each cluster: its status, how long it took, how many shards it searched, and how many of those succeeded, were skipped or failed. This is the first place to look whenever a number “looks too low.”

Shards, timeouts and partial results

The response also has a general summary of shards, covering both local and remote ones: how many were searched, how many were skipped because their time range could not match, and how many failed. A timed out flag says whether any part of the search ran out of time and returned only what it had found so far. Failures on some shards do not always break the search. Depending on your settings, Elasticsearch may return the hits it did collect and mark the response as partial, rather than failing entirely.

What “skip unavailable” really decides

Each remote cluster has a setting that controls what happens when it cannot be reached. If skipping is off for that cluster, an unreachable cluster makes the entire search fail with an error. If it is on, the cluster is left out, the search continues with the others, and the cluster summary records the cluster as skipped. Which is right depends on the question you are asking.

Investigating a suspicious result

Once the basics work, these features let you tune speed, accuracy and safety for your particular setup.

Choosing the round trip strategy per search

The minimize-round-trips choice can be made for each search request, not only globally. A dashboard that spans distant regions should stay with the efficient default. A text-search screen that must rank results consistently across nearby clusters might turn minimizing off and choose a search type that gathers global statistics. Making this choice per request is like choosing between a phone call and a video conference for each meeting depending on what it needs.

Asynchronous search

Some searches are simply big: months of data, heavy aggregations, many clusters. An asynchronous search starts the work, returns a handle immediately, and lets you fetch progress and partial results while the rest continues. With cross-cluster search this shows results from fast clusters early while slower ones are still working. Kibana uses this idea to show charts that fill in progressively. It is like ordering at a restaurant and receiving each dish as the kitchen finishes it, instead of waiting for the entire meal.

Skipping shards that cannot match

Before doing real work, Elasticsearch can ask each shard a cheap question: “Could you possibly have documents in this time range?” Shards that say no are skipped entirely. This is especially valuable across clusters holding long histories, because a request for the last hour then touches only a few recent shards on each cluster. Well-designed time-based indices make this pre-check extremely effective.

Aliases and filtered aliases on the remote side

The administrator of a remote cluster can publish an index alias, a friendly name that points to one or many indices, possibly with a built-in filter. Searchers on the local side then use the stable alias and never need to know the real index names. If the remote administrator later reorganizes indices, the alias stays the same. A filtered alias is also a simple way to expose only a slice of the data, for example only the documents of one customer. Think of it as a labeled window that shows only part of the warehouse.

Data streams

Time-based logs and metrics are often stored in data streams, which manage rolling indices automatically. Cross-cluster search can read remote data streams by name just as it reads plain indices, so a central view of a whole fleet of data streams needs no special handling.

Connections are not transitive

If cluster A can search B, and B can search C, it does not follow that A can search C. Each local cluster needs its own direct connection to every cluster it wants to search. This keeps the trust picture simple and safe, since access never flows secretly through a middleman.

Beyond ordinary searches

Other Elastic features can also use remote clusters. Kibana uses them through data views. The machine learning anomaly detection feature can read data for its jobs from remote clusters, so a central team can monitor many regions with jobs running in one place. Newer query languages and some analytic tools have been extended to accept remote targets. The set of supported features grows with each version, so check the documentation for yours before you design around a particular capability.

Preflight checks

Recent versions include interfaces that tell you, before you run a real search, which indices a pattern would match on each cluster and whether each remote is connected and compatible. Using them in scripts and health checks catches problems early. It is like calling ahead to confirm a restaurant is open before driving across town.

What happens when you go from three clusters to thirty, or from thousands of searches per hour to millions? This chapter explains the levers and the safety nets.

What actually costs time

Four things dominate the cost of a cross-cluster search. The first is network latency between the local cluster and each remote. The second is the amount of work each cluster does: how many shards it searches, how heavy the aggregations are. The third is the size of the results that must cross the network, because millions of returned hits cost more than a few hundred. The fourth is the slowest participant, since the local cluster waits for everyone. Understanding these four tells you where to look when something is slow.

Ways to make searches faster

Ways to scale

The local cluster does the merging, so its coordinating nodes need enough memory to hold partial results from every cluster while combining them. For heavy use, run dedicated coordinating nodes for user searches, separate from data nodes, and add more of them as the number of concurrent searches grows. If users are spread across the world, deploy a hub in each region, each connected to the same set of data clusters, so that people search a nearby hub and only the fan-out crosses the ocean. If one remote cluster is the bottleneck, look at its own health: more data nodes, better shard sizing and a tuned time-based index layout help far more than changing anything on the local side.

Capacity planning in plain steps

Planning capacity for a hub does not need a spreadsheet full of formulas. Follow four simple steps. First, estimate how many searches per second will arrive at the hub at the busiest time, and how many clusters an average search will touch. Multiply them: that is the number of remote requests the hub must handle each second. Second, estimate the typical result size per remote request, in documents or in aggregation buckets, because that decides how much memory the merging needs. Third, size the coordinating nodes so that they can hold several such merges at once with room to spare. Fourth, check the network: the total bytes per second returning from every remote must fit comfortably in the hub’s bandwidth, with headroom for bursts. Repeat the exercise whenever a new dashboard, a new cluster or a new team of users is added.

Think of packing a delivery van for many stops. You count the parcels, estimate their size, leave room for surprises, and only then decide how many vans you need. Sending one van too few is far more painful than sending one too many.

Reliability: what if a remote cluster fails?

Because cross-cluster search stores no data of its own, there is nothing at the local side to lose. The risk is different: if a remote cluster is unreachable, its data is missing from your answers until it returns. What happens then is governed by the skip-unavailable setting, as shown below.

flowchart TD
    S["Search sent to a remote cluster"] --> Q{"Remote reachable and answering?"}
    Q -->|"Yes"| OK["Results merged into the response"]
    Q -->|"No"| K{"Skip unavailable is on for this remote?"}
    K -->|"Yes"| SK["Cluster skipped and noted in the cluster summary"]
    K -->|"No"| ER["Whole search fails with an error"]
    SK --> PR["Partial results returned"]

To make links more resilient, give sniff mode several seed addresses so one failed seed does not block a connection. For proxy mode, use a load balancer that checks the health of the nodes behind it. Set a sensible connection timeout so a dead remote fails fast instead of making every search wait. And monitor connection status continuously, so a broken link is a page to a person and not a surprise on a dashboard.

No tool is perfect. Knowing where cross-cluster search shines and where it struggles helps you choose it, or something else, wisely.

Trade-off one: freshness versus speed

Searching in place gives fresh answers but pays the network price every time. Copying data to a nearby cluster gives fast answers but adds delay and cost. If a user needs the second-by-second truth from a far region, CCS is the honest tool. If a user needs a very fast, local, always-available copy, replication is better.

Trade-off two: availability versus independence

Because CCS depends on each remote being reachable at search time, the availability of a combined answer is lower than the availability of any single cluster. If three clusters each answer 99.9 percent of the time, a search that needs all three succeeds a little less often. Skipping unreachable clusters improves availability at the price of completeness.

Trade-off three: freedom versus consistency

Independent clusters let each team choose its own settings, schemas and upgrade schedule. But a combined view only works well when everyone agrees on the basics. The price of independence is a small amount of shared discipline about names, types and time fields.

How it compares with neighboring tools

These tools are teammates, not rivals. Many organizations combine them: replication for the small set of data that must survive a regional disaster, and cross-cluster search for the rest.

Cross-cluster search opens doors between systems that used to be separate. This chapter shows where the locks are and how to fit them.

Three layers of protection

Security for cross-cluster search has three layers. The first is the network: only the machines that need to talk to each other should be allowed to. The second is encryption and identity between clusters: the two clusters must prove who they are and keep the traffic private. The third is permissions for people: even when clusters trust each other, a person should see only the indices their role allows. A break in any layer weakens the others.

Certificate-based trust

In the older style, clusters use the same internal channel that their own nodes use, and they are configured to trust each other’s certificate authorities. The remote cluster decides what the searching person may read by looking at the person’s role names. This means that a role with the same name must exist on both sides, with suitable privileges on the remote indices. It works, but the trust is broad: a cluster that is trusted at this level is trusted quite deeply, and role names have to be kept in step across clusters.

API key-based trust

The newer style is designed to be narrower and easier to reason about. On the remote cluster, an administrator opens a dedicated interface for cross-cluster traffic and creates a cross-cluster API key. The key says, in plain terms, “the holder may search these indices” and, if replication is used, “may follow these indices.” The key is then stored in the local cluster’s protected settings store, never in a normal settings file, and is presented on every request. The remote cluster can revoke or expire the key at any time.

On the local side, the person’s role must also allow searching the remote indices, using a special kind of privilege that names both the remote clusters and the index patterns. The effective access is the overlap of the two: what the key allows on the remote and what the user’s role allows locally. Neither side can grant more than the other permits. It is like a building where a visitor badge opens only certain floors, and the visitor’s own employer has also approved only certain floors. The visitor can enter only the floors on both lists.

Keeping secrets secret

The API key is a secret. It must live in the cluster’s protected secure settings, which are kept outside ordinary configuration files and can be reloaded without a restart. Rotate keys on a schedule, give them an expiry date where possible, and remove keys that are no longer used. A leaked key that is never rotated is like a spare house key hidden under a doormat forever.

Least privilege

Give every key and every role the narrowest access that lets the work be done. If the central hub only needs to search the logs of a service, the key should name only those log indices, not everything. If analysts need to read only one region, their role should say that region. Narrow access reduces the harm from mistakes and from stolen credentials.

Data crossing borders

A subtle but important point: although CCS does not store remote data locally, search results still travel over the network and appear on the screen of the person who asked. If a person in one country searches a cluster in another and views the results, personal data may effectively cross a border. For regulated data, this can matter as much as copying would. Check with your legal and privacy teams whether viewing remote results is allowed, and restrict remote access accordingly. Filtered aliases, field-level restrictions and narrow keys are useful tools here.

Audit trails

Turn on auditing on both sides. The local cluster records who asked, and the remote cluster records which key or user reached it and what it read. When something goes wrong, or when an auditor asks who saw what, these two logs together tell the whole story.

A link between clusters can quietly break. Good monitoring tells you before your users do.

Connection status

The most basic check is whether each remote cluster is connected. The remote cluster information interface, and the Remote Clusters screen in Kibana, list every configured alias with its connection mode, whether it is currently connected, how many nodes are linked, and the settings in force, such as the skip-unavailable choice. A remote that shows as disconnected, or that has fewer connected nodes than usual, deserves attention. Put this check on a schedule and alert on it. It is a smoke detector for the link.

The response as a metric

Every cross-cluster response contains the cluster summary discussed earlier. If you log it, you gain a running history of which clusters answered, which were skipped and how long each took. A gradual increase in one cluster’s response time is an early warning of trouble at that cluster or on the path to it. A sudden rise in skipped clusters is a loud warning of an outage.

Slow logs and tasks

Each remote cluster can record slow searches in its own slow logs, once you set thresholds. When a cross-cluster search is slow, the remote’s slow log shows what it was asked to do and how long it took, which quickly separates “the network is slow” from “the query is heavy.” A task listing interface shows searches currently running, including the parts that are waiting on remote clusters, so you can spot a stuck request.

Health of the local coordinators

Because the local nodes do the merging, keep an eye on their memory, on the queue of waiting searches and on the count of rejected searches. Rising rejections mean that the coordinators are overloaded, and adding coordinating nodes or reducing result sizes is the cure.

A link health dashboard

A small dedicated dashboard for link health pays for itself the first time something goes wrong. Put on it, for each remote cluster, the connection state, the number of connected nodes, the recent average response time, the count of skipped or partial responses, and the days left before the current key expires. Add a simple test search that runs every minute against each remote and records how long it took. If any tile goes red, the on-call person sees at a glance which link is the problem, instead of hunting through logs. Think of the dashboard in a car: you do not want to open the bonnet every time, you want a light that tells you which system needs attention.

Logs

The cluster logs record when connections are opened, when they break and when they are retried. When a link misbehaves, these messages usually name the address that could not be reached or the certificate that was refused. Ship them into Elasticsearch, and then a dashboard can show the health of your links beside the health of the systems they serve.

Cross-cluster search is a building block. Here is how organizations assemble it into larger designs and how other Elastic features build on it.

The hub-and-spoke design

The most common design is a central hub cluster that users connect to, and several spoke clusters that hold data. The hub has Kibana, the user accounts and the shared dashboards. The spokes ingest and store data close to where it is produced. Every user request enters the hub, which fans it out to the right spokes.

flowchart TD
    ANALYST["Analysts and dashboards"] --> H
    subgraph HUB["Search hub cluster"]
        H["Kibana and coordinating nodes"]
    end
    H --> E["eu_west spoke - European data"]
    H --> U["us_east spoke - North American data"]
    H --> A["ap_south spoke - Asian data"]
    H --> M["monitoring spoke - health data"]

The advantages are clear. Users need to learn only one address. Access rules can be managed centrally. Spokes can be added, removed or upgraded independently. And each spoke can be small, simple and close to its data. The hub must be designed carefully, since it is the front door for everyone, so give it dedicated coordinating nodes and plenty of memory.

Regional hubs

For a worldwide organization, a single hub can be far from many users. Regional hubs solve this: each region has its own hub, all connected to the same spokes. People in Asia search the Asian hub, which is close to them, and only the fan-out crosses long distances. This adds some configuration but greatly improves the feel of the tools.

Hybrid and migration scenarios

Cross-cluster search is a friend during moves. A company migrating from its own data center to the cloud can run old and new clusters side by side and search both from one view while data is gradually moved. During a major upgrade, a new cluster can be built and joined by remote link, so users search old and new data together until the switch is complete.

Other features that use remote clusters

Patterns are recipes that experienced teams keep repeating because they work. Anti-patterns are the traps those same teams learned to avoid.

Patterns worth copying

Traps to avoid

A short, practical checklist drawn from the way careful teams roll this feature out, followed by the mistakes that trip up nearly everyone once.

Common mistakes and their cures

Here is how the same idea adapts to very different industries. Named companies below are used as familiar illustrations of the kind of scale and problem involved, not as claims about their internal systems.

A complete walk-through: the support desk

To make everything concrete, here is how a team might build the support-desk scenario from the start of this article, from a blank page to a working tool.

After the launch, the twenty-minute, three-dashboard hunt from the beginning of this article becomes one search, and the customer is off hold in seconds.

Short answers to the questions that beginners ask most often.

The whole article in one place. If you remember only this box, you still know the essentials.