Global Load Balancing Across Clusters in Kubernetes

Global Load Balancing Across Clusters in Kubernetes

One cluster in one region is a single point of failure and a long trip for faraway users. This guide explains, from first principles, how to spread traffic across several clusters around the world, keep it fast and safe, and recover gracefully when a whole region goes dark.

Imagine a pizza chain with a single shop in one city. Customers in that city get hot pizza quickly. Customers three cities away wait an hour for a lukewarm box. And on the day the shop’s oven breaks, nobody anywhere gets pizza at all. The obvious answer is to open shops in several cities. But that creates a new question: when a customer calls the central phone number, who decides which shop takes the order? The answer should be the closest shop that is open, has ingredients, and is not overwhelmed. Making that decision quickly, fairly, and correctly, thousands of times per second, is the whole subject of global load balancing.

Kubernetes clusters are like those shops. A single cluster is excellent at spreading work across its own machines, but it lives in one place and shares one fate. Real products serve people on several continents and cannot afford to disappear when one data center has a bad day. So teams run several clusters in different regions and put a smart layer in front that directs each user to the best one. This tutorial explains how that layer works, the different ways to build it, the traps along the way, and how to operate it with confidence.

1What Is Global Load Balancing?

Global load balancing decides which location should serve each request, not just which machine inside one location.

Local versus global

Ordinary load balancing happens inside a single site. Requests arrive at one entrance, and a balancer hands them out among many identical servers, so that no single server is overwhelmed. Global load balancing adds a layer above that one. It chooses between whole sites, such as a cluster in Mumbai, a cluster in Frankfurt, and a cluster in Virginia, and only then does each site’s local balancer spread the work among its own machines. You can think of it as two levels of decision: first choose the right city, then choose the right shop.

What the global layer is trying to achieve

The global layer pursues several goals at the same time, and they sometimes pull in different directions. It wants users to be served quickly, which usually means from a nearby location. It wants the service to stay available when a location fails, which means having another location ready and noticing the failure quickly. It wants to avoid overloading any one location, which means sharing load in proportion to capacity. And it often needs to obey rules, such as keeping certain users’ data inside a particular country.

Simple Analogy

Think of an airline’s reservation center that assigns passengers to flights. It tries to put each passenger on the nearest convenient departure, but if that flight is full or cancelled, it quietly rebooks them on another one. The passenger sees a single booking line, while behind it a set of rules juggles many flights. Global load balancing is that reservation center for your traffic.

Goal

Low latency

Send each user to a nearby healthy location so that responses travel a short distance.

Goal

High availability

Keep serving even when an entire site fails, by shifting traffic to the others quickly.

Goal

Capacity sharing

Spread load so that no single site is pushed past its limits while others sit idle.

Goal

Control and compliance

Honor rules about where certain users or data may be served from.

i
Good to Know

Global load balancing is not a single product or a single feature. It is a family of techniques, and a real system usually combines several of them. Understanding each technique’s strengths and limits is more useful than memorizing any one tool.

2 Levels
First choose the site, then choose the server
N Clusters
Several regions, each able to serve on its own
1 Front Door
One name that users see, many places behind it

2Why One Cluster Is Not Enough

Running several clusters adds work. Teams do it for five main reasons.

The five reasons

1

Distance and speed

Signals travel through cables at a fixed speed, so a user on the other side of the planet always waits longer than a user next door. Serving from a nearby region cuts that wait, and for interactive products the difference is felt directly.

2

Survival of big failures

A single cluster can fail as a whole: a bad upgrade, a control plane problem, a cloud region outage, or a network cut. Multiple clusters in separate failure domains turn those events from total outages into partial slowdowns.

3

Rules about location

Laws and contracts sometimes require that data about certain users stays within a country or region. Separate clusters per region make it practical to enforce such rules.

4

Size and scaling limits

Every cluster has practical limits on how large it can grow before its management layer strains. Splitting a very large workload across clusters keeps each one at a comfortable size.

5

Smaller blast radius for changes

Rolling a risky change out one cluster at a time means a mistake hurts a fraction of users, not all of them. Several clusters make gradual, safer change possible.

The price of running several clusters

These benefits are real, but so are the costs. More clusters mean more things to upgrade, secure, monitor, and pay for. Applications must now cope with data that lives in several places. Debugging becomes harder because a single user’s journey may cross regions. Before building a global design, ask honestly which of the five reasons truly applies to you. Teams that adopt multiple clusters only because it sounds mature often take on complexity without gaining much.

Advantages

  • Faster responses for distant users
  • Survival of site-level failures
  • Ability to meet data location rules
  • Smaller, safer rollouts of change
  • Room to grow beyond the limits of one cluster

Disadvantages / Trade-offs

  • More infrastructure to operate and pay for
  • Harder data consistency across regions
  • More complex debugging and monitoring
  • Extra traffic charges between regions
  • Greater need for automation and discipline
!
Common Mistake

Assuming that adding a second cluster automatically doubles availability. If both clusters depend on the same database, the same identity service, or the same deployment pipeline, a single failure can still take both down. True independence needs separate failure domains all the way down.

3A Quick Tour of Load Balancing Layers

Before going global, it helps to see where each kind of balancing lives in the stack.

From the user to the pod

A request from a user passes through several layers before reaching your application, and each layer can make its own balancing decision. At the very top, a name lookup decides which address the user will contact. At the edge, a proxy or network device may choose among clusters. Inside the chosen cluster, an entry point called an ingress or gateway chooses a service. Then the cluster’s internal networking chooses one of the pods behind that service. Global load balancing sits at the top layers, while the lower layers are the familiar Kubernetes machinery.

flowchart TD
  U["User"] --> D["Name lookup: which address?"]
  D --> G["Global layer: which cluster?"]
  G --> C1["Cluster A: ingress or gateway"]
  G --> C2["Cluster B: ingress or gateway"]
  C1 --> S1["Service"]
  C2 --> S2["Service"]
  S1 --> P1["Pod"]
  S1 --> P2["Pod"]
  S2 --> P3["Pod"]
  S2 --> P4["Pod"]
        
FIG 1 — Where the global decision sits above the familiar cluster-level decisions

Layer four and layer seven

Balancers are often described by the layer of the network model at which they work. A layer four balancer sees only connections: addresses and ports. It is fast and simple, and it spreads connections without understanding what is inside them. A layer seven balancer understands the application protocol, such as web requests, and can make choices based on the address path, headers, cookies, or the content type. It is more flexible, allowing features like routing by URL, but it does more work per request.

TypeWhat it seesStrengthsLimits
Layer fourConnections: source, destination, and portsVery fast, works with any protocol, simple to reason aboutCannot route by URL or header; cannot inspect or modify requests
Layer sevenFull requests: paths, headers, cookies, and methodsSmart routing, retries, header changes, detailed metricsMore processing per request; must understand each protocol
Name-basedOnly the name lookup, before any connection existsWorks with everything and needs no data path of its ownCoarse control; depends on how long answers are remembered
i
Good to Know

Many global designs use name-based or network-level steering at the top, because it scales well and works with any protocol, and layer seven features inside each cluster, where rich routing is most useful. Choosing the right layer for each decision is a core design skill.

4The Main Ways to Steer Traffic Between Clusters

There are five broad techniques. Each answers the question “which cluster?” in a different way and at a different moment.

Technique 1

Name-based steering

The name lookup returns the address of the best cluster for each user. Simple and universal, but limited by how long answers are remembered.

Technique 2

Anycast networking

The same address is announced from many places, and the internet’s routing sends each user to a nearby one automatically.

Technique 3

Global edge proxies

A worldwide network of proxies terminates user connections nearby and forwards requests to the best cluster over a managed backbone.

Technique 4

Multi-cluster gateways and controllers

Kubernetes-aware controllers that watch several clusters and program the global layer accordingly.

Technique 5

Mesh and client-side balancing

Services call each other across clusters through a service mesh or smart clients that know about locations and health.

TechniqueSpeed of reaction to failureGranularity of controlTypical effortWorks for any protocol?
Name-basedSeconds to minutes, depending on answer lifetime and client behaviorCoarse: chooses an address, per lookupLowYes
AnycastFast, as routes are withdrawn when a site failsCoarse: network-level proximityHigh if self-built; low if rentedYes, with care for long-lived connections
Global edge proxyVery fast, because the proxy checks health continuouslyFine: per request, path, or headerLow to mediumMostly web protocols, some support for others
Multi-cluster gateway or controllerDepends on underlying mechanismFine to medium, Kubernetes-awareMediumDepends on the implementation
Mesh or client-sideFast for service-to-service callsVery fine: per callMedium to highMostly application protocols supported by the mesh
i
Practical Advice

Think of these techniques as layers you can stack. A common recipe uses a global edge or name-based layer to choose a cluster for user traffic, Kubernetes gateways inside each cluster for routing, and a mesh or multi-cluster discovery for traffic between services. The following chapters take each layer in turn.

5Name-Based Global Load Balancing

The oldest and most widely used technique answers the question before the connection even begins.

How it works

When a user types a website name, their device asks a name server to translate it into a numeric address. Name-based global load balancing makes that translation smart. Instead of always returning the same address, the authoritative name server looks at who is asking, which clusters are healthy, and its configured policy, and then returns the address of the best cluster for that user. Because the choice is made at lookup time, the user’s connection goes straight to the chosen cluster with no extra hop.

sequenceDiagram
  participant U as User device
  participant R as Recursive resolver
  participant A as Smart authoritative name server
  participant H as Health checker
  participant C as Chosen cluster
  H->>A: Report health of every cluster
  U->>R: Where is the service?
  R->>A: Where is the service?
  A->>A: Apply policy and health to pick a cluster
  A-->>R: Address of the best cluster, with lifetime
  R-->>U: Address of the best cluster
  U->>C: Connect directly
        
FIG 2 — A smart name server picks the cluster at lookup time

The role of lifetime

Every name answer carries a lifetime, called time to live, which tells resolvers and devices how long they may remember it. A short lifetime means changes take effect quickly, because devices ask again soon. A long lifetime reduces lookup traffic and speeds repeat visits, but it means a changed answer takes longer to reach everyone. When a cluster fails and the name server stops advertising it, users who remember the old answer keep trying the dead address until their remembered answer expires.

Simple Analogy

Imagine you tell a friend the address of the best open pizza shop and say, “remember this for an hour”. If that shop closes five minutes later, your friend will still walk there for another fifty-five minutes, no matter how many times you shout the new address. The remembering time is the lifetime, and it is the main weakness of name-based steering.

Strengths of name-based steering

  • Works with every protocol, not only web traffic
  • No extra hop; users connect directly to a cluster
  • Simple, cheap, and widely understood
  • Easy to combine with geography and weights

Limits of name-based steering

  • Failover speed depends on how long answers are remembered
  • Some resolvers and devices ignore short lifetimes
  • The server sees the resolver’s location, not always the user’s
  • Cannot rebalance individual requests or existing connections
!
Common Mistake

Setting a very short lifetime and assuming failover will therefore be instant. Some resolvers and applications hold answers longer than requested, and some long-lived connections never ask again. Name-based failover is fast for most users but slow for a stubborn tail. Plan for that tail.

6Routing Policies: Deciding Who Goes Where

Once you can steer traffic, you need rules. A handful of policies cover most real needs.

Policy

Round robin and weighted

Rotate answers among clusters, optionally giving bigger clusters a bigger share. Good for spreading load without caring about distance.

Policy

Geographic

Send users to a cluster based on their country or continent. Useful for compliance and predictable behavior.

Policy

Latency-based

Send users to the cluster that has measured the lowest delay for their network. Usually the best choice for speed.

Policy

Failover (primary and secondary)

Send everyone to a primary cluster, and switch to a standby only when the primary is unhealthy.

Policy

Capacity-aware

Reduce traffic to a cluster as it approaches its limit, even if it is healthy and close.

Policy

Cost-aware

Prefer clusters where serving costs less, within limits on speed and fairness.

Geography is not the same as speed

A natural assumption is that the closest cluster on a map is the fastest. Often it is, but not always. Networks are built by companies and governments, and the path between two cities may travel a surprising route. A user in one country may reach a cluster in a neighboring country faster than a closer one in their own country, because of how providers connect their networks. This is why latency-based policies, which rely on measurements instead of maps, frequently beat purely geographic ones for speed. Geographic policies still have their place when the rule is legal, not about speed.

GoalBest-fit policyWatch out for
Fastest response for most usersLatency-basedMeasurements can lag behind changes in the network
Keep users inside a legal regionGeographic, with strict boundariesUsers traveling, using shared resolvers, or on privacy networks may appear to be elsewhere
Use bigger clusters moreWeightedWeights must be updated when capacity changes
Keep a standby cluster readyFailoverA standby that is never used may quietly stop working
Gradually move traffic to a new clusterWeighted, with small stepsResolvers may hold old answers, so changes take time to apply
Protect a cluster near its limitCapacity-awareNeeds fast, reliable load signals from each cluster
i
Practical Advice

Combine policies in layers. For example: use geography to enforce legal boundaries, latency to choose the best cluster inside each boundary, weights to shift traffic gradually, and failover as the last line of defense. A layered policy is easier to explain and safer to change than one giant rule.

7Health Checks: How the Global Layer Knows a Cluster Is Alive

A global balancer is only as smart as its idea of health. A bad health check sends users to broken places or abandons healthy ones.

Shallow and deep checks

A shallow health check only asks whether something answers at all, for example whether the cluster’s entry point accepts a connection. It is cheap and quick, but it may report health while the application behind it is broken. A deep health check exercises the real path: it sends a realistic request through the entry point to an application, which in turn touches the things it needs, such as a database, and reports whether the whole chain works. Deep checks are more truthful, but they are more expensive and can themselves cause trouble.

TypeWhat it provesRisk
ShallowThe entry point is reachableFalse comfort: the application may be failing while the entry point still answers
MediumA designated application endpoint responds correctlyMay miss problems with dependencies behind that endpoint
DeepThe full request path works, including key dependenciesA shared dependency problem can make every cluster look unhealthy at once, triggering a harmful mass failover

The dangerous case of checking shared dependencies

Imagine a deep check that fails whenever a database cannot be reached, and imagine that all clusters share the same database. If the database has a brief hiccup, every cluster reports itself unhealthy at the same moment. The global layer then believes the whole world is down, and it may stop sending users anywhere, or flip back and forth wildly. The result is an outage made worse by the health system itself. Careful designs ensure that health checks reflect each cluster’s own ability to serve, and that the global layer has a sensible behavior when everything appears unhealthy, such as continuing to send traffic rather than giving up entirely.

ANTI-PATTERN-01 Avoid
Problem

A health check fails a cluster whenever any shared dependency has a hiccup, and the global layer removes unhealthy clusters from service without any limit.

Why It’s Harmful

A brief problem in a shared dependency makes every cluster look dead at once, and the global layer withdraws all of them, converting a minor glitch into a total outage.

Correct Approach

Test each cluster’s own ability to serve, set a floor so the global layer never removes every cluster, and use gentle thresholds so a single failed check does not trigger a change.

Tuning the timing

1

Interval

How often the check runs. Frequent checks detect failure sooner but add load and noise.

2

Failure threshold

How many failed checks in a row mark a cluster unhealthy. Requiring several failures avoids reacting to a single glitch.

3

Recovery threshold

How many successes in a row bring a cluster back. Requiring several prevents rapid flapping between healthy and unhealthy.

4

Checker locations

Run checks from several places in the world, so that a local network problem near one checker is not mistaken for a cluster failure.

i
Good to Know

Many teams expose a dedicated health endpoint in each cluster that summarizes readiness, including a signal that the cluster can be gently drained. Draining lets operators remove a cluster from rotation on purpose, for maintenance, without waiting for it to fail.

8Anycast: One Address, Many Places

Anycast lets the network itself choose the nearest location, with no smart lookups required.

The idea

Normally an address belongs to one place. With anycast, the same address is announced from many places at once. The internet’s routing system, which moves traffic hop by hop, naturally delivers each user’s packets to the nearest announcing location in network terms. If one location stops announcing the address, because it has failed or has been removed on purpose, routing quickly forgets it and delivers traffic to the next nearest location. The user keeps the same address throughout.

flowchart LR
  U1["User in Asia"] --> A["Shared address announced from many sites"]
  U2["User in Europe"] --> A
  U3["User in Americas"] --> A
  A --> S1["Edge site: Asia"]
  A --> S2["Edge site: Europe"]
  A --> S3["Edge site: Americas"]
  S1 --> K["Backend clusters"]
  S2 --> K
  S3 --> K
        
FIG 3 — One shared address, with the network delivering each user to a nearby site
Simple Analogy

Think of a national emergency number. Everyone dials the same digits, and the phone network connects each caller to the nearest dispatch center. If one center loses power, calls simply flow to the next one. Callers never learn a new number.

Strengths and cautions

Strengths of anycast

  • Very fast reaction when a site stops announcing the address
  • No dependence on how long name answers are remembered
  • Natural protection against large floods of traffic, which spread across many sites
  • Users keep a single stable address

Cautions with anycast

  • Running your own anycast needs network expertise and your own address space
  • Routing changes can move a user mid-connection, breaking long-lived sessions
  • Proximity is decided by network routing, which may not match the best latency
  • Hard to control precisely which users reach which site
i
Practical Advice

Most organizations do not build anycast themselves. They rent it from a global edge network or a cloud provider’s global load balancer, which presents a small number of stable addresses and handles the routing work. This brings the benefits without the specialist burden.

!
Common Mistake

Using anycast for services with very long-lived connections without checking how route changes affect them. If a user’s route shifts to a different site mid-connection, that site has never heard of the connection and will reset it. Edge proxies that terminate connections close to the user soften this problem.

9Global Edge Proxies and Cloud Global Load Balancers

The most popular modern approach puts a worldwide network of smart proxies in front of your clusters.

How an edge proxy works

A global edge network has points of presence in many cities. Each runs proxies that accept user connections nearby, finish the secure handshake close to the user, and then forward requests to the best backend cluster over the provider’s own private backbone, which is often faster and more reliable than the public internet. Because the proxy sees every request, it can do detailed layer seven work: route by path, retry on another cluster if one fails, cache responses, filter attacks, and measure everything.

1

User connects to a nearby edge

Anycast or smart name answers bring the user to a close point of presence, so the slow, round-trip-heavy connection setup happens over a short distance.

2

Edge inspects and filters

The proxy applies security rules, rate limits, and caching before any backend is bothered.

3

Edge chooses a backend cluster

Using live health, latency, and capacity information, it picks the best cluster for this request.

4

Request travels over a private backbone

The proxy reuses warm, already-established connections to the cluster, avoiding repeated setup costs.

5

Retry on failure

If the chosen cluster errors or times out, the proxy can retry another cluster, often before the user notices anything.

Why this approach is so popular

Benefit

Fast failover

Proxies check health continuously and can move requests away from a failing cluster within seconds, without waiting for anyone’s remembered answers to expire.

Benefit

Fine-grained control

Routing can depend on path, header, cookie, or user group, which allows precise rollouts and experiments.

Benefit

Built-in protection

Attack filtering, certificate handling, and caching come as part of the same layer.

Benefit

Less to operate

The provider runs the worldwide infrastructure, so your team focuses on policy rather than plumbing.

Advantages

  • Quick, request-level reaction to failures
  • Rich routing and security features in one place
  • Shorter connection setup for distant users
  • Often integrates with Kubernetes through standard resources

Disadvantages / Trade-offs

  • Dependence on one provider’s network and features
  • Extra cost per request and per volume of data
  • The edge itself becomes a critical dependency
  • Limited support for some non-web protocols
!
Common Mistake

Treating the edge provider as infallible. Even the largest networks have had major outages. Decide in advance whether your business can tolerate that dependency, and consider whether a secondary path, such as a second provider or a direct route to clusters, is worth its cost and complexity.

10Multi-Cluster Ingress and the Gateway API

Kubernetes has its own vocabulary for describing how outside traffic enters. Extending that vocabulary across clusters is the next step.

Ingress and gateways in one cluster

Inside a single cluster, outside traffic enters through an object that describes the rules: which names and paths go to which services. The older object for this is called an Ingress. A newer, more expressive family of objects called the Gateway API separates roles: infrastructure teams define the gateways, application teams attach routes to them, and policies describe behavior like timeouts and retries. Gateway API is increasingly the preferred way to describe traffic entry in Kubernetes because it is richer, more portable, and designed for multi-team use.

Extending the idea across clusters

Several products and projects let a single description of routes apply to several clusters. A multi-cluster ingress or gateway controller watches a group of clusters, discovers where each service is running, and programs a global balancer with backends in every cluster. When a cluster becomes unhealthy or loses its pods, the controller or the underlying balancer stops using it. Your team describes intent once, and the controller turns it into actions in each place.

flowchart TD
  T["Team describes routes once"] --> CT["Multi-cluster controller"]
  CT --> W1["Watches cluster A"]
  CT --> W2["Watches cluster B"]
  CT --> W3["Watches cluster C"]
  CT --> GB["Programs global balancer"]
  GB --> BA["Backend in cluster A"]
  GB --> BB["Backend in cluster B"]
  GB --> BC["Backend in cluster C"]
        
FIG 4 — A controller turns one description of routes into backends across many clusters

What to look for in such a controller

QuestionWhy it matters
Which clusters and environments does it support?Some work only with one cloud provider or a specific cluster product, which limits portability
How does it discover services across clusters?Discovery quality decides how quickly changes in one cluster reach the global layer
How is health decided?Cluster-level and pod-level health should both count, and checks should be configurable
What routing policies can it express?Weights, locality, header-based rules, and failover rules determine how flexible your operations can be
Where does it run, and what happens if it fails?The controller itself must not be a single point of failure; existing routes should keep working if it is briefly down
How does it handle certificates and security policy?Consistent security across clusters is easier when the controller manages it centrally
i
Good to Know

A good design keeps traffic flowing even if the controller is briefly unavailable. The controller programs the balancer; it should not sit in the data path. Always ask: if this controller disappeared for an hour, what would users experience?

11Kubernetes-Native Global Balancing with DNS Controllers

If name-based steering is simple and universal, why not drive it directly from Kubernetes objects?

Two helpers that bridge Kubernetes and names

One kind of tool watches Kubernetes services and ingress objects and automatically creates and updates the matching name records at a name provider. This removes the manual step of keeping names in sync with a changing cluster, and it is widely used even in single-cluster setups. A second kind goes further. It runs inside each cluster, shares health information with its peers, and acts as a small, cluster-aware name server that answers lookups using the real health and policy of every participating cluster. Community projects of this type exist and are hosted by open foundations; they let teams express global balancing policy as Kubernetes resources, with no dependence on a particular cloud provider.

flowchart LR
  subgraph CA["Cluster A"]
    A1["Service and ingress"] --> A2["Cluster-aware name controller"]
  end
  subgraph CB["Cluster B"]
    B1["Service and ingress"] --> B2["Cluster-aware name controller"]
  end
  A2 |"Share health and addresses"| B2
  U["User lookup"] --> A2
  U --> B2
        
FIG 5 — Cluster-aware name controllers that answer lookups from real cluster health

Where this approach fits

Teams that want to avoid provider lock-in

Because policy lives in Kubernetes resources and the name answers come from your own controllers, the design is portable between clouds and on-premises environments.

Organizations with their own data centers

Where no rented global balancer exists, cluster-aware name controllers offer a practical way to get health-aware steering between sites.

Modest scale and simple needs

For a handful of clusters and a few services, this approach is easy to reason about, and its behavior is fully visible in your own cluster.

!
Keep Perspective

Because this approach is name-based, it inherits the same limits discussed earlier: failover speed depends on answer lifetimes and client behavior, and fine control over individual requests is not possible. It is a good fit for simple, robust steering, and a poorer fit where seconds of failover time or request-level routing are required.

i
Practical Advice

Make sure the name servers that answer for your domain are themselves spread across clusters and networks. If every authoritative name server lives in one cluster, losing that cluster makes your whole service unreachable by name, defeating the purpose of going global.

12Multi-Cluster Service Discovery: Finding Services Across Clusters

User traffic is only half the story. Services inside your clusters also need to find and call each other across cluster boundaries.

The problem of separate worlds

By default, each Kubernetes cluster is its own small world. A service named payments in one cluster is unknown to another cluster, even if the second cluster would love to call it. Each cluster has its own names, its own internal addresses, and its own idea of which services exist. To let clusters work as one system, there must be a way to say: this service exists here, and it may be reached from there.

A standard way to share services

The Kubernetes community defined a standard approach, often called the Multi-Cluster Services model. In this model, a cluster that owns a service marks it as exported. Other clusters in the same group then see a matching imported service, which is a stand-in that can be called by a shared, group-wide name. Behind that stand-in, a controller keeps track of all the places where the service actually runs, and calls to it can reach any of them. The group of clusters that share services this way is treated as a single set with a common namespace for sharing.

flowchart LR
  subgraph A["Cluster A"]
    SA["Service payments"] --> EA["Marked as exported"]
  end
  subgraph B["Cluster B"]
    SB["Service payments"] --> EB["Marked as exported"]
  end
  subgraph C["Cluster C"]
    IC["Imported stand-in for payments"] --> CA["Caller application"]
  end
  EA --> IC
  EB --> IC
        
FIG 6 — Exported services in two clusters appear as one imported service in a third
Concept

Export

The owner of a service declares that it may be reached from other clusters in the group.

Concept

Import

Other clusters receive a stand-in for the service, with a shared name and a set of backends gathered from every exporting cluster.

Concept

Sameness

Services with the same name and namespace in different clusters are treated as the same service. This requires discipline in naming and consistent configuration.

Concept

Group-wide name

A single, stable name that callers use no matter how many clusters host the service.

i
Good to Know

The model describes discovery, not routing policy. It tells callers which backends exist across clusters, but decisions such as preferring the local cluster or failing over to a remote one are usually made by a mesh, a gateway, or the networking layer that implements the model.

!
Common Mistake

Assuming two services with the same name in different clusters are interchangeable when they are not. If one cluster runs an older version or a different configuration, callers may get inconsistent behavior. Treat “same name” as a promise that must be kept through consistent deployment across clusters.

13Service Meshes Across Clusters

A mesh gives services smart, secure, observable communication, and it can extend that across clusters.

What a mesh adds

A service mesh places a small proxy beside each application, or in the node’s network path, and these proxies handle communication between services. They provide encrypted connections with strong identity, retries, timeouts, traffic splitting, and detailed metrics, all without changing application code. When a mesh spans several clusters, those same features work between services in different places.

Locality-aware balancing

The most useful multi-cluster mesh feature is locality-aware balancing. By default, a caller prefers backends that are close to it, first in the same zone, then in the same region, and only then elsewhere. This keeps traffic local and fast and avoids paying for cross-region data transfer. When local backends become unhealthy or overloaded, the mesh gradually spills traffic to the next nearest location. The behavior mirrors the global design, but for traffic between services instead of user traffic.

flowchart TD
  C["Calling service"] --> P{"Healthy backends in the same zone?"}
  P -->|"Yes"| Z["Use same-zone backends"]
  P -->|"No"| R{"Healthy backends in the same region?"}
  R -->|"Yes"| RR["Use same-region backends"]
  R -->|"No"| O["Use backends in another region"]
        
FIG 7 — Locality-aware preference: stay close, spill outward only when needed

Common multi-cluster mesh designs

DesignHow clusters relateStrengthsCautions
Single shared control for all clustersOne management layer governs every clusterConsistent policy and a single viewThe shared layer becomes a critical dependency
Control in each cluster, with shared trustEach cluster runs its own mesh management and they exchange service informationBetter fault isolation, as clusters can operate aloneMore components to keep consistent
Gateway-based connectionClusters connect through dedicated east-west gatewaysWorks when clusters cannot reach each other’s internal addresses directlyExtra hop and gateway capacity to plan
Flat network connectionPods can reach pods in other clusters directlyLowest latency and simplest data pathNeeds non-overlapping address plans and network integration

Failover between clusters

When all backends of a service in one cluster are unhealthy, the mesh routes calls to the same service in another cluster. Outlier detection removes individual misbehaving backends early, while whole-cluster failover covers larger failures.

Shared identity

A common root of trust lets services in different clusters verify each other’s identity and encrypt traffic between them, which is essential for secure calls across the open network.

Consistent observability

Metrics and traces from every cluster flow into the same system, letting you follow a request across clusters and compare behavior by location.

!
Common Mistake

Enabling cross-cluster failover without capacity planning. When one cluster fails, its callers flood the survivors, and if the survivors have no spare capacity, the failover itself triggers a wider outage. Failover rules must be backed by real headroom in the receiving clusters.

14Connecting Clusters: The Network Beneath It All

Everything above depends on clusters being able to talk to one another reliably and securely.

The address planning problem

Every cluster hands out internal addresses to its pods and services. If two clusters use the same ranges, which is the default in many setups, their internal addresses overlap, and a call to an address becomes ambiguous. Direct pod-to-pod communication across clusters therefore requires careful planning of non-overlapping ranges before you build anything. Retrofitting this after clusters are running is painful, so plan address space globally from the very start.

Ways to connect

Option

Private links between networks

Cloud or data center networks are peered or joined by dedicated links, so internal addresses can reach each other directly.

Option

Encrypted tunnels between clusters

Dedicated gateways in each cluster create encrypted tunnels across the public network and route cross-cluster traffic through them.

Option

Network-layer cluster mesh

A cluster networking layer extends its own routing and identity across clusters, so pods and services in different clusters can reach each other.

Option

Application-level gateways

Services are exposed to other clusters only through public or private gateways with their own addresses, with no pod-level connectivity at all.

ApproachLatency and efficiencyIsolation between clustersPlanning needed
Private linksExcellentWeaker, since networks are joinedCareful non-overlapping address plan
Encrypted tunnelsGood, with some overheadGood; only chosen traffic crossesGateway capacity and routing setup
Network-layer cluster meshVery goodModerate; policy controls reachCompatible networking layer in all clusters
Application gatewaysModerate; extra hopsStrongest; clusters stay separateFewer address concerns, more explicit exposure
i
Practical Advice

The more tightly you join clusters at the network level, the more their fates are linked. A loose connection through gateways keeps clusters independent, which is often wiser for failure isolation, even though it costs a little performance. Choose the loosest coupling that still meets your needs.

!
Common Mistake

Treating the link between regions as always fast, free, and reliable. Cross-region links have higher delay, limited capacity, and per-gigabyte charges, and they can fail. Design so that the common path stays inside one region and cross-region calls are the exception.

15Topology Patterns: How to Arrange Your Clusters

Where traffic goes in normal times, and in failure, depends on the pattern you choose.

flowchart LR
  subgraph AP["Active-passive"]
    U1["All users"] --> P1["Primary cluster serving"]
    U1 -.-> S1["Standby cluster ready"]
  end
  subgraph AA["Active-active"]
    U2["Users in region one"] --> A1["Cluster one serving"]
    U3["Users in region two"] --> A2["Cluster two serving"]
    A1  A2
  end
        
FIG 8 — Active-passive keeps a standby; active-active serves from every cluster
PatternHow it worksStrengthsWeaknesses
Active-passive (warm standby)One cluster serves; another is fully built and kept ready, receiving no user trafficSimple data story; clear primary; easy to reason aboutIdle cost; standby may silently rot; failover time includes warm-up and data catch-up
Active-passive (pilot light)Standby holds only the core pieces, such as data and images, and is scaled up on demandLower cost than a full standbySlowest recovery; scale-up under pressure may hit limits
Active-active with regional affinityEvery cluster serves; users normally go to the nearest; data is mostly owned by one region per userFast for users; failover uses already-serving clustersData ownership rules and cross-region writes need design
Active-active with shared global dataEvery cluster serves any user, using a globally replicated data layerMaximum flexibility and capacity sharingHardest data consistency, highest cost, subtle failure modes

Choosing a pattern honestly

The attractive pattern is active-active everywhere, because every cluster earns its keep and failover is smooth. But it is also the hardest to build correctly, because every write may happen anywhere. Many successful systems choose a more modest pattern that matches their needs: stateless frontends active in many regions, with a primary region for writes and replicas elsewhere for reads. This captures most of the latency and availability benefit with a fraction of the complexity.

i
Practical Advice

Ask this question first: what is the cost to the business of one hour of downtime versus the cost of running and operating the more complex pattern? The honest answer often points to a simpler design than the one that sounds most impressive.

ANTI-PATTERN-02 Avoid
Problem

Keeping a standby cluster that never receives traffic and is never tested.

Why It’s Harmful

Over months, the standby drifts out of date: versions differ, certificates expire, capacity is not enough, and data is stale. When the primary fails, the standby fails too, and the recovery plan collapses.

Correct Approach

Send some real traffic to every cluster continuously, or rehearse failover regularly, so that each cluster is proven to work before it is needed.

16Smart Steering: Beyond Simple Proximity

Sending everyone to the nearest cluster is a fine start. Real systems learn to be cleverer.

Signals worth using

Signal

Measured latency

Real delay from user networks to each cluster, collected continuously, not guessed from maps.

Signal

Cluster load

How busy each cluster is right now, such as request rate, saturation, or the share of capacity used.

Signal

Error rates

The share of requests failing in each cluster, which often reveals trouble before health checks do.

Signal

Cost of serving

Differences in price between regions or providers, used to prefer cheaper clusters when speed allows.

Spilling over gracefully

A good steering policy gives each cluster a comfortable share, and when a cluster reaches a threshold, sends the extra to the next best, instead of waiting for failure. This is sometimes called spillover or overflow. It keeps clusters out of overload and spreads a surge gradually. Without it, a cluster runs at full tilt until it falls over, and all its traffic suddenly lands on its neighbors at once.

1

Normal range

Each cluster serves its nearest users. Load stays comfortably below capacity.

2

Pressure threshold

When a cluster passes a chosen fraction of its capacity, a small share of its new requests is sent to the next nearest cluster.

3

Rising spillover

The share sent elsewhere grows as pressure grows, smoothly rather than in one jump.

4

Recovery

As pressure falls, traffic returns to the nearest cluster gradually, avoiding sudden swings.

!
Common Mistake

Reacting too fast to load signals. If steering flips traffic wildly every few seconds based on noisy measurements, load sloshes between clusters like water in a bucket, and every cluster sees unstable, spiky demand. Smooth the signals and change shares in small steps.

17Capacity Planning and the Failover Surge

Failover does not create capacity. It only moves load to where capacity must already exist.

The arithmetic of survival

Suppose you run three equal clusters, each carrying a third of the traffic and each running near its limit. If one cluster fails, its third must be shared by the other two, which then each face one and a half times their normal load. If they were already near their limits, both are now overloaded, and the failure spreads. To survive the loss of any one cluster, each remaining cluster must have headroom for its share of the extra work. The more clusters you have, the smaller the extra share each survivor must absorb, which is one genuine economic benefit of more, smaller clusters.

Number of equal clustersNormal load per clusterLoad per survivor after one failureNeeded headroom per cluster
TwoHalf of total trafficAll of total trafficEach must be able to carry everything alone
ThreeOne third of totalOne half of totalEach must be able to grow by half
FourOne quarter of totalOne third of totalEach must be able to grow by one third
FiveOne fifth of totalOne quarter of totalEach must be able to grow by one quarter

Two hidden threats during failover

The thundering herd

When a cluster fails, many clients simultaneously retry against the next cluster, creating a sudden spike far above normal. Surviving clusters receive not just redirected traffic but a burst of retries. Rate limits, backoff with randomness on clients, and load shedding protect survivors.

Cold caches and cold connections

Clusters receiving new traffic may have empty caches and no warm connections to databases, which makes them slower exactly when they are busiest. Pre-warming, gradual shifts, and sharing some traffic at all times soften the impact.

Autoscaling lag

Autoscalers take minutes to add capacity. During that window, survivors must absorb the surge with what they already have, which is why headroom matters more than reaction speed.

i
Good to Know

Load shedding is a valuable safety valve. When a cluster is overloaded, it deliberately rejects low-priority requests quickly, keeping critical ones working, instead of letting every request slow down together. Decide in advance which traffic is important enough to keep.

18The Hard Part: Data Across Regions

Spreading stateless web servers around the world is easy. Spreading data is where the real difficulty lives.

Why data is different

A stateless server holds nothing that cannot be recreated, so any copy can answer any request. Data is different. A customer’s order, balance, or message exists once in reality, and every copy must agree about it. When copies live far apart, agreeing takes time, because messages must cross oceans, and the speed of light cannot be negotiated. Every global design must decide how to handle this fundamental tension between speed, safety, and agreement.

Simple Analogy

Imagine two bank branches in different countries that both keep a copy of the same customer’s balance. If the customer withdraws money at one branch, the other branch must learn about it before allowing another withdrawal. Waiting for that confirmation makes every withdrawal slow. Skipping the confirmation makes it possible to withdraw the same money twice. There is no free lunch, only a choice of which cost to accept.

Common strategies

StrategyHow it worksStrengthsTrade-offs
Single primary region for writes, replicas elsewhereAll writes go to one region; other regions hold read-only copies updated shortly afterSimple and strongly consistent for writes; fast local readsWrites from distant users are slow; failover requires promoting a replica and may lose the newest writes
Partition data by owner or regionEach user’s data has a home region; requests are served thereFast, predictable, fits data residency rulesUsers who move or share data across regions need special handling
Multi-primary replication with conflict handlingSeveral regions accept writes and reconcile differences afterwardLow-latency writes everywhere; strong availabilityConflicts must be resolved by rules; some data types fit poorly
Globally consistent distributed databaseA database that coordinates writes across regions with strong guaranteesSimplifies application logic; strong consistencyHigher write latency and cost; operational complexity
Eventually consistent stores for suitable dataCopies converge over time without strict coordinationVery available and fast; scales wellApplications must tolerate stale or reordered data

Failover means data decisions too

When a region fails, moving traffic is the easy half. The hard half is the data. If a replica is promoted to take over, any writes that had not yet reached it are at risk of being lost. If the failed region later returns, it may hold writes that others never saw, producing conflicting histories. Teams must decide in advance how much data loss is acceptable, how failover is triggered, who or what decides, and how the returning region is reconciled.

ANTI-PATTERN-03 Avoid
Problem

Building a global traffic layer with beautiful failover, while every cluster still depends on a single database in one region.

Why It’s Harmful

When that region fails, every cluster is left without data. The impressive global layer routes users to clusters that cannot do anything useful.

Correct Approach

Design the data layer and the traffic layer together. Decide which data lives where, how it is replicated, and what each cluster can still do when its data source is distant or unavailable.

!
Split Brain Warning

If a network cut separates two regions and each believes the other has failed, both may start accepting writes as the primary. When the cut heals, their data diverges and cannot be merged cleanly. Reliable designs use agreement among a majority of participants, or a clear external authority, to ensure only one side acts as primary.

19Sessions, Sticky Routing, and Long-Lived Connections

Not every request is independent. Some conversations need to stay with the same cluster.

Why stickiness exists

If every request can be handled by any cluster, balancing is easy. But some applications keep session information in the memory of the server that handled the first request, such as a shopping cart held locally. If the next request lands on a different cluster, that information is missing. Sticky routing solves this by sending a user’s requests to the same place repeatedly, using a cookie, a header, or the user’s address as a hint.

Benefits of stickiness

  • Keeps local session data and caches useful
  • Reduces repeated work, since a user’s data stays warm
  • Makes behavior predictable for stateful applications

Costs of stickiness

  • Uneven load, because some users generate far more work than others
  • Harder failover, since stuck users lose their session when their cluster fails
  • Complicates gradual rollouts and traffic shifts
  • Name-level steering cannot honor stickiness reliably

Better than stickiness: externalize the session

The cleanest solution is usually to keep session data outside the individual servers, in a shared store or in a signed token carried by the client. Then any cluster can serve any request, and stickiness becomes an optimization rather than a necessity. Where a shared store must span regions, remember the data lessons of the previous chapter: replicating sessions across regions is itself a data problem with latency and consistency trade-offs.

Long-lived connections

Streaming and persistent connections

Chat, live updates, and streaming keep one connection open for hours. Such connections do not benefit from per-request balancing, and a failover cuts them. Clients should reconnect automatically, with randomized delays, and servers should be able to drain connections gently before maintenance.

Draining before maintenance

Before removing a cluster from rotation on purpose, stop sending it new connections, let existing ones finish or ask clients to reconnect elsewhere gradually, and only then shut it down.

Reconnection storms

If thousands of clients lose a connection at once and reconnect immediately, they can overload the destination. Randomized backoff on the client spreads the arrivals over time.

i
Practical Advice

Treat any dependence on stickiness as technical debt to be removed over time. Each request that can be served anywhere is a request that survives a cluster failure unnoticed.

20Certificates, Identity, and Trust Across Clusters

Secure connections need identity. With many clusters, managing that identity consistently becomes a project of its own.

Two kinds of trust

There are two places where secure identity matters. At the front, users must be able to trust that they are talking to your real service, which is shown by a public certificate for your site’s name. Between services inside and across clusters, each service must prove who it is to the others, which is shown by internal certificates issued by your own authority. Both must work in every cluster, and both must be renewed before they expire.

Front

Public site certificates

Present the same name in every location. Edge proxies or clusters must all hold valid certificates, renewed automatically.

Inside

Service identity certificates

Short-lived certificates that prove each workload’s identity, usually issued automatically by a mesh or a certificate manager.

Shared

Common root of trust

For services in different clusters to verify each other, they must trust certificates issued by a shared authority or a chain of trust.

Lifecycle

Renewal and rotation

Automatic renewal everywhere, with alerts for anything near expiry, since an expired certificate in one cluster looks like a mysterious regional outage.

Where mistakes hide

1

Renewal that works in one cluster only

Automatic issuance depends on proving control of the name. If proof depends on traffic reaching a particular cluster, renewal can fail elsewhere, especially when traffic is steered away.

2

Different authorities in different clusters

If each cluster has its own internal authority and they do not trust each other, cross-cluster calls fail or must be insecure.

3

Root authority as a single point of failure

If one cluster hosts the only signing authority and it fails, no new certificates can be issued anywhere, and existing ones eventually expire.

4

Clock differences

Certificates are valid for specific times. Machines with wrong clocks may reject valid certificates, a subtle cause of regional failures.

!
Common Mistake

Putting the root of trust for every cluster in a single place with no backup. Protect the root carefully, keep it offline where possible, use intermediate authorities per cluster or region, and practice replacing them so the procedure is known before an emergency.

21Security at the Global Edge

The global front door is the first thing attackers see, and with many entrances there are more places to guard.

What the edge should protect against

Threat

Floods of traffic

Large volumes of junk requests meant to overwhelm a service. Spread across many edge locations, absorbing them becomes far easier.

Threat

Application attacks

Malicious requests aimed at vulnerabilities, such as injection attempts. Edge filters can block known patterns before they reach clusters.

Threat

Abusive clients

Scrapers, credential guessers, and bots. Rate limits and challenges applied at the edge protect every cluster at once.

Threat

Direct attacks on clusters

Attackers who discover a cluster’s real address may bypass the edge entirely. Clusters should accept traffic only from the edge.

Keeping the back door closed

A common weakness is a cluster that can still be reached directly, bypassing all the edge protections. Restrict clusters so that they accept user traffic only from the edge network, through address allow lists, private connectivity, or strong mutual authentication between the edge and the cluster. Verify this regularly, because new exposure can be introduced by accident when someone creates a new public entry point.

Zero trust between clusters

Just because two clusters belong to the same company does not mean they should trust each other completely. Apply the principle of least privilege: allow only the specific services that must call each other across clusters, require strong identity on every call, encrypt everything, and log it. A compromise of one cluster should not hand an attacker the keys to the rest.

LayerControlPurpose
EdgeTraffic filtering, rate limiting, bot protectionStop abuse before it consumes cluster resources
Edge to clusterAllow lists or mutual authenticationEnsure only the edge can reach cluster entry points
Cluster entryAuthentication, authorization, request validationDefense in depth if something slips past the edge
Cluster to clusterMutual identity, encryption, explicit policyLimit the damage if one cluster is compromised
EverywhereLogging and alertingSee and investigate attacks early
!
Common Mistake

Protecting the global edge carefully while leaving each cluster’s own entry point exposed on the public internet. Attackers need only find one unprotected door. Make direct access impossible, not merely undocumented.

22Rolling Out Changes Across Many Clusters

Multiple clusters can make change safer, if you roll out one at a time, and more dangerous, if you roll out everywhere at once.

Consistency without synchronization

Clusters that serve the same product must run the same applications and compatible configuration. Doing this by hand across many clusters guarantees drift. Teams manage it by storing the desired state in version control and letting automation apply it to every cluster, a practice often called GitOps. Dedicated fleet management tools can help place workloads across clusters, apply policy to groups of clusters, and report on drift. The goal is that no cluster is special, and that any cluster can be rebuilt from the same source.

flowchart LR
  A["Change merged to version control"] --> B["Deploy to canary cluster"]
  B --> C{"Healthy after observation period?"}
  C -->|"No"| R["Stop and roll back"]
  C -->|"Yes"| D["Deploy to second region"]
  D --> E{"Healthy?"}
  E -->|"No"| R
  E -->|"Yes"| F["Deploy to remaining regions in waves"]
        
FIG 9 — A wave-based rollout that limits the damage of a bad change

Practices that pay off

Start with a canary cluster

Choose one cluster with a modest share of traffic to receive every change first. It acts as an early warning for the others.

Move in waves with observation periods

Wait, watch error rates and latency, and only then continue. A rollout that is fast but blind spreads mistakes faster than it spreads value.

Separate application and infrastructure changes

Do not upgrade the platform, the mesh, and the application together in one wave. If something breaks, you want to know which change did it.

Shift traffic gradually for risky releases

Use weights at the global layer to move a small share of users to a new version or cluster, and increase it only as confidence grows.

i
Good to Know

Have a fast way to drain a cluster from the global layer as part of your rollback plan. Removing a misbehaving cluster from rotation often restores user experience faster than fixing the cluster itself.

!
Common Mistake

Pushing the same configuration to every cluster at the same moment because the tool makes it easy. The ease of synchronized change is exactly what makes it dangerous. Use the structure of multiple clusters to stagger change deliberately.

23Seeing the Whole World: Observability for Global Systems

You cannot steer what you cannot see. A global system needs views that combine every location.

Measure from where users are

Metrics from inside a cluster tell you how the cluster feels, not how users feel. A cluster may report perfect health while a network problem makes it unreachable from a whole country. Synthetic probes, which are automated test requests sent from many locations around the world, measure the experience as users see it. Real user measurements collected from browsers and apps add the truth about actual devices and networks. Together they reveal problems that cluster-level metrics cannot.

Probe

Global synthetic checks

Test requests from many regions, through the real front door, measuring success and speed from the outside.

Probe

Real user measurements

Timings reported by actual users’ browsers and apps, grouped by country and network.

Metric

Per-cluster golden signals

Request rate, errors, latency, and saturation for each cluster, so differences between clusters stand out.

Metric

Traffic distribution

How much traffic each cluster actually receives, compared with how much your policy intended.

Trace

Cross-cluster traces

A single request’s journey across clusters, with the time spent in each, to find where delay arises.

Log

Steering decisions

Records of when the global layer changed its choices and why, to connect user impact with causes.

Global objectives

Define service level objectives for the whole service, such as the share of requests that succeed within a target time, measured at the edge across all regions. Then break the measurement down by region, to see whether a healthy global average hides one struggling location. A global number is good for reporting; a regional breakdown is good for fixing.

1

Top view: global user experience

Success rate and latency across all users, from synthetic and real-user data.

2

Second view: by region and by cluster

The same measures split by location, with traffic share and capacity use beside them.

3

Third view: the steering layer itself

Health check results, policy decisions, answer lifetimes, and edge errors.

4

Fourth view: dependencies

Data replication lag, cross-region link health, and shared service status.

i
Practical Advice

Make sure your monitoring does not live in only one of the regions it watches. If the region hosting your dashboards and alerting fails, you lose sight at exactly the moment you need it most. Host observability across independent locations.

24What Really Happens During a Failover

Failover is not one moment. It is a sequence of detection, decision, and propagation, and each step takes time.

The timeline of a regional failure

When a cluster stops serving, the first thing that happens is nothing at all: users hit errors, but the global layer does not yet know. Health checks need several failures in a row before they declare a problem. Then the steering layer changes its answers. Then those new answers must reach users, which depends on how the steering works. Finally, clients with long-lived connections or stubborn caches catch up later. The total time from failure to full recovery is the sum of all these stages, and understanding each one lets you shorten the right one.

sequenceDiagram
  participant C as Cluster A
  participant H as Health checks
  participant G as Global layer
  participant U as Users
  C->>C: Failure begins
  U->>C: Requests fail
  H->>C: Check fails
  H->>C: Check fails again
  H->>G: Cluster A marked unhealthy
  G->>G: Stop sending traffic to Cluster A
  G-->>U: New users sent to Cluster B
  U->>U: Remembered answers expire over time
  U->>G: Remaining users move to Cluster B
        
FIG 10 — Failover happens in stages: detection, decision, and propagation
StageWhat determines its lengthHow to shorten it
DetectionCheck interval multiplied by the number of failures requiredCheck more often from more places, with sensible thresholds, and use passive error signals
DecisionHow quickly the global layer reacts once a check failsPrefer layers that act automatically on health, with no human approval for clear cases
PropagationAnswer lifetime for name-based steering; near zero for edge proxies and anycastUse shorter lifetimes, or an edge layer that does not depend on remembered answers
StragglersResolvers that ignore lifetimes, and long-lived connectionsClient reconnect logic, connection draining, and acceptance that a small tail exists
Warm-upCold caches, scaling, and data catch-up in the receiving clusterHeadroom, pre-warming, and continuous partial traffic to every cluster

Two ways to fail: hard and soft

Hard failure

The cluster is simply gone: unreachable or not answering. This is the easy case to detect, since health checks fail clearly and quickly.

Soft failure, or gray failure

The cluster still answers health checks but serves many requests slowly or incorrectly. This is far harder, because checks say yes while users suffer. Passive signals such as error rates and latency from real traffic are needed to catch it.

i
Good to Know

Edge layers that retry failed requests on another cluster can hide hard failures from users almost entirely, because the retry happens within the same user request. This is one of the strongest reasons to prefer request-level proxies for user-facing web traffic.

!
Common Mistake

Assuming that failover works because the configuration says it should. Many teams discover during a real incident that detection is slower than expected, that remembered answers linger, or that the survivor cannot hold the load. Only a rehearsal tells you the real number.

25Proving It Works: Testing Global Failover

A global design is a set of promises. Testing is how you find out which ones are true.

Exercises worth running

Exercise

Drain a cluster on purpose

Remove one cluster from rotation gently and watch where its users go, how long it takes, and whether the survivors cope.

Exercise

Make health checks fail

Cause a cluster’s health endpoint to report failure and confirm the global layer reacts as expected.

Exercise

Cut a cross-region link

Block traffic between regions in a test environment and observe data and service behavior on both sides.

Exercise

Simulate a slow cluster

Add delay or errors to one cluster to test whether passive signals trigger spillover or failover.

Exercise

Promote a data replica

Practice the data side of failover, measuring how long it takes and how much recent data is at risk.

Exercise

Restore a returning region

Bring a failed cluster back and confirm that traffic returns gradually and data reconciles correctly.

What to measure during the exercise

1

Time to detect

From the moment of failure to the moment the global layer knows.

2

Time to shift most traffic

How long until the large majority of users are served elsewhere.

3

Size of the straggler tail

What share of users or connections still hit the failed cluster after several minutes, and for how long.

4

Impact on survivors

How the surviving clusters’ latency, errors, and saturation behaved under the extra load.

5

Data outcomes

Whether any writes were lost or conflicting, and how long reconciliation took.

i
Practical Advice

Write a hypothesis before each exercise, such as “after draining Cluster A, ninety-five percent of traffic moves within two minutes and survivors stay below eighty percent capacity”. Compare the result to the prediction, and treat each difference as a finding to fix. Start in a test environment and move to production gradually, with a clear way to stop.

!
Common Mistake

Testing only the happy path of failure, where a cluster dies cleanly. Real incidents are messier: slow clusters, partial failures, flapping health checks, and failures that happen during a deployment. Include at least one messy scenario in every exercise series.

26The Cost Side: Money Matters in a Global Design

Global designs bring new bills. Knowing where they come from helps you keep them under control.

Where the money goes

Cost sourceWhy it appearsHow to manage it
Idle or spare capacityHeadroom for failover means some capacity is rarely usedUse more, smaller clusters; autoscale; run lower-priority batch work in spare space
Cross-region data transferProviders charge for data leaving a region or zoneKeep calls local with locality-aware balancing; replicate only what is needed; compress and cache
Global edge and balancing servicesCharges per request, per volume, and per ruleCache aggressively at the edge; review rule counts; compare providers
Data replicationStoring and shipping several copies of dataReplicate selectively, tier old data, choose consistency levels appropriately
Duplicate tooling and observabilityEvery cluster needs monitoring, logging, and security toolsCentralize where safe; sample traces; set retention limits
People and complexityMore systems need more skills and on-call effortAutomate, standardize clusters, and avoid unnecessary variety

The hidden cost of chatty services

A service that makes many small calls to another service is fine inside one cluster, where calls are fast and free. If those two services end up in different regions, every call crosses an ocean, adding delay and data charges multiplied by the number of calls. Latency and cost both grow quickly. Keeping tightly coupled services together, and spilling to another region only when needed, protects both speed and budget.

i
Good to Know

Track cross-region traffic as an explicit metric and set an alert for unexpected growth. A surprising bill is often the first sign that a routing change quietly moved a chatty service to the wrong side of an ocean.

Ways to save without losing safety

  • Prefer locality in all balancing decisions
  • Cache static and semi-static content at the edge
  • Right-size spare capacity using measured failure data
  • Run flexible batch work in the headroom

Savings that backfire

  • Removing headroom so failover overloads survivors
  • Dropping a standby cluster without a tested alternative
  • Replicating too little data to recover meaningfully
  • Cutting monitoring in the regions that need it most

27Rules About Location: Compliance and Data Residency

Sometimes the best technical choice is not allowed. Legal and contractual rules shape global designs.

Why location can be regulated

Many countries and industries have rules about where certain kinds of data may be stored or processed. A rule may demand that personal data about citizens stays inside the country, that certain financial records never cross a border, or that some processing happens only in approved facilities. Customers may add contractual restrictions of their own. For a global load balancing design, this means that routing decisions are not only about speed: some users must be served only from certain places, even when another cluster would be closer or healthier.

Rule

Data must stay in a region

Storage and processing for certain users happen only in clusters located in that region.

Rule

Restrictions on movement

Data may not be copied abroad, even for backup or failover, unless specific conditions are met.

Rule

Approved providers and locations

Certain workloads must run only in facilities that meet particular standards.

Rule

Access and audit requirements

Proof of who accessed what, and from where, must be kept and available.

How the design adapts

1

Classify data and users

Identify which users and which data types carry location restrictions, and write the rules down in plain terms.

2

Use boundaries in routing

Apply geographic policies as hard boundaries, so restricted users are never sent outside their allowed set of clusters, even during failover.

3

Plan failover within the boundary

Keep at least two clusters inside each restricted region, so that a failure can be handled without crossing the line.

4

Control data flows, not only requests

Logs, metrics, caches, and backups may also carry restricted information. Include them in the rules and the design.

5

Document and verify

Keep clear records of the design and test that restricted data really stays where it should.

!
Common Mistake

Forgetting that failover can itself violate a rule. If the only standby for a restricted region is in another country, sending users there during an outage may be unlawful. Decide in advance whether the rule or availability wins, and design so that you do not have to choose in the middle of an incident.

i
Keep Perspective

This book of rules is a design input, not legal advice. Involve your legal and compliance colleagues early, since their interpretation of what counts as storage, processing, or transfer will decide what your design may do.

28Choosing an Approach: A Decision Framework

There is no single best design. The right one depends on your traffic, your tolerance for downtime, and your team.

Questions to ask first

1

What kind of traffic do you serve?

Web requests allow rich edge proxies. Other protocols, such as raw connections or streaming, may favor name-based or network-level approaches.

2

How fast must failover be?

If seconds matter, prefer layers that react to health continuously. If a few minutes are acceptable, name-based steering may be enough.

3

Where does your data live?

Your data layer sets the real limits. Choose a traffic design that matches what your data can actually do in each region.

4

How many clouds or sites do you use?

A single provider’s global balancer is simple. Several providers or on-premises sites favor portable, Kubernetes-native or name-based designs.

5

What can your team operate?

A design that needs deep network expertise is a poor fit for a small team. Prefer the simplest approach that meets your real needs.

SituationSuggested starting pointWhy
Web product, single cloud provider, fast failover neededProvider’s global load balancer with multi-cluster gateway supportRequest-level health and retries, low operational burden
Multiple clouds or own data centers, modest failover needsName-based steering with Kubernetes-native controllersPortable, simple, and fully under your control
Non-web protocols or very large connection countsAnycast or network-level load balancing plus name-based fallbackProtocol-independent and resilient
Many internal services calling across clustersMulti-cluster service discovery plus a mesh with locality-aware balancingKeeps traffic local, fails over automatically, secures calls
Strict data residency rulesGeographic policies with regional clusters and within-region redundancyBoundaries are enforced at the routing layer
Small team, early stageActive-passive with a tested standby and simple failoverLowest complexity that still survives a regional loss
“The best global design is the simplest one that survives the failures you actually expect, and that you have actually tested.”
i
Practical Advice

Write a short decision record explaining what you chose, which alternatives you considered, and what would make you revisit the decision. In a year, when growth or a new rule changes the situation, that record will save an argument.

29A Maturity Path: Growing Your Global Setup Step by Step

Nobody should build a perfect worldwide system on the first day. The healthiest teams grow their design in stages, and each stage earns its place.

Real-Life Analogy

A small shop becomes a chain one branch at a time. The owner learns how to stock, staff, and supply a second shop before opening a third. Opening ten shops in a single week, in ten countries, usually ends badly. A global traffic setup grows best the same way.

Stage by stage

Each stage below solves a real problem the previous stage could not. If you do not yet have that problem, you can safely stay where you are.

1

One cluster, several zones

Spread workloads across zones inside a single cluster. This already protects you from the loss of a building or a rack, and it is far simpler than anything global.

2

A tested standby cluster

Add a second cluster in another region, kept ready but receiving little or no traffic. Practice the switch. A standby that has never taken real traffic is a hope, not a plan.

3

Name-based steering with real health checks

Put a smart name service in front, with deep health checks and sensible lifetimes for its answers. Failover becomes automatic and measurable.

4

Active-active for users, with capacity headroom

Serve users from several clusters at once, nearest first, with enough spare room that the survivors can absorb a lost neighbor. Data design must keep up at this stage.

5

Request-level control and service-to-service failover

Add a global edge or mesh that retries, shifts traffic in small steps, and keeps internal calls local. This is where fine control and automatic recovery live.

6

Continuous proof

Failover tests run on a schedule, dashboards show the real picture, and rollouts move in waves. The setup is no longer a project; it is a habit.

i
Practical Advice

Before moving up a stage, write down the specific problem that stage will solve. If you cannot name one, the extra complexity will cost more than it returns.

30More Anti-Patterns to Avoid

Some mistakes appear again and again in global setups. Knowing their shapes helps you spot them in your own plans.

Mistakes that look reasonable at first

The patterns below rarely come from carelessness. They come from sensible decisions made one at a time that, together, produce a fragile whole. A short review against this list, before every major change, is cheap insurance.

ANTI-PATTERN-04 Avoid
Problem

Running two clusters, each at seventy percent load, and calling the pair redundant.

Why It’s Harmful

When one fails, the survivor must carry one hundred forty percent of what it was built for. It overloads, fails too, and the whole service goes down together.

Correct Approach

Plan capacity for the loss of the largest failure you intend to survive. Keep enough headroom, or enough ability to grow quickly, in every cluster that might receive traffic.

ANTI-PATTERN-05 Avoid
Problem

Letting one team control the global layer by hand, with changes made directly in a console and no record of who changed what.

Why It’s Harmful

Mistakes spread worldwide in seconds, and nobody can explain or undo them. Knowledge lives in a few heads.

Correct Approach

Keep the global configuration in version control, review changes like any other, and roll them out in waves. Every change should have an author, a reason, and a way back.

ANTI-PATTERN-06 Avoid
Problem

Trusting automatic failover that has never been triggered on purpose.

Why It’s Harmful

Untested failover often fails in surprising ways: stale settings, missing certificates, a forgotten firewall rule, or a cluster that was never sized for real traffic.

Correct Approach

Practice failover on a schedule, starting with low-risk drills and growing toward realistic ones. Fix every surprise and repeat.

ANTI-PATTERN-07 Avoid
Problem

Making health checks so sensitive that a brief hiccup moves all traffic away from a healthy cluster.

Why It’s Harmful

Traffic flaps back and forth, caches churn, and the cluster that receives the surge may itself struggle. The cure becomes worse than the illness.

Correct Approach

Require several failures before acting, several successes before returning, and limit how fast traffic may shift. Calm, steady reactions beat nervous ones.

!
Watch Out

Many of these mistakes combine. A global layer that is hand-edited, untested, and twitchy is far more dangerous than any single flaw would suggest. Fix the process along with the technology.

31Real-World Scenarios

Principles become clearer when we see them applied. Here are four common situations and how a thoughtful team might handle each.

A shopping site with a sale coming

Traffic will triple for two days. The team adds capacity in advance in every active region, shifts traffic gradually toward the clusters with the most room, and lowers the lifetime of name answers a day earlier so changes take effect quickly. After the sale, they scale down in the same careful order.

A bank with strict location rules

Customer records must stay in their home country. The team runs separate regional clusters, routes users by geography, and gives each country two clusters inside its borders for resilience. Global tooling is limited to routing and monitoring, and carries no customer data across borders.

A game studio with players worldwide

Players need low delay and long-lived connections. The studio uses anycast-style entry points to land players at the nearest edge, pins sessions to a cluster for the length of a match, and drains clusters gently by waiting for matches to end before maintenance.

A company with two clouds and an old data center

No single provider’s global balancer covers everything, so the team chooses portable name-based steering with Kubernetes-native controllers. Each environment reports its health to one place, and the same rules apply everywhere. It is less elegant than a single provider’s tool, but it works across all three.

i
Practical Advice

Notice that each team started from its constraints, such as law, protocol, or provider mix, and not from a favorite tool. Let your constraints choose the design.

32A Worked Example: Losing a Region on a Busy Afternoon

Let us walk through one story from start to finish, so that all the parts of this guide connect in one picture.

Real-Life Analogy

Think of a busy airport when one runway closes. Controllers do not panic. They notice, redirect planes to the other runways, tell crews about longer waits, and reopen the runway only after it has been inspected. A good global traffic setup behaves the same way.

The setup

A company runs a video learning service on three clusters, one in each of three regions. Users are sent to the nearest healthy cluster. Each cluster normally runs at about half its safe capacity, so any two clusters could carry the whole load. A shared database has a primary in one region and replicas in the others, and the application is built to read from the nearest replica.

The incident

On a busy afternoon, a power problem takes down the entire region of the second cluster. Within seconds, users who were sent there start seeing errors. The health checks, which test a real page that touches the database, begin failing. After the required number of failures in a row, the global layer marks the second cluster unhealthy and stops answering with its address. New users are now sent to the nearest remaining clusters. Users whose devices remembered the old answer keep trying the failed cluster until the remembered answer expires, which takes a few minutes because the team kept the lifetime short.

sequenceDiagram
  participant U as Users in the failed region's area
  participant G as Global layer
  participant B as Cluster B (failed)
  participant A as Cluster A (healthy)
  participant T as On-call team
  U->>B: Requests begin to fail
  G->>B: Health checks fail repeatedly
  G->>G: Mark Cluster B unhealthy
  G-->>U: New answers point to Cluster A
  U->>A: Traffic arrives, within spare capacity
  G->>T: Alert about the lost region
  T->>A: Confirm load and error rate are normal
  T->>G: Later, return Cluster B gradually after repair
        
FIG 11 — One regional failure, from the first error to a gradual return

What went well

Because each cluster had spare capacity, the survivors absorbed the extra traffic without slowing down. Because the health check tested the database path, the failure was detected for the right reason. Because the lifetime of name answers was short, most users moved quickly. Because clients retried with randomized delays, there was no wave of simultaneous reconnections.

What needed attention

A small group of users with long-lived video sessions were disconnected and had to restart. The team noted this as expected, and added a gentle reconnect behavior to the player. The database primary was in a healthy region, so no data decision was needed this time; the team used the incident to rehearse the question of what would happen if the primary had been the one lost.

The recovery

When power returned, the team did not flip the cluster back into service at once. They verified that it was healthy, that its data was caught up, and that its certificates and settings were current. Then they sent a small share of traffic, watched the results, and increased the share in steps. Full return took under an hour, with no new errors for users.

i
Practical Advice

The calm outcome was not luck. It came from earlier decisions about headroom, health checks, answer lifetimes, client behavior, and rehearsal. Incidents go smoothly when the work was done before they began.

33A Plain-English Glossary

Keep these short definitions nearby while you plan and review your design.

Term

Global load balancing

Spreading user traffic across several clusters in different places, based on health, distance, and capacity.

Term

Name-based steering

Choosing which cluster a user reaches by giving different answers when the service name is looked up.

Term

Lifetime of an answer

How long a name answer may be remembered before it must be asked for again. Shorter lifetimes allow faster changes.

Term

Anycast

Announcing one address from many places, so each user reaches the nearest one by the normal rules of the network.

Term

Global edge

A worldwide set of entry points that receive user connections near the user and forward them to a cluster.

Term

Health check

A repeated test that decides whether a cluster is fit to receive traffic.

Term

Active-passive

One cluster serves users while another waits as a backup.

Term

Active-active

Several clusters serve users at the same time.

Term

Failover

Moving traffic away from a failed cluster to a healthy one.

Term

Failback

Returning traffic to a cluster after it has recovered.

Term

Headroom

Spare capacity kept free so a cluster can absorb extra traffic during a failure.

Term

Thundering herd

A sudden surge of requests from many clients at once, often after a failover or restart.

Term

Split brain

A situation where two parts of a system each believe they are in charge and accept conflicting changes.

Term

Locality-aware balancing

Preferring nearby destinations for each call, and moving farther away only when nearby ones are unhealthy.

Term

Service export and import

The model in which a cluster announces a service for others to use, and other clusters learn about it and can call it.

Term

Data residency

A rule that certain data must stay within a particular country or region.

Term

Draining

Stopping new traffic to a cluster and letting existing work finish before taking it out of service.

Term

Wave-based rollout

Applying a change to one small group of clusters first, checking the results, then moving to larger groups.

34Frequently Asked Questions

Short answers to questions that teams ask most often when they begin to spread traffic across clusters.

Q1Do we really need more than one cluster?

Not always. If a short outage of a whole region is acceptable and your users are in one area, a single well-built cluster across zones may be enough. Multiple clusters make sense when you need to survive regional loss, serve distant users quickly, meet location rules, or limit the blast radius of mistakes.

Q2Is name-based steering good enough?

For many products, yes. It is simple, portable, and inexpensive. Its main weakness is that remembered answers delay changes, so failover is slower and less precise than request-level control. Use short lifetimes and good health checks to narrow the gap.

Q3When should we choose a global edge or provider load balancer?

When you need fast failover, request-level retries, small gradual shifts of traffic, or strong edge protection, and when your clusters live mostly with one provider. The price is some lock-in to that provider’s way of working.

Q4How is this different from a service mesh?

Global load balancing mainly steers users into your clusters. A mesh mainly steers calls between your own services. Many setups use both: a global layer at the front door and a mesh behind it.

Q5How much spare capacity do we need?

Enough that the remaining clusters can carry the traffic of the largest failure you plan to survive, at your acceptable level of service. With three equal clusters, that often means running each at no more than about two-thirds of its safe capacity, or being able to scale up fast.

Q6What is the biggest hidden risk?

Shared dependencies. If all clusters rely on one database, identity service, or configuration source, a failure there takes everything down no matter how clever the traffic layer is. Map shared dependencies early.

Q7How do we handle user sessions during failover?

Prefer designs where any cluster can serve any user, using shared or replicated session data or signed tokens that carry the needed information. If sessions are tied to one cluster, accept that failover will log some users out, and keep that rare.

Q8How often should we test failover?

Regularly enough that the knowledge stays fresh and changes cannot quietly break it, often monthly or quarterly for drills, with a larger realistic exercise at least yearly. Retest after any major change to routing, networking, or data design.

Q9What should we monitor?

Success rate and response time per cluster and per region as users experience them, health check results, traffic share by cluster, capacity headroom, and the delay between a failure and the full move of traffic. Compare the real numbers with your goals.

Q10Can we use different providers for different clusters?

Yes, and it can improve resilience. It also increases the work of keeping configuration, security, and observability consistent. A portable, name-based or Kubernetes-native approach usually fits best.

Q11How do we avoid flapping between clusters?

Require several consecutive failures before removing a cluster and several successes before returning it. Return traffic in small steps. Limit how quickly shares can change.

Q12What about costs?

Expect extra spend on spare capacity, data transfer between regions, and the global layer itself. Weigh it against the cost of downtime and slow service for distant users. Often the simplest approach that meets your goals is also the cheapest.

Q13Who should own the global layer?

A platform or networking team, with clear review rules and input from application teams. Because mistakes here reach every user, changes should be versioned, reviewed, and rolled out in waves.

Q14Where should a beginner start?

Build a second cluster, make it a tested standby, add deep health checks and name-based steering, and practice a failover. Only then consider active-active designs and finer controls.

35Summary and Key Takeaways

Kubernetes manages one cluster very well, but it leaves the question of many clusters to the layers around it. Global load balancing fills that gap. It decides which cluster each user reaches, notices when a cluster is unwell, and moves traffic before most people notice. The tools range from simple name-based steering to anycast, global edges, multi-cluster gateways, and meshes with locality-aware balancing, and each trades simplicity against precision and speed. Whatever you choose, the lasting lessons are the same: keep health checks honest, plan capacity for the loss you expect, design data and traffic together, shift traffic in small steps, and rehearse failure until it is boring. A global setup earns trust not from its diagram but from the failures it has already survived in practice.

Key Takeaways

  • Start from your needs — resilience to regional loss, speed for distant users, legal boundaries, and blast-radius limits are the reasons to run many clusters; choose the design that serves your real constraints.
  • Pick the right steering layer — name-based steering is simple and portable, anycast and global edges are faster and more precise, and meshes handle calls between services.
  • Make health checks honest — test the real path, require several failures before acting, and watch for shared dependencies that no check on one cluster can reveal.
  • Plan capacity for failure — survivors must be able to absorb the lost cluster’s traffic, and clients must retry politely so recovery does not become a stampede.
  • Design data with traffic — a perfect routing layer cannot rescue clusters that depend on one distant database; decide where data lives and what each cluster can do without it.
  • Change slowly and visibly — keep configuration in version control, roll out in waves, shift traffic in small steps, and measure what users actually experience.
  • Prove it by practicing — rehearse failover on a schedule, fix the surprises, and grow complexity only when a real problem demands it.