Start with the plain meaning, then add detail one layer at a time.

A Kubernetes cluster is a group of machines, called nodes, managed together as one system. Applications run inside the cluster as Pods, and Kubernetes gives them networking, storage and scheduling. Many organisations do not stop at one cluster. They run several: one per region, one per team, one per cloud provider, or one per environment. Once there is more than one, a natural question appears: how do applications in one cluster talk to applications in another?

Multi-cluster networking is the answer to that question. It covers three jobs:

Where Submariner and KubeFed fit

Submariner is an open-source project that connects the networks of different clusters and provides cross-cluster service discovery. KubeFed (Kubernetes Cluster Federation) is a project that spreads Kubernetes objects such as Deployments from one control cluster to many member clusters. Put very simply, Submariner builds the roads between the cities. KubeFed sends the same building plans to all of them. A complete multi-cluster design often needs both kinds of help, but they are not interchangeable. Chapter 5 is devoted to this distinction because it is the most common source of confusion.

One large cluster is simpler to operate, so people usually start there. Over time, strong reasons push them towards several.

What breaks the moment you have two clusters

When everything lived in one cluster, a lot of things were quietly provided for free: every Pod could reach every other Pod, every Service had a stable name, and there was only one place to look for anything. The moment a second cluster appears, each of those comforts has to be recreated on purpose. That is the work of the rest of this guide.

To see what must be rebuilt between clusters, first see what a single cluster provides.

Pod addresses and the flat network

Every Pod receives its own IP address from a range called the Pod CIDR. CIDR (Classless Inter-Domain Routing) is simply a compact way to write a block of addresses, for example 10.1.0.0/16, which means all addresses beginning 10.1. The Kubernetes network model says that any Pod can reach any other Pod directly by its IP, without address translation. People call this a flat network.

The CNI plugin

The piece of software that actually builds this flat network on the nodes is the CNI plugin (Container Network Interface). Well-known examples are Calico, Cilium, Flannel and the plugins offered by cloud providers. They create virtual network interfaces, set up routes and sometimes build overlay tunnels between nodes. A CNI plugin knows only about its own cluster.

Services and kube-proxy

Pods come and go, so their addresses change. A Service gives a group of Pods one stable virtual address, the ClusterIP, taken from a separate range called the Service CIDR. A component called kube-proxy, or an equivalent built into some CNI plugins, programs rules on every node so that traffic sent to the ClusterIP is spread across the healthy Pods behind it. The ClusterIP is not a real machine. It exists only as rules inside the cluster.

DNS

People do not type IP addresses. Each cluster runs a DNS server, usually CoreDNS, which turns a name such as orders.shop.svc.cluster.local into the Service’s ClusterIP. The suffix cluster.local is the cluster’s own domain. Note the important detail: every cluster uses the same default suffix, so the name means “the one in my own cluster”, never “somewhere else”.

Five specific problems appear when a second cluster is added. Each one has a standard remedy.

Which problem does which tool address?

This is the single most important idea in the whole guide.

When people say “multi-cluster”, they mix up two layers that work separately.

flowchart TB
  subgraph CP["Control plane: what to run where, for example KubeFed"]
    HUB["Hub cluster holding the desired state"]
  end
  subgraph DP["Data plane: how Pods and Services talk, for example Submariner"]
    C1["Cluster East"] |"encrypted tunnel"| C2["Cluster West"]
  end
  HUB -->|"propagates Deployments and Services"| C1
  HUB -->|"propagates Deployments and Services"| C2
    

Why mixing them up causes trouble

• If you install KubeFed and expect Pods in different clusters to talk, nothing happens. KubeFed will happily copy a Deployment to three clusters, but it builds no network between them.
• If you install Submariner and expect your Deployments to appear in every cluster, nothing happens either. Submariner connects networks, but it never copies workloads.
• A real multi-cluster platform usually needs both layers, chosen independently. You can have connectivity without federation, and federation without connectivity.

A quick comparison

Submariner is one of several ways to connect clusters. Knowing the others helps you choose well.

How to think about the choice

• If users enter from the internet and clusters never need to call each other, a global load balancer is often enough and the simplest answer.
• If services in different clusters must call each other directly, you need a connectivity layer such as tunnels, a mesh or a routed network.
• If you want strong per-request identity and traffic control, a service mesh across clusters adds value, usually on top of a base connectivity layer.
• If your clusters all use one CNI and are in the same trust zone, the CNI’s own mesh can be the smoothest option.
• If you span clouds and CNIs and need Layer 3 reachability, a CNI-independent tunnel project such as Submariner fits the need.

KubeFed answers one question: how do I define something once and have it appear in many clusters?

The cast of characters

The three parts of every federated resource

flowchart TB
  USER["User or pipeline applies a federated resource"] --> HOST["KubeFed host cluster API"]
  HOST --> CTRL["KubeFed controllers: sync, status, scheduling"]
  CTRL -->|"push template with overrides"| M1["Member cluster East"]
  CTRL -->|"push template with overrides"| M2["Member cluster West"]
  CTRL -->|"push template with overrides"| M3["Member cluster South"]
  M1 -->|"report status"| CTRL
  M2 -->|"report status"| CTRL
  M3 -->|"report status"| CTRL
    

How propagation works

KubeFed uses a push model. A sync controller in the host watches federated resources. For each one it reads the placement, builds the final object for every chosen cluster by applying that cluster’s overrides to the template, and creates or updates the real object in the member. It also watches the member objects. If someone edits or deletes the copy in a member by hand, the controller notices and puts it back. The host is therefore the single source of truth.

Enabling a type

Not every kind is federated automatically. An administrator enables each kind with an object called a FederatedTypeConfig. This creates the matching federated wrapper kind and starts the controllers for it. Namespaces are federated first, because a resource can only be propagated into a namespace that exists in the member.

Smart replica spreading

A feature called ReplicaSchedulingPreference lets you say “run twelve replicas in total, spread across these clusters by weight, with at least two in each”. KubeFed then divides the replicas and adjusts the per-cluster counts. It is a simple form of multi-cluster scheduling, working at the level of replica numbers.

Multi-cluster DNS through the outside world

KubeFed also included objects for publishing the addresses of a service from several clusters into an external DNS system. This provides name-based discovery for traffic arriving from outside, using a separate tool that edits records at a DNS provider. It is not a replacement for in-cluster discovery of pods and services, and it is not a network connection.

The life of a federated application

Status of the project

KubeFed was developed under the Kubernetes multi-cluster special interest group, but it never reached a stable release and the project has since been archived by the community. Its repository is no longer under active development. If you meet it in an existing platform, you should plan a path away from it. If you are starting fresh, choose a currently maintained alternative. Please confirm the current status on the project page before deciding, since open source projects change.

The need that KubeFed addressed is real, and several active projects now meet it.

How to choose

• If your team already lives in Git and wants simple, reviewable delivery to many clusters, a GitOps tool is usually enough.
• If you want a dedicated multi-cluster API with placement policies and failover, a project like Karmada or Open Cluster Management fits.
• If you want a cluster to borrow capacity from another one, look at virtual-node approaches.
• In every case, decide the connectivity layer separately, because none of the above removes the need for it, except where a project explicitly bundles networking.

Now to the connectivity layer. Submariner is made of a handful of small, well-defined parts.

Submariner is an open-source project in the Cloud Native Computing Foundation. It was started at Rancher Labs and grew with strong contributions from Red Hat and others. Its goal is simple to state: let Pods and Services in different clusters communicate directly, securely and with names, regardless of which CNI plugin or cloud each cluster uses.

flowchart LR
  subgraph A["Cluster East, Pod CIDR 10.1.0.0/16"]
    PA["Pod: frontend"] --> RA["Route Agent on every node"]
    RA --> GA["Gateway Engine, active"]
    LA["Lighthouse Agent and DNS plugin"]
  end
  subgraph B["Cluster West, Pod CIDR 10.2.0.0/16"]
    GB["Gateway Engine, active"] --> RB["Route Agent on every node"]
    RB --> PB["Pod: database"]
    LB["Lighthouse Agent and DNS plugin"]
  end
  GA |"IPsec or WireGuard tunnel"| GB
  BR["Broker: shares cluster, endpoint and service information"]
  GA -.-> BR
  GB -.-> BR
  LA -.-> BR
  LB -.-> BR
    

The components

How do two clusters that have never met learn to trust and find each other?

The broker is a noticeboard, not a road

Picture a community noticeboard in a town square. Each cluster pins a card saying “here are my address ranges, here is my gateway’s public address, here are the keys I use”. Other clusters read the cards. The noticeboard carries no traffic. If it burns down, the roads between towns still work, though no new cards can be posted or read until it is rebuilt.

Technically, the broker is a Kubernetes API endpoint with a few custom resource kinds installed, and with a service account and permissions that let each cluster read and write only what it should. It can be a dedicated small cluster, or it can be one of the member clusters. Every member must be able to reach its API.

Joining a cluster

The gateways are the border crossings between clusters. They deserve the most attention in any design.

What a gateway does

A gateway node is an ordinary worker node that an administrator marks with a special label. The Gateway Engine runs there. It reads the broker to learn about other clusters, then builds a secure tunnel to the gateway of each one. A tunnel is a protected channel through which packets travel wrapped inside other packets, so that outsiders on the way see only unreadable traffic between two gateway addresses.

The choice of tunnel technology

Submariner calls the pluggable tunnel implementations cable drivers. Three kinds are available.

High availability with active and standby gateways

A cluster can have several gateway nodes, but only one is active at a time. The others are on standby. They use leader election, the same technique that keeps one scheduler active in a cluster, to choose the active one. If the active gateway fails, a standby wins the election and builds the tunnels. Traffic resumes after a short interruption, normally seconds. Gateways also check the health of the remote end of each tunnel, and report connection status and measured round-trip time.

stateDiagram-v2
  [*] --> Standby : gateway node starts
  Standby --> Active : wins leader election
  Active --> Standby : loses leadership or fails
  Active --> Connected : tunnels to peers are up
  Connected --> Degraded : peer unreachable or health check fails
  Degraded --> Connected : peer reachable again
    

Network address translation and public addresses

In public clouds, a virtual machine often knows only its private address, while the world reaches it through a separate public address. The two gateways must learn each other’s reachable addresses. Submariner discovers and publishes a gateway’s public address through the broker. Tools in the command line client can also prepare cloud accounts by opening required ports and disabling checks that would otherwise drop forwarded packets. Even so, you should confirm the firewall and routing rules in each cloud, because they differ.

Why only one gateway is active

It may seem wasteful to keep a standby machine idle. The reason is simplicity and safety. Two gateways both trying to own the same tunnel to the same remote cluster would fight over keys, routes and connection tracking state. Letting exactly one machine own the tunnel makes behaviour predictable, and a failover is a clean hand-over rather than a negotiation. The cost is that the throughput of one machine is the ceiling, which is why the gateway size matters so much.

What a failover looks like to an application

Imagine a client with a connection open across the tunnel when the active gateway dies. For a few seconds, packets vanish. The client’s operating system retransmits, and if the pause is short, the connection simply resumes once the new gateway is ready. If the pause is longer than the application’s timeout, the application sees an error and must reconnect. A well-built client treats this as normal weather. It retries with a small delay, spreads its retries randomly so that thousands of clients do not all return at the same instant, and gives up politely if the outage is long.

Choosing nodes for the gateway role

• Dedicate them if you can. A node that only does gateway work will not be disturbed by a noisy application Pod.
• Spread them across failure zones. Two gateways in the same rack or the same zone fail together during a zone outage.
• Give them stable public or routable addresses. Changing addresses force the other side to learn the new address before the tunnel returns.
• Keep them patched. They sit at the edge of your network, and encryption software deserves prompt security updates.

The tunnel is only the middle of the trip. Getting packets to and from it is the job of the route agent.

The gateway node is only one machine in the cluster. A Pod on any other node that wants to reach a remote Pod must first be sent to the gateway. The route agent running on every node installs the rules that make this happen. It creates a small overlay between nodes of the same cluster and routes for the remote clusters’ address ranges that point towards the active gateway. When the active gateway changes, the route agents update the rules.

sequenceDiagram
  autonumber
  participant P as Pod in East
  participant N as East worker node
  participant G1 as East gateway node
  participant G2 as West gateway node
  participant N2 as West worker node
  participant Q as Pod in West
  P->>N: Packet addressed to the West Pod
  N->>G1: Route agent rules forward it to the active gateway
  G1->>G2: Encrypt and send through the tunnel
  G2->>N2: Decrypt and route within West
  N2->>Q: Deliver to the destination Pod
  Q-->>P: Reply follows the reverse path
    

Step by step

Reaching a Service, not just a Pod

Submariner routes the Service address range as well, so a packet for a remote ClusterIP can cross the tunnel. At the remote gateway node, the ordinary kube-proxy rules for that cluster translate the ClusterIP into the address of one healthy backing Pod, just as for a local client. The remote cluster’s own load balancing therefore keeps working unchanged, which is elegant and means no new load balancer is needed.

Why the packet does not simply leave from any node

A natural question is why every node does not hold its own tunnel to every remote cluster. With ten clusters and fifty nodes each, that would be thousands of tunnels, each with keys to rotate and health to monitor. Concentrating the work on a few gateways keeps the number of tunnels small and the security surface tight. The price is the extra hop inside the cluster from the ordinary node to the gateway node, which is usually well under a millisecond on a good local network.

What happens when the gateway moves

When the active gateway changes, the route agents on all nodes notice through the shared view of the cluster and rewrite their routes so that remote ranges now point at the new gateway. Existing connections may break, new connections work immediately after the update. This is another reason to keep timeouts short and retries sensible.

How this coexists with your CNI

Your CNI plugin remains in charge of Pod networking inside each cluster. The route agent adds rules next to it. Some network plugins need special handling because they manage routing tables in their own way. Submariner includes helper logic for several popular plugins, but you should always read the compatibility notes for your exact version and test a join in a lab before touching production.

When you cannot change the address plan, Globalnet adds a layer of translation.

The problem it solves

Suppose East and West were both built with the same Pod range. A packet addressed to a Pod in that range is ambiguous. Renumbering a running cluster is often impractical. Globalnet provides a way out by giving the workloads that need cross-cluster access an extra, unique address taken from a separate global range that all clusters agree on. By default this is a large private block, and each cluster receives its own slice of it.

flowchart LR
  P1["Pod A in East, real address in the shared range"] -->|"source rewritten"| E["East global egress address, unique"]
  E --> T["Tunnel between gateways"]
  T --> I["West global ingress address of the exported Service"]
  I -->|"destination rewritten"| S["Service in West, real address in the shared range"]
  S --> P2["Backing Pod in West"]
    

How it works in plain words

• When a Pod in East calls a service in West, its source address is rewritten to a unique global egress address as it leaves East.
• Each exported service in West is given a unique global ingress address. The caller uses that address, found through discovery.
• On arrival, West rewrites the destination from the global ingress address to the real service address inside West.
• Replies are rewritten back the same way.

Trade-offs of using Globalnet

A worked picture

Say both East and West use 10.1.0.0/16 for Pods. The planners reserve a global range, and East is given a slice for its egress and ingress addresses, while West is given a different slice. A frontend Pod in East with real address 10.1.4.9 calls an exported service in West. West publishes the service’s global ingress address, say from its own slice. East’s gateway rewrites the source of the outgoing packet to an East global address. West’s gateway rewrites the destination to the real service address in West. The two clusters never need to understand each other’s real addresses, only the unique global ones.

Choosing between renumbering and Globalnet

Connectivity is useless if applications cannot find each other. Lighthouse supplies the names.

A standard, not a private trick

The Kubernetes multicluster special interest group defined the Multi-Cluster Services API, often written MCS API. It adds two resource kinds that describe sharing services between clusters, and a shared DNS domain. Lighthouse is Submariner’s implementation of this standard, and other vendors, including major cloud providers, implement it too. That means skills and manifests carry over.

sequenceDiagram
  autonumber
  participant App as Pod in East
  participant DNS as CoreDNS with Lighthouse plugin
  participant LH as Lighthouse data from the broker
  App->>DNS: Look up orders.shop.svc.clusterset.local
  DNS->>LH: Find the ServiceImport for orders in namespace shop
  LH-->>DNS: Providers are West and South
  DNS-->>App: Return one service address
  App->>App: Connect across the tunnel
    

How the pieces fit

Which cluster answers?

If the service exists in several clusters, Lighthouse spreads requests across them in turn. It also prefers a copy in the caller’s own cluster when one exists, which avoids paying for cross-cluster traffic when it is not needed. Callers can also name a particular cluster by putting its identifier in the name, which is useful for debugging or for sending traffic to a chosen region. For headless services, individual Pod names can be resolved with the cluster identifier in the name.

Headless services and individual Pods

A normal service gives callers a single address that hides the Pods behind it. A headless service instead lets the caller see each Pod, which stateful systems such as databases and message brokers rely on. The multi-cluster standard covers this case. Each Pod can be addressed using its own name together with the identifier of the cluster that owns it. This allows a database client to find the primary in one cluster and replicas in another, as long as the underlying network allows direct Pod access.

What happens when the same name exists in several clusters

The standard says that services with the same name and namespace across the group are treated as one logical service. This is powerful and risky in equal measure. It is powerful because the same application deployed in three regions becomes one name that works everywhere. It is risky because an accidental service with a common name, created by a different team in a different cluster, silently joins the group. Agree on naming rules, and review exports during change approval.

Consistency rules

• Ports and protocols of same-named services should match, or the merged service becomes confusing.
• Namespaces should be created consistently in all clusters that take part.
• Delete the export when a service is retired, so that stale records do not send callers to a place that no longer answers.

Put all the parts together with a concrete shop application.

The scene

An online shop runs its web frontend in cluster East. Its orders database runs in cluster West, for data residency reasons. The two clusters were planned with unique ranges.

The story, step by step

Day two: what the team does after launch

The first connection is the easy part. Real life is made of the days that follow. A new cluster is added, so someone must update the address registry and join it. A node image is upgraded, so gateway labels must survive the change. A certificate approaches its expiry date, so a rotation must be planned. A service changes its port, so the export and its consumers must be updated together. Each of these deserves a short written procedure and, ideally, automation that is tested in a staging fleet.

Questions a reviewer should ask

• Which services cross clusters, and what is the business reason for each?
• What happens to each of them if the link is down for ten minutes?
• Who owns the broker, the gateways and the address registry?
• How do we find out within minutes if a tunnel breaks?
• How would we remove a cluster from the group cleanly?

Most of the work of a successful rollout happens before the first command is typed.

A simple method for planning address ranges

Start by choosing one large private block that is not used anywhere in your offices, data centres or partner networks. Divide it in two halves. One half is for Pods, the other half for Services. Then divide each half by the maximum number of clusters you will ever plan to have, with generous headroom. For example, a company expecting up to 64 clusters might give each cluster a Pod range able to hold tens of thousands of Pods, and a smaller Service range, since a cluster usually has far fewer services than Pods. Record the plan in a table that every platform engineer can see, and let a person or a pipeline hand out ranges so two teams never receive the same one.

Which ports must be open

The exact numbers depend on the driver and version, so read the documentation for your release. In general you need a pair of user datagram protocol ports between gateways for the tunnel, a port for the tunnel’s health checks, and ports for the internal overlay that connects ordinary nodes to the gateway. Allow them in cloud security groups, host firewalls and any hardware firewall between your sites. Use the narrowest rule that works, such as only the gateway addresses of known clusters.

Connecting clusters is powerful, but it changes the failure behaviour of your whole platform.

Deciding whether you need multiple clusters at all

Before accepting these costs, ask whether a single cluster, possibly with stronger isolation through namespaces, quotas and network policies, would meet the goal. Many teams adopt multiple clusters for good reasons: limiting the blast radius of an upgrade, keeping regulated data in a region, serving users close to where they live, separating production from experiments, or surviving the loss of a whole region. Others adopt them by habit, and inherit the cost of coordination without any of the benefit. A useful test is to write down the specific failure or requirement that a second cluster solves. If the sentence is vague, wait.

The hidden people cost

Every extra cluster needs upgrades, security patches, capacity planning, access reviews and on-call attention. Joining clusters adds shared failure stories, such as a broken route affecting everyone. The team size and skills needed grow with the fleet. Budget for training, runbooks and rehearsals alongside the technology itself, because people are usually the scarcest resource in any platform effort.

Physics sets the floor. Configuration decides how far above it you land.

Latency

Light in fibre travels roughly 200 kilometres per millisecond. Two regions 4,000 kilometres apart have a floor of about 20 milliseconds each way, about 40 for a round trip, before any processing. The tunnel adds a little more through encryption and extra hops. A chatty application that makes thirty sequential calls across such a link will spend over a second just waiting. The most important performance decision is therefore which services talk across clusters at all.

Throughput

All traffic between a pair of clusters passes through one active gateway at each end. Its network card, its CPU, and its encryption speed set the ceiling. Lighter tunnels such as WireGuard commonly give better throughput than IPsec for the same hardware, though results depend on the kernel and the instance type. Always measure on your own machines, with the real packet sizes your applications use.

MTU, the silent troublemaker

The maximum transmission unit, or MTU, is the largest packet a link carries. A tunnel adds its own header, so inner packets must be smaller than normal. If sizes are not adjusted, small requests succeed while large responses stall, a pattern that looks like a random application bug. Check this early when a connection works for pings but hangs on bigger transfers.

Ways to reduce cost and delay

• Cache. Store frequently read answers close to the caller, with an expiry that the business can accept.
• Batch. One request that asks for fifty items beats fifty requests that ask for one.
• Compress. Large text payloads shrink dramatically, saving both time and transfer charges.
• Prefer local. Let the discovery layer favour a healthy local copy and use remote ones only as a fallback.
• Replicate data, not calls. For read-heavy data, a replica in each region removes most cross-region traffic.
• Pick regions wisely. Two regions in the same geography usually cost less and respond faster than two on different continents.

How to measure properly

Test with traffic that looks like the real thing. A single large file transfer shows the best-case bandwidth. A swarm of small requests shows the cost of per-packet processing. Measure both, measure during busy hours, and record the results so that you notice if a future upgrade changes them. Measure the gateway’s CPU during the test. If it is near its limit while the network card is not, encryption is the bottleneck and a faster driver or a larger machine will help.

Everything breaks. A good design says in advance what happens when it does.

Design principles for failure

• Fail local first. Every cluster must be able to serve its own users when all links are cut.
• Use timeouts and retries. A call across clusters needs a short timeout so that a dead link does not hold a thread for minutes.
• Avoid a hidden single point. Count the broker, the gateway pair and the DNS agent as dependencies when you draw the architecture.
• Practise. Shut down a gateway during a quiet hour and watch what happens. Surprises are better found by you than by customers.

Partial failures are the hardest

A cleanly dead link is easy to understand. Harder is a link that is half alive: packets are dropped at random, or one direction works while the other does not, or only big packets fail. Health checks may say the tunnel is up while real traffic suffers. For this reason, measure the success rate and latency of actual cross-cluster calls inside your applications as well as the health of the tunnel. The application’s own view is the final judge.

Avoiding a stampede on recovery

When a link returns after an outage, every client retries at once, and the freshly recovered service is flooded. Use exponential backoff with random jitter, limit concurrent retries, and consider a circuit breaker which stops calling a failing dependency for a short time. These patterns turn a dangerous surge into a gentle ramp.

Graceful removal of a cluster

To retire a cluster safely, first remove its exports so that other clusters stop sending it traffic. Wait until traffic has drained. Then disconnect the cluster from the group and finally delete it. Doing it in this order avoids sending customers to a place that is going away.

When clusters are joined, a weakness in one can become a weakness in all.

A least-privilege walk through

1. List every component that writes into a cluster from outside it: the federation controller, the broker clients, the gateway engines.
2. For each, give it a separate identity and only the permissions it truly needs, such as write access to a single namespace instead of the whole cluster.
3. Store its credentials in a secret manager, rotate them regularly, and alert on use from unexpected places.
4. Limit network access to the broker and the API servers to the addresses of known clusters.
5. Log every change to exports, placements and cluster membership, and keep the logs outside the clusters themselves.

Compliance considerations

Auditors ask where data lives and who can reach it. Multi-cluster networking can make these questions harder, because traffic flows that used to stop at a cluster boundary now continue. Keep an up-to-date map of exported services and their consumers, document the encryption used between clusters, and show that regulated data is never exposed to a cluster outside the approved region. A clear diagram and an exports list are often the most persuasive evidence you can give.

A calm, ordered checklist beats guessing every time.

What to watch

• Tunnel status per remote cluster, and its round-trip time.
• Gateway role changes. Frequent flips signal an unstable node or network.
• Bytes sent and received through each tunnel, to find capacity limits and costs.
• Lighthouse DNS query rates and failures.
• For federation: the sync status of each federated resource in each member, and the age of the last successful sync.
• Heartbeat or readiness of each member cluster from the host’s point of view.

flowchart TD
  A["Cross-cluster call fails"] --> B{"Does the name resolve?"}
  B -->|"No"| C["Check ServiceExport and ServiceImport, then the DNS agent and CoreDNS forwarding"]
  B -->|"Yes"| D{"Is the tunnel connected?"}
  D -->|"No"| E["Check gateway pods, firewall ports, public addresses, keys and time"]
  D -->|"Yes"| F{"Do small packets work but large ones hang?"}
  F -->|"Yes"| G["Check MTU and tunnel overhead"]
  F -->|"No"| H{"Does the remote service have healthy endpoints?"}
  H -->|"No"| I["Fix the workload in the remote cluster"]
  H -->|"Yes"| J["Check network policies, address overlap and Globalnet rules"]
    

The order of questions

Signals worth alerting on

Habits that save hours

Keep a small debug Pod image with network tools ready in every cluster. Save the verification command output after a healthy rollout, so that you can compare it when something breaks. Document known quirks of your cloud providers. And always write the incident notes: a team that records what it learned from each outage gets steadily calmer.

Learn from the mistakes other teams already made.

Governance: who decides what is shared

Technology makes sharing easy, so rules must make it deliberate. A good practice is a lightweight review for each new export. The owner explains who will call the service, what data it carries, how it behaves under load, and what the callers should do when it is unavailable. Approved exports are recorded in a catalogue. This is not bureaucracy for its own sake. It prevents the quiet growth of a tangled web of dependencies that nobody understands.

Signs that the design is drifting

• Nobody can draw the list of cross-cluster dependencies from memory.
• An outage in one cluster surprises owners of services in another.
• Data transfer charges grow faster than traffic from users.
• New clusters are created by copying old ones without checking address ranges.

A short, ordered list you can print and follow.

Upgrades and version management

Network components in a fleet must keep rough agreement with one another. Read the release notes for supported version combinations between the broker, the gateways, the route agents and the discovery components. Upgrade one cluster at a time. Start with a non-production cluster, run the verification tests, watch graphs for a day, and only then repeat in production. Keep the previous version’s installation instructions handy so you can roll back. Upgrades of Kubernetes itself, of the network plugin and of the operating system on gateway nodes should be treated as events that can break the connection, and verified afterwards.

Documentation that pays for itself

• A one-page diagram of clusters, gateways, brokers and links.
• The address registry, with owners.
• The list of exported services and their callers.
• A runbook for each failure in the table of failure modes.
• The contact details of every team that depends on the shared services.

Capacity thinking turns vague worry into a clear decision.

The situation

A company runs an orders service in West and a recommendations service in East. East calls West for each product page view. At peak there are 2,000 page views per second. Each call sends 2 kilobytes and receives 18 kilobytes. The round-trip time between the regions is 35 milliseconds.

The arithmetic

What the numbers tell us

• 320 megabits per second at peak is fine for a gateway on a modern instance, but encryption CPU must be watched. If the gateway nodes are small, they will saturate first.
• 35 milliseconds per page is acceptable if there is one call. If the page makes five sequential calls, 175 milliseconds is noticeable.
• Thirty terabytes a month of cross-region transfer may be a significant bill. Ask finance for the provider’s rate before approving the design.

The improvement

The team caches popular product data in East for 60 seconds, removing about 90 percent of the calls. Traffic falls to about 4 MB per second, around 32 megabits, and the average page view stops waiting on the link. The monthly transfer drops to a few terabytes. The network design is unchanged. The application simply stopped crossing the border when it did not have to.

Checking the gateway capacity

Suppose the gateway machine can encrypt about 1.5 gigabits per second before its processors saturate. At 320 megabits per second, the original design uses roughly one fifth of that. That leaves comfortable room for growth, perhaps four times more traffic, before action is needed. If the company expects the page views to triple next year, the same machine still has room. The team writes the number 1.5 gigabits into the capacity plan, with a note to retest after each major upgrade, and sets an alert at 60 percent of that figure so there is time to act.

Where teams actually use these ideas, described in general terms.

Three short stories

No. They solve different problems. Submariner connects networks and shares service names. KubeFed-style tools deploy configuration to many clusters. You can use either alone, or pair the network layer with a different deployment tool.

Check its current status first. The project did not progress beyond beta, and community activity has been reported as archived or reduced. Many teams now choose newer multi-cluster managers or GitOps tools for the deployment side.

Yes, with Globalnet, within its limits. Unique ranges remain the simplest and preferred choice.

No. Your CNI continues to handle networking inside each cluster. Submariner adds routes and tunnels between clusters, working alongside it.

It is when you use an encrypted driver such as IPsec or WireGuard. The plain VXLAN option is not encrypted and should be used only over links that are already protected.

Existing tunnels and previously learned information keep working. New changes are not shared until the broker returns.

It depends on version and design, since every gateway keeps tunnels to the others. Check the project’s documented limits and test your own scale before committing.

Usually only the hostname it calls. Applications should also handle slower calls and failures sensibly, but no new protocols are required.

A ClusterIP service is reachable inside one cluster under cluster.local. An exported service is also reachable from other clusters in the group under clusterset.local.

Yes, several meshes offer multi-cluster modes with their own discovery and identity. Choose by how much of the mesh you want to adopt. A simple network layer suffices if you only need connectivity and names.

Usually an MTU mismatch. The tunnel’s overhead makes large packets too big. Lower the packet size or adjust settings so they fit.

It gives you the connection that disaster recovery needs. You still have to replicate data, run copies of the services, and test failover.