Sharing one cluster is cheap but cramped, and giving everyone their own is roomy but expensive.

Kubernetes was designed to run many applications on a shared pool of machines. In theory, one big cluster should be enough for an entire company. In practice, organisations end up with dozens or even hundreds of clusters, and the reasons are surprisingly consistent.

The hidden bill of cluster sprawl

Every real cluster carries a fixed cost. It needs a control plane, which means several machines or a paid management fee. It needs monitoring, logging, ingress, certificates, security tools and upgrades. Someone must patch it, watch it and be on call for it. When there are fifty clusters, those fixed costs are paid fifty times, and most of the machines inside them sit mostly idle, because each cluster must be sized for its own peak demand.

Namespaces are the first answer to sharing a cluster, and they stop short of what many teams need.

Kubernetes already has a way to divide a cluster: the namespace. A namespace is a named area inside a cluster. Objects such as Pods, Services and ConfigMaps live inside a namespace, and administrators can attach rules to it, such as quotas that limit total resource use, role-based access rules that say who may do what, and network policies that limit which Pods may talk to each other.

For many situations, namespaces are exactly right. A team gets a namespace, works freely inside it, and cannot easily disturb the neighbours. But namespaces have a built-in boundary. They only divide the objects that belong to a namespace. Many important things in Kubernetes belong to the whole cluster and cannot be divided.

Where teams feel the pain

A platform team that wants to test a new version of an operator must install its custom definitions on the shared cluster, where they may clash with the production version. A developer who wants cluster administrator rights to experiment cannot be given them. A team that needs several namespaces to model a multi-part application has to ask an administrator each time. These are not rare edge cases. They are the daily friction that makes teams ask for their own clusters.

The other extreme gives the best separation and the highest cost.

The opposite approach is to give each team or customer a completely separate cluster. This provides the strongest separation, because nothing is shared. A team can be administrator of its own cluster, install anything, run any version and break anything without affecting others.

The costs, however, are heavy and multiply with each new cluster. Creating one takes minutes at best and much longer if approvals and networking changes are involved. Each cluster needs its own control plane, its own add-ons and its own upgrades. Idle capacity is wasted separately in each cluster. And people have to remember to delete clusters nobody uses anymore, which they often forget, so abandoned clusters keep costing money.

The gap in between

Namespaces are cheap and fast but too limited for teams that need real freedom. Real clusters give freedom but are too slow and costly to hand out to every developer and every test run. There is a clear gap between them, and that gap is exactly where virtual clusters sit. They aim to deliver most of the freedom of a real cluster at something close to the speed and cost of a namespace.

A Kubernetes cluster that lives inside a namespace of another Kubernetes cluster.

A virtual cluster is a fully working Kubernetes control plane that runs as ordinary workloads inside a namespace of an existing cluster. The existing cluster is called the host cluster. The virtual cluster has its own API server, so a user can connect to it with the usual tools and see what looks like a normal cluster. They can create namespaces, install custom resource definitions, define cluster roles and even run as the administrator of their own cluster.

The trick is that the virtual cluster does not bring its own machines. When a user creates a Pod inside the virtual cluster, a helper copies a version of that Pod into the host namespace, where the host cluster’s scheduler places it on one of the real machines and the host’s container runtime starts it. The user sees the Pod running in their virtual cluster, and the host sees an ordinary Pod in an ordinary namespace.

flowchart TD
    subgraph HOST["Host cluster with shared machines"]
        subgraph NSA["Namespace for virtual cluster A"]
            A1["Virtual cluster A control plane"]
            A2["Workload Pods of cluster A"]
        end
        subgraph NSB["Namespace for virtual cluster B"]
            B1["Virtual cluster B control plane"]
            B2["Workload Pods of cluster B"]
        end
        N["Real nodes, scheduler, network and storage"]
    end
    UA["Team A uses its own API"] --> A1
    UB["Team B uses its own API"] --> B1
    A2 --> N
    B2 --> N
        

The open source project

The best known implementation is an open source project called vcluster. It packages a virtual control plane, a component that copies resources to the host, and tooling to create, connect to and manage virtual clusters. It has been certified as a conformant Kubernetes distribution by the Cloud Native Computing Foundation, which means that applications written for standard Kubernetes should behave the same inside it. The project was created by the company Loft Labs and is developed in the open.

A small group of Pods that together act as a complete cluster brain.

When a virtual cluster is created, the host namespace receives a small set of workloads that together form the virtual cluster’s brain. The details vary between versions and configurations, but the main pieces are always the same.

flowchart TD
    U["Tenant using kubectl"] --> API["Virtual API server"]
    API --> DS["Data store holding virtual objects"]
    API --> CM["Virtual controller manager"]
    CM --> API
    API --> SY["Syncer"]
    SY --> HAPI["Host cluster API server"]
    HAPI --> HS["Host scheduler and kubelets"]
    HS --> HP["Real Pods in the host namespace"]
    DNS["Virtual cluster DNS"] --> API
        

It is just a few Pods

The control plane of a virtual cluster is ordinary software running in ordinary Pods, usually as a small stateful workload. It needs modest resources, typically a fraction of a processor core and a few hundred megabytes of memory for a small cluster, though the amount grows with the number of objects and activity. That is why dozens of virtual clusters can share a host cluster that could never afford dozens of real control planes.

What kind of Kubernetes runs inside, and where it keeps its memory.

Distributions

The virtual API server and controllers do not have to be the same as the host’s. The project supports several Kubernetes distributions for the virtual control plane, including lightweight ones such as K3s and K0s, and upstream Kubernetes. A lightweight distribution uses less memory and starts faster. Upstream Kubernetes is closest to what runs in many production clusters and may suit teams that need exact behaviour. The default has changed between versions of the project, so check the current documentation.

Data stores

The data store holds every object of the virtual cluster. Options range from a simple embedded database to an embedded copy of etcd, or an external database for stronger durability and scale. A simple embedded store is quick to set up and light on resources, and fits throwaway environments. An embedded or external etcd, or an external database, suits clusters that must survive failures or hold more data.

Version independence

Because the virtual control plane is its own software, it can run a different Kubernetes version from the host, within the range that the project supports. This lets a platform team test an application against a newer or older version of Kubernetes without building a new real cluster. The workloads still run on the host’s nodes, so the node-level behaviour follows the host, even if the API behaviour follows the virtual control plane.

The syncer is the clever bridge that makes shared machines possible.

If the virtual cluster has no machines, something must connect it to the real ones. That something is the syncer. Its job is simple to describe. It watches the virtual cluster for objects that need real resources, mainly Pods, and creates matching objects in the host namespace. It also watches the host objects and copies status information back, so that a Pod shown as running in the virtual cluster really is running on the host.

Not everything is copied. The idea is to copy only what the host must see to run workloads, and to keep everything else purely virtual. Objects that exist only inside the virtual cluster cost the host almost nothing, because they are just rows in the virtual data store.

sequenceDiagram
    participant T as Tenant
    participant V as Virtual API server
    participant S as Syncer
    participant H as Host API server
    T->>V: Create a Deployment
    V->>V: Virtual controllers create Pods
    S->>V: Notice a new Pod
    S->>H: Create a matching Pod in the host namespace
    H->>H: Host schedules and starts the Pod
    S->>H: Watch the host Pod status
    S->>V: Copy status back to the virtual Pod
    T->>V: See the Pod running
        

Why this keeps costs low

A Deployment with fifty replicas becomes fifty Pods on the host, but the Deployment itself, its ReplicaSets, the namespaces, the roles and any custom resources add nothing to the host’s own database. A tenant who installs a large operator with dozens of custom definitions does not burden the host API server at all, which matters when many virtual clusters share one host.

Following one Pod from creation to running, and seeing how its two lives connect.

Why name rewriting matters

Different virtual namespaces may contain Pods with identical names, yet all of them land in a single host namespace, where names must be unique. The syncer solves this by encoding the original namespace and virtual cluster name into each host Pod name. Administrators looking at the host see long, composite names, and a little practice is needed to read them. The same applies to other copied objects such as Services and ConfigMaps.

How Pods in a virtual cluster find each other and the outside world.

Networking in a virtual cluster is deliberately simple. Pods run on the host, so they use the host’s network plugin and receive addresses from the host’s Pod ranges. They can talk to each other as usual. What changes is how names and Services are presented to the tenant.

flowchart LR
    P1["Pod in virtual namespace shop"] --> D["Virtual cluster DNS"]
    D --> VS["Virtual Service address"]
    VS --> HS["Synced Service in host namespace"]
    HS --> P2["Backend Pod on a host node"]
    I["External client"] --> IC["Ingress controller on the host"]
    IC --> HS
        

Isolation of traffic

By default, Pods from different virtual clusters live in different host namespaces and the host’s normal rules apply. If the host network plugin allows all traffic by default, Pods from different virtual clusters may be able to reach each other. To prevent this, administrators apply network policies to the host namespace, and some setups create them automatically as part of an isolation mode. Because enforcement is done by the host network plugin, a plugin with good policy support is important.

Persistent data uses the host’s storage, presented through the tenant’s own claims.

When a tenant creates a persistent volume claim in a virtual cluster, the syncer creates a matching claim in the host namespace. The host’s storage provisioner then creates a real disk using the host’s storage classes. The tenant sees a bound claim and a volume in their cluster, and Pods mount it as usual.

What to keep in mind

The tenant can only use the storage classes that the host offers, and the host’s quotas on storage still apply. Features such as volume expansion and snapshots depend on the host’s storage driver and on whether the syncer passes the needed information through. The data is physically stored on the host, which has two consequences. Deleting the virtual cluster may delete the claims and, depending on the reclaim rules, the underlying disks. And host administrators can see the volumes in their storage system, which matters for privacy.

Who decides where Pods run, and which machines they are allowed to use.

By default, a virtual cluster uses the machines of the host cluster. The tenant sees node objects in their virtual cluster, but these may be stand-ins that represent the host nodes in a simplified form, so that tools expecting to see nodes keep working. The host scheduler decides where Pods land, following the usual rules.

Custom schedulers

Some virtual clusters run their own scheduler to apply custom placement rules inside the virtual cluster. This is useful when tenants need to test scheduler behaviour. In the common setup, though, the host scheduler does the work, which is simpler and ensures that the host sees consistent usage across all tenants.

What a virtual cluster protects, what it does not, and how to strengthen it.

The most important thing to understand about virtual clusters is where the boundary lies. They isolate the control plane strongly and the data plane only as much as the host and the settings allow.

flowchart LR
    A["Shared namespace in one cluster"] --> B["Virtual cluster with shared nodes"]
    B --> C["Virtual cluster with hardened host settings"]
    C --> D["Virtual cluster with dedicated or private nodes"]
    D --> E["Separate real cluster"]
        

Sharing machines only works if nobody can take more than their share.

Because all virtual clusters share the same machines, one busy tenant could take most of the processor and memory unless limits exist. The tool for this is the same one used for ordinary namespaces. Since every Pod of a virtual cluster lands in one host namespace, a quota on that namespace caps the total resources the entire virtual cluster can use, no matter how many namespaces the tenant creates inside it.

The quota is the real budget

Tenants can create as many namespaces and as many objects as they like inside the virtual cluster, but the host quota is the real budget. When the tenant reaches it, new Pods fail to be created on the host, and the failure appears in the virtual cluster as Pods that cannot start. This can confuse tenants who see plenty of free capacity in their own view. Good documentation and clear error messages help.

Two sets of doors: one into the virtual cluster, and one into the host.

A user of a virtual cluster receives a kubeconfig, the connection file that tells tools where the virtual API server is and how to authenticate. Access can go through a temporary tunnel set up by the project’s command line tool, through a load balancer or ingress that exposes the virtual API server, or through a management platform that handles identity and sign-in centrally.

The everyday life cycle, from request to retirement.

Creating a virtual cluster is quick, often taking seconds to a minute or two depending on image availability and the chosen configuration. It is typically done in one of three ways: with the project’s command line tool, by installing its packaged chart on the host with a package manager, or through a GitOps controller that applies a configuration file stored in a repository. A single configuration file describes the choices, such as the distribution, the data store, which resources to sync, resource limits and isolation settings.

flowchart TD
    A["Request a virtual cluster"] --> B["Create a host namespace with quotas and policies"]
    B --> C["Install the virtual control plane in that namespace"]
    C --> D["Wait until the virtual API server is ready"]
    D --> E["Issue credentials to the tenant"]
    E --> F["Tenant deploys workloads"]
    F --> G{"Still needed?"}
    G -->|Yes| H["Operate, monitor and upgrade"]
    G -->|No| I["Delete the virtual cluster and its host namespace"]
    H --> G
        

Configuration as code

Because a virtual cluster is described by a configuration file and a few host objects, it fits naturally into version control. Teams keep these definitions in a repository and let a GitOps tool create, update and remove virtual clusters, with the same review process as any other infrastructure change. This also makes it easy to create identical copies for different teams.

The cheapest virtual cluster is the one that is switched off when nobody is using it.

Because virtual clusters are quick to create, it becomes tempting to create many, and because they are easy to forget, many accumulate. Several habits and features keep costs under control.

Virtual clusters are light, but the ones that matter still deserve protection.

For throwaway environments, the best recovery plan is simply to delete the cluster and create a new one from its definition. For longer-lived virtual clusters that hold real work, there are two kinds of state to protect: the contents of the virtual data store, which describe every virtual object, and any persistent volumes holding application data.

Upgrading

A virtual cluster has two things to upgrade: the project’s own components, such as the syncer, and the Kubernetes version of its control plane. They can be upgraded separately and independently of the host cluster. The usual approach is to test on a copy first, take a backup, upgrade one step at a time and verify that workloads behave properly. Because virtual clusters are cheap, creating a test copy of a production virtual cluster for upgrade rehearsal is practical in a way that is expensive with real clusters.

Two layers to look at, and a calm order in which to look at them.

Because every workload has a virtual view and a host view, monitoring needs to cover both. Host metrics, grouped by namespace, show how much each virtual cluster really uses. The virtual cluster’s own components, such as the API server and syncer, produce logs and metrics that show its health. Tenants see their own metrics, for example through a metrics server inside the virtual cluster, while platform teams watch the host.

A calm troubleshooting routine

Where virtual clusters earn their place.

flowchart LR
    PR["Pull request opened"] --> VC["Pipeline creates a virtual cluster"]
    VC --> DEP["Deploy the application and dependencies"]
    DEP --> TEST["Run integration tests"]
    TEST --> RES["Report results"]
    RES --> DEL["Delete the virtual cluster"]
        

A checklist that experienced platform teams follow.

A checklist before giving a virtual cluster to a team

Five habits that regularly cause trouble.

A balanced view before you decide.

How it compares with the alternatives

Virtual clusters are not a replacement for every other approach. They are a very good fit when teams need freedom and a clean environment more than they need hard hardware boundaries. When regulations or threat models demand physical separation, real clusters, or virtual clusters with private nodes, remain the right answer. Many organisations use all of these side by side, choosing the lightest tool that satisfies each team’s real needs.

How organisations use virtual clusters in practice.

Short answers to the questions beginners ask most often.

It has a real Kubernetes API server and controllers, and the project is certified as conformant, so standard tools and applications work. What it lacks by default is its own machines, which it borrows from the host cluster.

On the nodes of the host cluster. The syncer creates a matching Pod in a host namespace, and the host scheduler and kubelet run it.

Yes. Definitions and custom resources live in the virtual cluster’s own data store, so they do not clash with those of other tenants or the host.

Not by default. The control plane is separate, but workloads share the host’s nodes, kernel and often network. Stronger isolation needs host policies, dedicated or private nodes, or sandboxed runtimes.

Yes, within the range the project supports. The virtual control plane has its own version, though node behaviour still follows the host.

With quotas and limit ranges on the host namespace. The quota caps the total usage of the whole virtual cluster, however many namespaces the tenant creates inside it.

Yes, for everything that is synced to the host, such as Pods, Services, claims and the secrets that Pods use. Virtual clusters do not hide workloads from the host operator.

Typically seconds to a couple of minutes, depending on the configuration and whether container images are already available on the host. This is far quicker than provisioning a real cluster.

Its control plane and synced objects are removed from the host namespace. Persistent volumes may or may not be deleted, depending on the reclaim rules, so check them before deleting anything that holds important data.

Use namespaces when simple sharing among trusting teams is enough. Choose virtual clusters when teams need their own API, custom definitions, administrator rights or clean environments, and the stronger separation of real clusters is not required.

Namespaces are cheap but cramped, and real clusters are roomy but costly. Virtual clusters sit between them. Each one is a small Kubernetes control plane running in a namespace of a host cluster, with its own API server, data store, namespaces, roles and custom definitions. A syncer copies only what the host needs to run workloads, mainly Pods, Services and claims, so the real machines, network and storage are shared. This gives teams the freedom of their own cluster in seconds and at a fraction of the cost, and makes ephemeral environments, developer sandboxes, operator testing and tenant platforms practical. The trade-off is that isolation of the data plane is only as strong as the host and its settings, so quotas, security policies, network rules, dedicated nodes and a clear understanding of what the host administrator can see are essential. Used with those guardrails, virtual clusters turn cluster creation from a project into a routine.