Kubernetes is excellent at managing applications. Strangely, it was not built to manage itself. That gap is what this chapter is about.

A cluster has a whole life, not just a birth

People often think of cluster setup as one event: run an installer, wait, done. In reality a cluster lives through many stages, and each stage brings its own risks. The set of activities from birth to retirement is called the cluster lifecycle.

Pets and cattle

Operations teams use a famous picture. A pet is a server you name, nurse back to health and cannot bear to lose. Cattle are servers you number, replace when sick and never worry about individually. Healthy infrastructure treats machines as cattle. But a hand-built cluster tends to become a pet: it has a story, odd settings, special fixes and a person who knows its secrets. When that person leaves, the cluster becomes a risk.

Cluster sprawl

Why would anyone need many clusters? Good reasons are common. Separate clusters for production and testing. Separate clusters per region for lower delay and legal rules about where data may live. Separate clusters per team or customer for stronger isolation. Separate clusters for different Kubernetes versions during migrations. Edge locations, such as stores and factories, each with a small cluster. Before long, an organization may own dozens or hundreds.

What goes wrong without a system

• Drift. Clusters that began identical slowly become different, and nobody can say how.
• Slow upgrades. Each upgrade is a nervous, manual project, so clusters fall behind and become insecure.
• Knowledge locked in heads. Only a few people know the steps.
• Slow recovery. After a failure, rebuilding takes days instead of minutes.
• Orphaned resources. Forgotten disks, load balancers and networks keep costing money.

Cluster API is one choice among several. Seeing the whole landscape helps you understand what it is good at, and what it is not trying to be.

Common approaches

The key difference: continuous reconciliation

Most older tools are imperative runs. You run them, they do work and they stop. If a machine disappears an hour later, nothing notices. Cluster API is built from controllers that run forever. They compare the clusters you asked for with the clusters that exist, every few seconds, forever. A missing machine is noticed and replaced, not because someone ran a script, but because the controller always wants reality to match the description.

Cluster API is easy to understand once you see its one central trick: it uses a Kubernetes cluster to manage other Kubernetes clusters.

Two kinds of clusters

Every Cluster API setup has two roles. They may be played by different clusters, and the names are worth learning because they are used everywhere.

• The management cluster is the control room. It runs the Cluster API controllers. It stores descriptions of all the clusters you want. It is where you apply changes, and where the work is decided.
• A workload cluster, sometimes called a target cluster, is a cluster that Cluster API creates and looks after. It is where your applications run. There can be as many as you like.

Clusters as ordinary objects

In the management cluster, a workload cluster is represented by a small set of Kubernetes objects. Want a cluster with three control plane machines and five workers on a certain Kubernetes version? You write those wishes as objects and apply them with the same tools you use for deployments. The Cluster API controllers read them and start building. Want to upgrade? Change the version in the object. Want to add workers? Change a number.

flowchart TB
    ADMIN["Platform engineer or GitOps tool"] -->|"applies cluster descriptions"| MGMT
    subgraph MGMT["Management cluster"]
        OBJ["Cluster API objects"] --> CTRL["Cluster API controllers"]
    end
    CTRL -->|"calls cloud and data center APIs"| INFRA["Infrastructure: networks, machines, load balancers"]
    INFRA --> W1["Workload cluster A"]
    INFRA --> W2["Workload cluster B"]
    INFRA --> W3["Workload cluster C"]
    CTRL -.->|"watches health and reconciles"| W1
    CTRL -.->|"watches health and reconciles"| W2
    CTRL -.->|"watches health and reconciles"| W3

Why this works so well

• One language. Teams already know how to write, review and apply Kubernetes objects. Clusters become just another kind of object.
• One workflow. The same version control, review and GitOps practices that manage applications now manage infrastructure.
• Built-in self-healing. Controllers reconcile continuously, so drift is corrected automatically.
• Common interfaces. Providers for different clouds follow shared rules, so most of your habits carry over from one infrastructure to another.

Where it comes from

Cluster API is a project of the Kubernetes community, run by the special interest group responsible for cluster lifecycle, the same group that maintains kubeadm. It started in 2018 and has grown into a mature project with a large set of providers. Its command-line helper is called clusterctl.

Cluster API defines a small family of object types. If you already understand pods and deployments, you are halfway there, because the design deliberately mirrors them.

The mirror: applications and machines

Kubernetes manages application containers through pods, ReplicaSets and Deployments. Cluster API manages the machines that make up a cluster through three objects with the same relationships. This is the single most helpful idea for learners.

The full cast of objects

flowchart TB
    CL["Cluster"] --> CP["Control plane object - KubeadmControlPlane"]
    CL --> INF["Infrastructure cluster object"]
    CL --> MD["MachineDeployment - workers"]
    MD --> MS["MachineSet"]
    MS --> M1["Machine 1"]
    MS --> M2["Machine 2"]
    CP --> M3["Control plane Machine"]
    M1 --> IM1["Infrastructure machine - the real server"]
    M1 --> BC1["Bootstrap config - how to join"]
    MHC["MachineHealthCheck"] -.->|"watches"| MS

The cleverest part of Cluster API is how it separates the general idea of a cluster from the messy details of any particular cloud.

The problem providers solve

Creating a virtual server on Amazon Web Services is different from creating one on Azure, on a VMware data centre or on bare metal. The calls, the names and the rules all differ. If the core Cluster API controllers had to know every infrastructure, the project would be impossible to maintain. Instead, the core knows only the general ideas, such as “a machine”, “a cluster endpoint” and “a node that must join”, and delegates the details to plug-ins called providers.

The four kinds of providers

Many infrastructure providers exist

The community maintains providers for the major public clouds, for VMware vSphere, for OpenStack, for bare metal through projects such as Metal3, and for many smaller platforms. There is also a provider that uses Docker containers as pretend machines. It is meant for local development and testing, not for production. A rich ecosystem of providers means you can often use the same habits and even the same blueprints across very different environments.

How a provider plugs in: the contract

Providers do not talk to the core by guesswork. They follow a published contract, a set of rules about what their custom objects must look like and how they must report progress. For example, a provider’s machine object must report when the server is ready and give an identifier for it, called a provider ID. The core controllers watch for these signals and act when they appear. Because the contract is public, anyone can write a new provider for a new platform.

flowchart TB
    CORE["Core Cluster API controllers"] -->|"Machine refers to"| IMT["Infrastructure machine object - provider specific"]
    CORE -->|"Machine refers to"| BTS["Bootstrap config object - provider specific"]
    CORE -->|"Cluster refers to"| ICL["Infrastructure cluster object - provider specific"]
    CORE -->|"Cluster refers to"| CPO["Control plane object - provider specific"]
    IMT --> AWS["AWS"]
    IMT --> AZ["Azure"]
    IMT --> VS["vSphere"]
    IMT --> BM["Bare metal"]
    BTS --> KB["kubeadm bootstrap"]

Why the references are objects, not fields

You will notice that a Machine does not hold settings like “instance size” or “disk type”. It holds a reference to another object that does, owned by the provider. This keeps the core small, lets each provider evolve freely and makes it possible to reuse the same shape across clouds. The cost is that a full cluster description is made of several objects that point at each other, which can feel verbose at first. Blueprints, covered later, hide much of that.

There is a small chicken-and-egg question at the start: to create clusters with a cluster, you first need a cluster. Here is how teams solve it.

Step one: get a first cluster

The management cluster can be any working Kubernetes cluster. For learning, it is often a tiny throwaway cluster created on a laptop with a tool that runs Kubernetes inside containers. For real use, it may be a cluster created by hand, a managed cloud cluster or a cluster built by a previous Cluster API run.

Step two: install the controllers with clusterctl

clusterctl is the command-line helper for the project. Its first job is to install Cluster API into the management cluster. A single initialization command installs the core controllers, plus the infrastructure, bootstrap and control plane providers you choose. It also installs a helper called cert-manager, which issues the certificates that the controllers’ own web hooks need.

Other jobs of clusterctl

• Describe. Show a tree of a cluster’s objects with the health of each one, which is excellent for troubleshooting.
• Get kubeconfig. Fetch the access file that lets you talk to a workload cluster.
• Upgrade. Update Cluster API and its providers themselves, which is separate from upgrading the Kubernetes version of workload clusters.
• Move. Transfer all the cluster descriptions from one management cluster to another. This allows the pivot described in a later chapter.

flowchart LR
    BOOT["Bootstrap cluster - for example a small local cluster"] --> INIT["clusterctl init installs Cluster API and providers"]
    INIT --> MGMT["Management cluster is ready"]
    MGMT --> GEN["Generate and apply cluster description"]
    GEN --> WL["Workload cluster is built"]
    WL -.->|"optional pivot with clusterctl move"| SELF["Workload cluster becomes the new management cluster"]

When you apply the objects for a new cluster, a chain of controllers wakes up. Following the chain shows how much hidden work the system does for you.

The chain of events

sequenceDiagram
    participant U as User or GitOps
    participant C as Core controllers
    participant I as Infrastructure provider
    participant K as Control plane provider
    participant B as Bootstrap provider
    participant W as New cluster
    U->>C: Apply Cluster and related objects
    C->>I: Build network and load balancer
    I-->>C: Endpoint ready
    C->>K: Create control plane
    K->>K: Generate certificates
    K->>B: Request init bootstrap data
    B-->>K: Startup data in a secret
    K->>I: Create first machine with startup data
    I->>W: Server boots and forms cluster
    W-->>K: Control plane responds
    K->>I: Add remaining control plane machines
    C->>I: Create worker machines
    I->>W: Workers join
    C-->>U: Cluster reported ready

Why order matters

Notice that each step waits for the previous step to report ready. Controllers do not run a script from top to bottom. Each one watches the objects it cares about and acts when its inputs are ready. If something fails in the middle, for example a cloud quota is hit, the process pauses at that point, reports the reason in a status condition and retries. When you fix the cause, it continues from where it stopped, without starting over.

The control plane is the brain of a cluster. Managing it safely is the most delicate part of the lifecycle, and Cluster API gives it special treatment.

Why the control plane is special

Worker machines are interchangeable. If one dies, another can take its place and nothing is lost. Control plane machines are different, because they hold etcd, the database that stores everything the cluster knows. Etcd works by agreement among members, called a quorum. With three members, the cluster survives the loss of one. With five, it survives the loss of two. Replace members carelessly, and you can lose agreement or lose data.

What the control plane controller does

Choosing the number of control plane machines

Even numbers such as two or four add cost without adding failure tolerance, so odd numbers are the rule.

The stable address

All the nodes and users must reach the control plane at an address that does not change when machines come and go. The infrastructure provider usually creates a load balancer for this purpose, and the cluster’s endpoint points to it. Without a stable endpoint, replacing a control plane machine would break every node’s connection to the cluster.

A newly created server knows nothing about Kubernetes. The bootstrap process is how it learns who it is and which cluster to join.

The startup data

When the infrastructure provider creates a server, it can hand it a block of text that the server runs on first boot. This is usually in a format called cloud-init, or a similar format for some operating systems. The bootstrap provider writes this text. It contains everything the server needs: files to place on disk, certificates, settings for the Kubernetes agent and commands to run.

For a control plane machine that starts a new cluster, the text tells the machine to initialize the cluster. For every other machine, it tells the machine to join an existing cluster using a time-limited token, the cluster’s address and a fingerprint to check that the cluster is genuine. The bootstrap data is stored in a secret in the management cluster, and the provider passes it to the server at creation.

The join sequence

Why the provider ID matters

The provider ID is the glue between two worlds: the cloud’s idea of a server and Kubernetes’ idea of a node. If these cannot be matched, because of a missing cloud integration for example, the Machine never gets its node reference and appears stuck. This is one of the most common reasons for a half-built cluster.

Access to the new cluster

While the cluster is being built, the control plane provider produces a kubeconfig file, which is the document that tells tools where the cluster is and how to authenticate. It is stored as a secret in the management cluster, named after the cluster. Authorized users fetch it to run commands against the workload cluster. The controllers use it too, to look inside the cluster, for example to find nodes.

Worker nodes carry your applications. Cluster API manages them with the same rolling approach that Kubernetes uses for application pods.

The three layers of workers

A MachineDeployment describes a group of workers: how many, which Kubernetes version, and which templates to use for the infrastructure and the bootstrap. It creates one MachineSet for each version of the template. The MachineSet creates and maintains the individual Machines. When you change the template, the MachineDeployment makes a new MachineSet and slowly moves machines over from the old one.

Templates are immutable

Here is an important rule. Infrastructure and bootstrap templates are not edited after use. If you want workers with a bigger instance size or a different disk, you do not change the existing template. You create a new template and point the MachineDeployment to it. This triggers a rollout that replaces old machines with new ones. The rule keeps history clear and ensures that all machines made from one template are truly identical.

Controlling the pace of change

The life of one Machine

Each Machine reports a phase, which tells you where it is in life. Reading phases is the quickest way to see what is happening.

Deletion is careful too

When a Machine is deleted, Cluster API does not just yank the server away. It first cordons the node, which means telling the scheduler to place no new pods on it. Then it drains the node, which means politely evicting the running pods so they move elsewhere, respecting disruption budgets. Only after that does it delete the node object and finally the server itself. This graceful sequence protects your applications during scale-downs and upgrades.

Upgrading Kubernetes is the task that teams fear most. Cluster API turns it into a repeatable, observable rollout.

The core approach

Cluster API upgrades clusters by replacing machines, not by logging into each one and updating software. To move to a newer Kubernetes version, you change the version field in the description. The controllers then create new machines with the new version and remove old ones, a few at a time. The result is that every machine always matches its description, with no half-upgraded leftovers or hand-made differences.

The order of operations

flowchart TB
    START["Change version in the description"] --> CPN["Create new control plane machine at the new version"]
    CPN --> CPH{"New member healthy and in quorum?"}
    CPH -->|"yes"| CPD["Remove one old control plane machine"]
    CPH -->|"no"| WAIT["Wait, report condition, retry"]
    CPD --> CPM{"More old control plane machines?"}
    CPM -->|"yes"| CPN
    CPM -->|"no"| WK["Start worker rollout"]
    WK --> WN["Create new worker, respecting max surge"]
    WN --> WD["Drain and delete an old worker, respecting max unavailable"]
    WD --> WM{"More old workers?"}
    WM -->|"yes"| WN
    WM -->|"no"| DONE["Cluster fully upgraded"]

What makes upgrades safe

• Quorum checks. The control plane controller will not remove a member if doing so would break etcd.
• Surge. New workers are created before old ones are removed, so total capacity does not drop.
• Draining and disruption budgets. Pods are moved off gracefully, and applications that define budgets are protected.
• Pause and resume. A rollout can be paused if something looks wrong.
• Roll back by replacing. Since machines are disposable, returning to the previous description rolls the fleet back the same way.

Things that still need your attention

• Applications must tolerate movement. Pods will be evicted and rescheduled. Applications with one replica, no disruption budget or local-only state will feel it.
• Add-ons must be compatible. The network plugin, storage drivers and cloud integration have their own supported versions.
• Removed APIs. Newer Kubernetes versions drop old API versions. Check your manifests before upgrading.
• Cloud limits. Surging needs spare quota for the extra machines.

Machines fail. Disks die, hosts are retired by the cloud and networks glitch. Cluster API can notice and replace sick machines without waiting for a human.

What a MachineHealthCheck does

A MachineHealthCheck is an object that selects a group of machines by label and describes what “unhealthy” means. The usual rule looks at the node’s ready condition. If a node has been not ready, or has been unreachable, for longer than a stated time, the machine is considered unhealthy. The controller then remediates it, which normally means deleting the Machine. Since the MachineSet that owns it wants a fixed number of machines, it immediately creates a replacement.

The important settings

Why the safety brake matters

Imagine the link between the management cluster and a workload cluster breaks. From the controllers’ point of view, every node suddenly looks unreachable. Without a brake, the health check would happily delete every machine in the cluster. The max unhealthy setting prevents this: when more than the allowed share of machines look sick, it concludes that something larger is wrong and stops, waiting for a human.

Control plane healing

Control plane machines can be remediated too, but with extra care. The control plane controller only replaces a member if the remaining members keep etcd healthy, and it replaces them one at a time. A cluster with a single control plane machine cannot safely heal itself, which is one more reason to run three in production.

Spreading risk across zones

Self-healing works best when machines are spread across failure domains, such as availability zones. If one zone fails, only the machines in it are lost. The control plane keeps quorum and the workers in other zones keep running, and replacements are created in healthy zones. Planning failure domains in advance turns a disaster into a routine recovery.

flowchart TB
    MON["MachineHealthCheck watches nodes"] --> BAD{"Node unhealthy longer than the timeout?"}
    BAD -->|"no"| OK["Keep watching"]
    BAD -->|"yes"| LIM{"Share of unhealthy machines below the limit?"}
    LIM -->|"no"| STOP["Stop remediation and alert a human"]
    LIM -->|"yes"| DEL["Delete the unhealthy Machine"]
    DEL --> MS["MachineSet notices a missing machine"]
    MS --> NEW["New Machine is created"]
    NEW --> JOIN["New node joins and takes workloads"]

A single cluster description is made of many objects that point at each other. For one cluster that is fine. For fifty it becomes a pile of nearly identical files. ClusterClass solves this.

The problem of repetition

Suppose you create ten clusters for ten teams. Each needs a Cluster object, a control plane object, an infrastructure cluster object, templates for machines and a MachineDeployment. Copy these ten times with small changes, and soon the copies drift. A fix made in one is forgotten in the others. Reviews become tiring because every change touches dozens of files.

The idea of a class

A ClusterClass is a reusable blueprint. It holds the shared design: the templates for the control plane, the templates for the infrastructure, and the named kinds of worker groups, for example “general” and “high-memory”. Individual clusters then refer to the class and state only what makes them different, such as the Kubernetes version, the number of control plane machines and how many workers of each kind they want. This part of a cluster is called its topology.

Variables and patches

To allow differences without breaking the blueprint, a class can define variables, which are named settings that each cluster may provide, such as an instance size or a region. It also defines patches, which are rules that take the variable values and apply them into the right places in the templates. This keeps the class in control: clusters can customize only what the class permits.

Managed topology: changes by editing one place

When a cluster uses a class, the controllers keep the underlying objects in line with the class and the topology. Change the Kubernetes version in the topology, and the controllers roll the control plane and the worker groups in the right order. Change a worker count, and the right MachineDeployment is adjusted. Update the class itself, and every cluster using it can be brought along, subject to your rollout process. One edit, many consistent results.

flowchart TB
    CC["ClusterClass - shared blueprint"] --> T1["Cluster A topology: version, 3 control plane, 5 general workers"]
    CC --> T2["Cluster B topology: version, 3 control plane, 2 general and 4 high memory workers"]
    CC --> T3["Cluster C topology: version, 1 control plane, 2 general workers"]
    T1 --> O1["Generated Cluster API objects for A"]
    T2 --> O2["Generated Cluster API objects for B"]
    T3 --> O3["Generated Cluster API objects for C"]

Cluster API handles scaling at the machine level. It works together with the pod-level and node-level autoscalers of Kubernetes.

Manual scaling

The simplest scaling is to change a number. Increase the replicas of a MachineDeployment and new machines appear. Decrease it and surplus machines are drained and removed. For the control plane, scaling changes the number of members, which should be done carefully and only between supported counts such as one, three and five.

Automatic scaling with the Cluster Autoscaler

The Kubernetes Cluster Autoscaler has an integration with Cluster API. It treats each MachineDeployment as a node group. When pods cannot be placed because there is not enough room, it increases the replicas of a suitable MachineDeployment. When nodes are underused, it reduces them. You mark each MachineDeployment with minimum and maximum sizes, using annotations, so that the autoscaler can only move within safe bounds.

Each level of the system does one job. The workload autoscaler asks for more pods, the node autoscaler asks for more machines, and Cluster API builds the machines. They talk through ordinary Kubernetes objects.

Machine pools

Some clouds offer their own scaling groups, such as groups of identical virtual machines that the cloud manages. Cluster API can map a MachinePool to such a group. The cloud then handles creating and replacing machines, and Cluster API watches over the group. This can be faster and cheaper, but it moves some control to the cloud and may be less uniform across providers, so check the maturity of the feature in your provider.

A cluster that merely exists is not yet useful. It needs networking, storage, cloud integration and policy. Cluster API deliberately leaves most of this to other tools, but gives you ways to fit them in.

What a new cluster is missing

Ways to install add-ons

• ClusterResourceSet. A built-in feature that applies a bundle of resources, such as a network plugin’s manifests, to every new cluster matching a label. It is simple, and good for the first essentials. It is mostly “apply once” and does not manage later updates of the add-ons well.
• Add-on providers. Community add-on providers install packages such as Helm charts into workload clusters and keep them in step with a declared version.
• GitOps tools. A tool running in the management cluster, or in each workload cluster, applies the add-ons from a repository and keeps them reconciled. This is the most popular approach for fleets because it handles updates and drift too.
• Baked into images. Some teams include certain components in the machine image or bootstrap data. This is quick but ties add-on updates to machine replacement.

The ordering challenge

Add-ons depend on each other and on the cluster state. The network plugin must exist before pods can start. The cloud controller manager must exist before nodes get their provider IDs and become ready. Storage drivers need the cloud integration. If you install them in the wrong order or too late, nodes sit not ready, and the Machines appear stuck. Plan a clear startup sequence for each cluster type and test it end to end.

Day-two tasks

• Certificate renewal. Certificates expire. The control plane controller can renew them as machines are replaced, and can be set to rotate machines that are close to expiry.
• Operating system patching. Because machines are replaced, security patches reach the fleet by building new images and rolling the machines, which is cleaner than patching in place.
• Backups. Back up etcd or cluster state for workload clusters, as well as the management cluster’s own objects.
• Cost control. Track which clusters exist, who owns them and when they should be removed.
• Decommissioning. Deleting a Cluster object removes the machines and the cloud resources it created, in order, which prevents leftover costs.

The management cluster holds the keys to your whole fleet. Designing it for safety and recovery is one of the most important decisions you will make.

What happens if it is lost?

Here is a comforting fact. Workload clusters keep running if the management cluster is down. Their control planes and nodes do not depend on the controllers for their daily operation. Applications continue to serve users. However, you lose the ability to create, upgrade, scale and automatically heal those clusters until the management cluster comes back. If a machine fails during that time, nothing replaces it.

The bigger danger is data loss. The management cluster stores the descriptions, the certificate authorities and the kubeconfig secrets of every workload cluster. If those are lost with no backup, you may be unable to manage or even reach your clusters.

Protecting it

Self-managed clusters and the pivot

A common pattern is to start with a temporary management cluster, create a real cluster with it, and then move the Cluster API controllers and objects into that new cluster. This is the pivot, done with the clusterctl move command. After it, the new cluster manages itself, and the temporary one can be discarded. Self-managed clusters reduce the number of systems to look after, but they have a delicate property: a broken cluster may have lost the very controllers needed to fix it. Keep strong backups and a rebuild plan for this case.

Layouts for the management layer

Cluster API turns clusters into data. Once clusters are data, all the habits of good software delivery apply to them.

The pattern

In a GitOps setup, a Git repository holds the desired state, and a tool inside the cluster continuously compares the repository with the live objects and fixes differences. For clusters, this means the cluster descriptions live in Git. A pull request that adds a cluster file creates a cluster. A change of version upgrades it. Removing the file removes the cluster. Everything is reviewed, recorded and reversible.

flowchart LR
    DEV["Engineer opens a pull request"] --> REV["Review and approval"]
    REV --> GIT["Git repository: cluster descriptions"]
    GIT --> SYNC["GitOps tool in management cluster"]
    SYNC --> CAPI["Cluster API objects"]
    CAPI --> CTRL["Cluster API controllers"]
    CTRL --> FLEET["Workload clusters created, upgraded, repaired"]
    FLEET -.->|"status"| CAPI

Benefits

• Audit trail. Every cluster change has an author, a reviewer and a date.
• Easy rollback. Revert a commit, and the fleet follows.
• Self-service with guard rails. Teams request clusters by opening a pull request against an approved class, and checks verify that they stay within policy.
• Disaster recovery. The repository is a complete record of what should exist.
• Consistency. The same process creates every cluster, so drift disappears.

A sensible repository layout

Promotion across environments

Treat cluster upgrades like application releases. A new Kubernetes version or a class change first goes to a development cluster, then to staging clusters, then to production in waves. Each wave is a separate pull request, with checks between them. This slow, observable path replaces a risky big-bang upgrade with a series of small, reversible steps.

A system that can build and destroy clusters is a very attractive target. Security here means protecting both the factory and its products.

What is at stake

Protective measures

• Least-privilege cloud permissions. Give the provider only the rights it needs, scoped to specific accounts, regions or resource groups. Avoid broad administrator keys.
• Separate identities per team or environment. Many providers let a cluster refer to its own credentials. Then a team’s cluster uses a team-scoped identity, so a mistake or break-in in one area cannot reach the rest.
• Short-lived credentials. Prefer identity systems that issue temporary credentials over long-lived keys, where the provider supports them.
• Encrypt secrets at rest in the management cluster and restrict who can read secrets. Reading a kubeconfig secret is the same as being an administrator of that workload cluster.
• Strong role-based access control. Few people should be able to edit Cluster API objects directly. Most changes should flow through reviewed repository changes.
• Network isolation. Limit which networks can reach the management cluster’s API. The controllers need to reach workload clusters, but ordinary users usually do not need to reach the management cluster at all.
• Audit logging. Record all changes to the management cluster, and alert on unusual actions such as bulk deletions.
• Supply chain care. The providers and controllers are software you run with high privilege. Install them from trusted sources, verify image signatures, pin versions and update promptly when security fixes arrive.
• Protect deletion. Consider extra approval steps for deleting production clusters, because a single mistaken delete can remove a whole environment.

Security of the workload clusters themselves

Cluster API builds clusters with sensible defaults, but it does not replace a security baseline. Use hardened machine images, enable audit logging on the control plane, apply pod security rules, restrict network traffic with policies, encrypt secrets stored in etcd, and rotate certificates. Capture these choices in the class or template, so every new cluster starts secure rather than relying on a post-creation checklist.

Tenancy

If many teams request clusters through the same management cluster, make sure teams cannot see or change each other’s clusters. Use namespaces per team for the Cluster API objects, role bindings that restrict access to those namespaces and policies that limit which classes, regions and machine sizes each team may use.

When a cluster refuses to appear or an upgrade stalls, you need to know exactly where to look. Cluster API is designed to explain itself, if you know how to read it.

The language of conditions

Every Cluster API object reports conditions, which are short statements about its state with a reason and a message. They roll up through the family tree. A Cluster is ready only if its control plane and infrastructure are ready, and the control plane is ready only if its machines are ready. When something is wrong, the conditions point toward the failing part. The clusterctl describe cluster command draws this tree for you, with the health of each object.

What to watch

Troubleshooting walk-through

Cluster API is powerful, but it is not the right tool for every team. An honest ledger will help you decide.

Compared with other approaches

They can work together

These choices are not strict opposites. Cluster API has providers that create managed Kubernetes clusters on major clouds, so you can keep the same declarative habits while the cloud runs the control plane. Teams also use Terraform to create the management cluster and its surrounding network, and then Cluster API for the workload clusters. The right mix depends on your size, your skills and how many platforms you must support.

Teams that succeed with Cluster API repeat a small set of habits. Teams that struggle repeat a small set of mistakes.

Patterns that work

Anti-patterns to avoid

Best practices at a glance

• Start small. Learn with a local development setup, then a single real cluster, and add classes and fleet features as your needs grow.
• Keep everything in Git, reviewed and versioned, with automated checks on every change.
• Plan networking and add-ons first. Know how nodes will become ready before you create the first production cluster.
• Use odd control plane counts of three or five, spread across zones.
• Test upgrades regularly, not only when forced. Frequent small upgrades are safer than rare large ones.
• Prepare applications for movement with multiple replicas, disruption budgets and readiness checks.
• Monitor the controllers as production services in their own right.
• Control credentials tightly, using least privilege and separate identities.
• Practice disaster recovery. Rebuild a management cluster from backups and Git at least once before you need to.
• Track version support. Keep Cluster API, providers and Kubernetes within supported combinations.

Common mistakes

• Expecting Cluster API to choose and install networking and storage by itself.
• Running a single control plane machine in production.
• Editing machine templates in place instead of creating new ones.
• Forgetting cloud quotas, so surge machines or new clusters cannot be created.
• Skipping backups of the management cluster and its secrets.
• Forcing finalizers off and leaving orphaned cloud resources.
• Upgrading across several minor versions in one jump.
• Letting two tools control the same replica count.

Cluster API is used both directly by platform teams and as a building block inside commercial products.

Short answers to the questions that new users ask most often.