Profiling is the practice of measuring where time and resources are spent, so you improve the real problem instead of a guess.

Profiling in everyday words

To profile something is to measure it carefully and break the result into parts. If your morning commute takes ninety minutes, you can complain about traffic, or you can time each stage: walking to the station, waiting for the train, riding, and walking to the office. Suddenly you learn that thirty of those minutes are spent waiting on a single platform. That breakdown is a profile. It tells you where to act, and just as importantly, where acting would be a waste of effort.

Profiling the control plane works the same way. Instead of saying “the cluster feels slow”, we measure how long the interface server takes to answer different kinds of requests, how long the database takes to save changes, how long the scheduler takes to place a pod, and how long the controllers take to react to a change. The result is a timeline that shows which stage consumes the time.

Why this matters more as clusters grow

A small cluster with a few dozen workloads rarely shows control plane stress. The components have plenty of room. As a cluster grows to hundreds of nodes, thousands of pods, and many automation tools constantly watching and updating objects, the work done by the control plane grows faster than the cluster itself. Each new controller, each new custom resource type, and each new dashboard that polls the interface adds load. Problems that were invisible at small scale suddenly appear: deployments that take minutes to start, pods stuck waiting to be scheduled, and commands that hang before returning.

What profiling is not

Profiling is not the same as monitoring, though they are close relatives. Monitoring watches known numbers over time and warns you when they cross a line. Profiling goes deeper: it investigates why a number looks the way it does by drilling into the internals. Monitoring tells you the interface server is slow today. Profiling tells you that a particular kind of list request, issued by a particular tool, is responsible for most of the slowness. You need both, and good profiling usually starts because monitoring raised a question.

You cannot profile what you cannot name. Meet the parts, and what each one is responsible for.

The components and their jobs

The control plane is a small group of cooperating programs. Each has a narrow responsibility, and they communicate almost entirely through one shared hub, the interface server. Understanding this hub-and-spoke design is the key to profiling, because it means that almost every performance problem eventually shows up as extra work at the hub or at the database behind it.

flowchart TD
  U["Users, tools, and automation"] --> A["API server"]
  K["Kubelets on nodes"] --> A
  A  E[("etcd database")]
  S["Scheduler"]  A
  C["Controller manager"]  A
  X["Cloud controller manager"]  A
  A --> W["Admission webhooks and aggregated APIs"]
        

Why the hub-and-spoke design shapes profiling

Notice that the scheduler never talks directly to the controller manager, and neither of them ever touches the database. They all go through the interface server. This keeps the system simple and consistent, but it also means the interface server is a shared resource. If one client sends heavy requests, every other client feels it. If the database is slow, the interface server becomes slow, and then everything attached to it slows down together. When you profile, you will often find that symptoms appear in one component while the cause lives in another.

Follow one request from arrival to answer. Each stop is a place where time can be spent, and where you can measure it.

The journey through the API server

When you ask Kubernetes to create a pod, your request does not go straight to the database. It passes through a series of stages, like a visitor moving through airport security. Each stage has a purpose, and each stage costs some time. For a healthy cluster each stage is tiny. For a struggling cluster, one stage often grows to dominate the total.

flowchart LR
  R["Request arrives"] --> A1["Authentication: who are you"]
  A1 --> A2["Authorization: are you allowed"]
  A2 --> A3["Mutating admission: adjust the request"]
  A3 --> A4["Schema validation"]
  A4 --> A5["Validating admission: final checks"]
  A5 --> A6["Write to etcd"]
  A6 --> A7["Response returned"]
  A6 --> A8["Notify watchers of the change"]
        

Reads are different from writes

A read request, such as listing all pods in a namespace, takes a different and often cheaper path. The server can answer many reads from a copy it keeps in its own memory, called the watch cache, without bothering the database. But a read that demands the very freshest data, or one that asks for an enormous result, can bypass this shortcut and cause real work. Understanding which requests use the shortcut and which do not is one of the most valuable pieces of control plane knowledge.

A handful of ideas explain almost every performance conversation. Learn them once and the rest becomes much easier.

Latency: how long one thing takes

Latency is the time between asking and receiving. If you request a pod listing and the answer arrives in twenty milliseconds, the latency of that request was twenty milliseconds. Latency is what humans feel directly. A person running a command notices when it takes three seconds instead of three hundred milliseconds, even if they could not describe why.

Throughput: how many things per unit of time

Throughput is the number of requests handled each second. A control plane might serve two hundred requests per second comfortably and struggle at two thousand. Throughput describes capacity, while latency describes experience, and the two are linked: as throughput approaches the limit, latency rises sharply.

Saturation: how full the system is

Saturation measures how close a resource is to its limit, such as processor usage, memory, disk activity, or the number of requests waiting in a queue. A resource at ninety-five percent saturation has almost no room to absorb a spike, and a tiny burst of extra work will cause a pile-up. Saturation is the best early warning, because it rises before users complain.

Why averages lie and percentiles tell the truth

Suppose ninety-nine requests finish in ten milliseconds and one request takes ten seconds. The average is about one hundred and ten milliseconds, which sounds fine and hides the slow request completely. Percentiles fix this. The ninety-ninth percentile, often written p99, is the value below which ninety-nine percent of requests fall. It reveals the slow tail that real users hit regularly. Control plane health is almost always judged by percentiles, because the slow tail is where controllers time out, retries pile up, and people lose patience.

SLIs and SLOs: turning feelings into targets

A service level indicator, or SLI, is a measured number that reflects user experience, such as “the share of write requests finishing within one second”. A service level objective, or SLO, is the target you set for that number, such as “ninety-nine percent of write requests finish within one second over a month”. The Kubernetes community itself publishes scalability targets in this style, including expectations that common write and single-object read requests complete within about a second at the ninety-ninth percentile, with larger target times for broader list requests, and a short startup target for ordinary stateless pods. Setting your own objectives, informed by these published ones, turns vague complaints into measurable goals.

The control plane tells you a great deal about itself. You only need to know where to listen.

Metrics exposed by every component

Each control plane component publishes a rich set of numbers describing its own behavior, available over a standard web endpoint in a format that popular monitoring systems understand. A monitoring system scrapes these endpoints every few seconds, stores the history, and lets you chart and alert on them. This is the foundation of all control plane profiling, and it costs almost nothing to turn on.

Three levels of insight

The front door is where most control plane stories begin and where most of them can be understood.

Reading request duration the right way

The interface server records how long each request takes, grouped by the kind of action, such as get, list, create, update, patch, delete, and watch, and by the type of object involved. These durations are stored in buckets, which lets a monitoring system calculate percentiles. The first habit to build is to never look at all requests together. A flood of fast reads can hide a handful of painfully slow writes, and a single enormous list request can distort every chart. Always split by action and by resource type.

Long-running watch requests need special care. A watch is a request that stays open for minutes, so its duration is long by design and says nothing about health. Good dashboards exclude watches from latency charts and track them separately, as a count of open connections and as a rate of events delivered.

Signals that point to trouble

Request size and response size matter

Time is not the only cost. A request that returns forty megabytes costs processor time to assemble, memory to hold, and network time to send, even if it finishes quickly. Tracking response sizes by resource type reveals the heaviest consumers. It is common to discover that one resource type, perhaps a collection of configuration maps or events, accounts for the majority of bytes served.

When demand exceeds capacity, someone has to wait. This mechanism decides who, and keeps important traffic moving.

The problem it solves

Before modern fairness controls, the interface server protected itself with a simple limit on the total number of requests in flight at once. When the limit was reached, extra requests were rejected. This crude approach had a nasty flaw: one noisy client could consume the entire limit and starve everyone else, including critical components like the node agents and controllers that keep the cluster healthy. A runaway script could effectively take the cluster down.

How fairness works

The modern mechanism, called API Priority and Fairness, sorts incoming requests into categories using rules that look at who is asking and what they are asking for. Each category belongs to a priority level, and each priority level has its own share of the server’s capacity and its own set of waiting lines. Within a priority level, requests from different sources are placed into separate lines and served in a fair rotation, so that one chatty source cannot crowd out the rest. Requests that must wait are held in a line for a short time; if the line is too long or the wait too great, the request is rejected so the client can back off.

flowchart LR
  I["Incoming requests"] --> C["Classify by caller and request type"]
  C --> P1["High priority level: system and leader election"]
  C --> P2["Medium priority level: workload controllers"]
  C --> P3["Lower priority level: general clients"]
  P1 --> Q1["Fair queues"]
  P2 --> Q2["Fair queues"]
  P3 --> Q3["Fair queues"]
  Q1 --> E["Executed within capacity share"]
  Q2 --> E
  Q3 --> E
        

What to look for when profiling fairness

A few request patterns cause most control plane pain. Learn to recognize them on sight.

The unbounded list

A list request asks for every object of some type, possibly across the entire cluster. In a cluster with fifty thousand pods, an unfiltered cluster-wide list produces an enormous response. Building that response uses memory proportional to its size, and sending it ties up network and processor time. If many clients do this at once, the interface server can run out of memory and restart, and the restart makes every client reconnect and list again, which causes the problem to repeat. This feedback loop is one of the classic ways clusters fall into a spiral.

How good clients avoid the trap

Well-written controllers do not repeatedly list. They list once, then open a watch, and from then on they receive only changes, keeping a local copy in memory. This pattern, implemented by shared helper libraries often called informers, reduces load dramatically. Poorly written tools, by contrast, poll the full list every few seconds, and a handful of them can dominate the server.

The watch cache and the cost of freshness

The interface server keeps a recent copy of objects in memory and can answer many reads from it. A client can usually choose between “give me the freshest possible data”, which forces a trip to the database to confirm the latest state, and “any reasonably recent data is fine”, which can be served from memory. The first choice is safer for correctness but costs much more. Many tools request the freshest data out of habit, even when they do not need it, and those habitual requests can make up a large share of database reads.

Watches at scale

Every watch is a long-lived connection that costs memory and requires the server to check each change against it. Ten thousand watchers across a large cluster mean that every single write must be considered for delivery to many listeners. The rate of change multiplied by the number of watchers is the true workload. Tools that open many separate watches, instead of sharing one, multiply this cost needlessly.

Large objects

Objects have a size limit, but even objects well below the limit can cause trouble when there are many of them or when they change often. A configuration object holding a megabyte of data that is updated every few seconds creates constant database traffic, constant cache updates, and constant watch notifications. Oversized objects are an especially common cause of unexplained database growth and slow writes.

Kubernetes is deliberately extensible. Every extension is also a place where performance can leak away.

Admission webhooks

An admission webhook is a separate program that the interface server calls during a request, to inspect or modify the object before it is saved. Security tools, policy engines, and sidecar injectors all use them. Because the server waits for the answer, a webhook’s speed becomes part of every matching request. If a webhook takes half a second, every pod creation takes at least half a second longer. If the webhook is unreachable, requests may fail or stall until a timeout expires, depending on how the webhook is configured.

Profiling webhook cost

The interface server records how long admission steps take, broken down by the individual webhook. Reviewing these durations quickly reveals which webhooks consume the most time. Also review how many requests each webhook intercepts. A webhook limited to specific namespaces and object types costs far less than one that intercepts everything.

Custom resources and controllers

Custom resource definitions let you add new object types to the cluster. Each one adds to the amount the interface server must track and cache, and each usually comes with its own controller that watches and updates objects. A cluster with hundreds of custom resource types and tens of thousands of instances carries a significantly heavier load than one with a few. Review which custom types are truly needed and how many instances exist.

Aggregated APIs

Some extensions register entire API groups served by separate programs. The interface server forwards matching requests to them. Their speed and availability become part of the cluster’s perceived speed, and a failing aggregated API can cause slow or failing discovery for every client, which affects tools that scan all API groups. Monitor the health and latency of every aggregated service just as you monitor the core.

If the interface server is the front door, etcd is the vault behind it. Its speed sets the ceiling for everything above.

What etcd is and why it is special

etcd is a distributed key-value database that stores every object in the cluster: every pod, every secret, every configuration, every lease. It is designed for correctness above all else. It promises that once a change is confirmed, it is durable and that all members agree on the order of changes. That promise is valuable, and it comes at a performance price that you must understand to profile it well.

How a write works inside etcd

etcd runs as a group of members, typically three or five. One member is the leader. All writes go to the leader, which records the change in a log on its own disk and sends it to the other members. Each member records it on its own disk too. Only when a majority of members have safely written the change to disk does the leader confirm it. This means that the speed of a write is limited by the slowest disk among the majority and by the network between the members.

sequenceDiagram
  participant A as API server
  participant L as etcd leader
  participant F1 as Follower one
  participant F2 as Follower two
  A->>L: Write request
  L->>L: Append to log on disk
  L->>F1: Send log entry
  L->>F2: Send log entry
  F1->>F1: Append to log on disk
  F2->>F2: Append to log on disk
  F1-->>L: Acknowledge
  F2-->>L: Acknowledge
  L->>L: Majority reached, commit
  L-->>A: Write confirmed
        

The three resources etcd cares about

A handful of etcd numbers tell you almost everything about the health of the cluster’s memory.

The vital signs

Just as a doctor checks pulse, temperature, and blood pressure first, an engineer profiling etcd checks a short list of numbers before anything else. Each one maps to a specific kind of problem, so reading them together quickly narrows the diagnosis.

Why storage is the usual suspect

In a great many real incidents, slow control planes turn out to be caused by slow disks on the database machines. This happens most often when etcd shares a disk with other busy programs, when it runs on network-attached storage with variable latency, or when a cloud volume hits its input and output limits. The symptoms are subtle: everything works, but writes take longer, leader elections become more frequent because heartbeats miss their deadlines, and the whole cluster feels sluggish.

Why the database grows and what to do about it

etcd keeps a history of changes so that watchers can resume from where they left off. Over time, old versions pile up. A process called compaction discards history older than a chosen point, which marks space as reusable. A second process called defragmentation actually returns that space so the file shrinks. Without regular compaction, the database grows. If it reaches its size quota, etcd stops accepting writes and the whole cluster becomes read-only, which is one of the most disruptive failures a control plane can suffer. Kubernetes normally triggers compaction automatically, but you must still watch size and plan defragmentation during quiet periods, since defragmenting a member briefly blocks it.

flowchart TD
  A["Many updates create many versions"] --> B["Database file grows"]
  B --> C["Compaction discards old history"]
  C --> D["Free space inside the file"]
  D --> E["Defragmentation returns space to the disk"]
  B -->|"Quota reached"| F["Writes refused: cluster becomes read-only"]
        

Finding what fills the database

When the database grows unexpectedly, the question is always which object types and which objects take the most space. Usually a small number of types account for most of it, commonly events, large configuration objects, custom resources written by operators, or leftover objects from jobs that are never cleaned up. Counting objects by type and measuring their combined size points straight to the culprit. A common remedy is to move high-churn, low-value data such as events to a separate database so it cannot crowd the main one.

The scheduler solves a matching puzzle thousands of times a day. Profiling shows whether it keeps up.

What the scheduler actually does

When a new pod appears without a node, the scheduler picks one. It does this in two broad phases. First it filters, removing every node that cannot run the pod, for reasons such as insufficient resources, mismatched labels, or rules forbidding the placement. Then it scores the remaining nodes, ranking them by how well they fit, and picks the best. Finally it records the decision by telling the interface server which node was chosen, and the node’s agent takes over from there.

flowchart LR
  A["Pod waiting in queue"] --> B["Filter: remove nodes that cannot run it"]
  B --> C["Score: rank remaining nodes"]
  C --> D["Choose best node"]
  D --> E["Reserve resources in scheduler memory"]
  E --> F["Bind: tell API server the choice"]
        

What to measure

Common reasons scheduling feels slow

Controllers are the muscle of Kubernetes. When they fall behind, the cluster reacts late to everything.

How a controller works

A controller follows a simple loop. It watches objects of some kind, and whenever one changes, it places a small note in a work queue. A set of workers takes notes from the queue, compares desired state with actual state, and takes action to close any gap. For example, the controller for replicated sets notices that fewer pods exist than requested and creates the missing ones. The loop repeats forever. Because everything flows through queues, queue behavior is the best window into controller health.

flowchart LR
  W["Watch events from API server"] --> Q["Work queue"]
  Q --> K["Worker takes an item"]
  K --> R["Compare desired and actual state"]
  R --> A["Act: create, update, or delete objects"]
  A -->|"Failure"| Y["Retry later with backoff"]
  Y --> Q
        

The queue metrics that matter

Client-side rate limiting: the hidden brake

Every controller talks to the interface server through a client library that limits how fast it may send requests, to protect the server. This limit has two settings: a steady rate of requests per second, and a burst allowance for short spikes. If a controller needs to send more requests than its limit allows, the library makes it wait, silently. The controller appears slow, its queue grows, yet the interface server shows no stress at all, because the delay happens inside the client. This confuses many investigations.

Reading client-side throttling

Look for metrics and log messages showing time spent waiting on the client’s own rate limiter. Compare this with server-side latency: if the server answers quickly but the controller is slow, client-side limits are the likely explanation. Raising a client’s limits can fix this, but only after confirming that the server and database have spare capacity, because a higher limit simply moves the pressure forward.

Custom controllers and operators

The controllers you write or install yourself deserve the same scrutiny as built-in ones. Operators for databases, message brokers, and certificates often become the heaviest clients of the interface server, because they reconcile large numbers of objects frequently. Check that they use shared informer caches, that they avoid updating objects when nothing has changed, and that they do not trigger endless reconciliation loops by reacting to their own writes.

Thousands of node agents send constant status reports. At scale, their chatter is a major part of the load.

Heartbeats and node status

Each node runs an agent called the kubelet, which regularly tells the control plane that the node is alive and reports its condition. Early versions sent a full status update frequently, and each update was a write to the database. In a cluster with thousands of nodes, this created enormous write traffic. Modern versions use lightweight lease objects for the frequent heartbeat and send the full status much less often, which greatly reduces the load. Still, the number of nodes multiplied by their reporting rate remains an important contributor to control plane work.

Pods and their status updates

Every pod also generates status updates as it starts, becomes ready, changes, or stops. A cluster that frequently creates and destroys short-lived pods, for instance one running many small batch tasks, produces a constant stream of writes. Likewise, readiness and liveness checks that flip frequently cause repeated updates. Profiling often reveals that a few noisy workloads generate a large share of status traffic.

When you know which component is slow, code-level profiling shows exactly which part of its program is responsible.

What a code-level profile shows

The control plane components are written in the Go programming language, which has excellent built-in profiling support. A profile is a sampled record of what the program was doing. At regular tiny intervals, the profiler notes which functions were running, and over thousands of samples, a clear picture emerges of where time and memory go. The components expose profiling endpoints that authorized users can request, and the results can be viewed with standard tools.

Reading a flame graph

A flame graph turns sampled data into a picture. Each horizontal bar represents a function. The bar’s width shows how much of the total time that function and everything it called consumed. Bars are stacked, with callers below and callees above. The wide bars are your suspects. A wide bar near the top, with nothing above it, means that function itself uses a lot of time. A wide bar lower down with many narrower bars above it means time is spread across the things it calls.

Continuous profiling

Because slowdowns are often brief and unpredictable, some teams run continuous profiling, which samples programs at a very low rate all the time and stores the results. When an incident occurs, the profile from that exact moment is already available. The cost is small, and the benefit is that you stop missing the evidence of problems that vanish before anyone can investigate.

Metrics show that load exists. Traces and audit records show who creates it and why.

Audit records

The interface server can record each request: who sent it, what they asked for, when, and what the response was. At a modest detail level, this creates a searchable history of activity. Analyzing it answers questions metrics cannot, such as which service account made the most list requests last night, which tool updated the same object ten thousand times, or which user deleted a particular resource.

Distributed tracing

Tracing follows a single request across components and records how much time was spent in each stage. The interface server and the database can both emit traces, which lets you see, for one slow request, whether the time went to authentication, admission webhooks, or the database write. Because tracing is more expensive than metrics, it is usually sampled: only a small fraction of requests are traced, which is enough to reveal patterns without adding much load.

flowchart LR
  M["Metrics: something is slow"] --> A["Audit records: who is sending the heavy requests"]
  A --> T["Traces: which stage of those requests is slow"]
  T --> P["Profiles: which code is responsible"]
  P --> F["Fix at the source"]
        

Waiting for a real traffic spike to discover your limits is an expensive way to learn. Controlled tests are cheaper.

Why test the control plane on purpose

Metrics from a healthy production cluster tell you how the control plane behaves under ordinary load, but they cannot tell you where the cliff is. Load testing pushes the system deliberately, in a safe environment, to reveal the point where latency begins to climb and the component that gives way first. Knowing your cliff in advance lets you set alerts at sensible distances from it and plan capacity before demand arrives.

What a good test looks like

A useful test imitates real behavior instead of simply hammering one endpoint. It creates many objects of realistic shapes and sizes, runs many watchers, mixes reads and writes in realistic ratios, and exercises the scheduler and controllers by creating real workloads. The Kubernetes community maintains dedicated testing tools for this purpose, and other open-source projects specialize in creating large numbers of objects and measuring how the control plane responds. The best of them measure the same service level indicators you use in production, so results are directly comparable.

Simulated nodes and their limits

Creating thousands of real machines for a test is costly, so some tools simulate nodes: lightweight programs that pretend to be node agents and report status to the control plane. They are excellent for measuring control plane load from node traffic. They cannot reproduce everything real nodes do, such as actually running workloads, so use them to test the control plane and not the nodes.

Good engineers do not guess. They follow a method that narrows the search with every step.

Seven steps from symptom to fix

flowchart TD
  S["Symptom reported"] --> A{"Did it start with a change?"}
  A -->|"Yes"| B["Review recent changes and consider rollback"]
  A -->|"No, gradual"| C["Check growth in objects, nodes, and clients"]
  B --> D["Locate slow stage"]
  C --> D
  D --> E{"Which stage?"}
  E -->|"Front door or admission"| F["Check request mix, webhooks, fairness queues"]
  E -->|"Database"| G["Check disk latency, size, leader changes"]
  E -->|"Scheduler or controllers"| H["Check queues, client limits, placement rules"]
  F --> I["Fix source, then verify"]
  G --> I
  H --> I
        

Two helpful lenses

Two simple frameworks organize this thinking. The first examines each resource by its utilization, its saturation, and its errors. The second examines each service by its request rate, its error rate, and its duration. Used together, they ensure you look at both the machines and the work they serve, and they prevent the common trap of staring at only one kind of chart.

Patterns repeat across clusters. This table maps common symptoms to the places you should look first.

Kubernetes publishes tested limits. Knowing them tells you how far you can push, and when to plan ahead.

Published scalability thresholds

The Kubernetes community tests and documents the largest clusters it considers supported, expressed as upper bounds on things like the number of nodes, the number of pods in total, the number of pods per node, and the number of services. These limits describe configurations the project continuously tests against its published service level objectives. Exceeding them does not mean failure, but it means you are in territory that is less thoroughly tested and where you must rely more on your own measurements. Check the current documentation for the exact numbers of the version you run, because they are revisited as the software improves.

Dimensions of scale

Sizing the machines

Control plane machine sizes are usually chosen by how many nodes the cluster has and how busy it is. The interface server mostly needs processor and memory in proportion to the number of objects and clients. The database needs fast storage first, then adequate memory for its data file. Rather than picking a size from a generic chart, observe the actual utilization of existing machines during peak periods and keep comfortable headroom, since control plane problems tend to appear as sudden cliffs rather than gentle slopes.

Running several copies of each component keeps the cluster alive during failures. It also changes how performance behaves.

Different components, different redundancy

The interface server is stateless, so you can run several copies and spread requests across them with a load balancer. All copies are active at the same time. The scheduler and controller manager are different: only one copy of each does the work at any moment, because two schedulers choosing placements independently would conflict. The copies use a leader election process, where each competes to hold a lease stored in the cluster, and only the holder acts. If the leader fails or loses its lease, another takes over.

flowchart TD
  LB["Load balancer"] --> A1["API server one"]
  LB --> A2["API server two"]
  LB --> A3["API server three"]
  A1 --> E[("etcd group: three or five members")]
  A2 --> E
  A3 --> E
  S1["Scheduler one: leader"] --> A1
  S2["Scheduler two: standby"] -.-> A2
  C1["Controller manager one: leader"] --> A1
  C2["Controller manager two: standby"] -.-> A3
        

Performance consequences of leader election

Uneven traffic across interface server copies

Because clients keep long-lived connections, traffic can become lopsided. After a rolling restart, the first instance restarted may end up with few connections while the last ones carry many. Watch per-instance request rates and memory, and consider settings that gently recycle connections so load spreads out again.

Many teams use a cloud provider’s managed Kubernetes. Profiling still matters, but the levers are different.

What changes with a managed service

In a managed service, the provider runs and scales the control plane. You usually cannot log into its machines, change its low-level settings, capture code-level profiles, or choose its disk types. The provider handles patching, failover, and often automatic scaling of capacity. That removes a great deal of work, but it also removes some of your visibility, and it does not remove your responsibility for how your own workloads and tools use the control plane.

What you can still profile

Charts nobody understands and alerts nobody trusts are worse than none. Design them for the person using them at three in the morning.

A dashboard that tells a story

The best control plane dashboard reads from top to bottom like a short story. The top row shows overall health against your objectives: is the front door meeting its latency targets, and are errors low? The next rows break the story down by component: interface server, fairness queues, database, scheduler, and controllers. Lower rows show resource usage, and a final row lists the heaviest clients. A person arriving during an incident can start at the top, see whether the objective is violated, and work downward to find the guilty component, without needing to remember dozens of metric names.

Alerting on symptoms and on early warnings

Alerts fall into two useful groups. Symptom alerts fire when users are affected, such as when write latency violates its objective for a sustained period. These should wake someone up. Early-warning alerts fire when something is trending toward trouble, such as database size passing a comfortable fraction of its quota or disk sync times creeping upward. These should usually create a ticket for working hours, because they give you time to act calmly.

Burn rate: alerting on how fast you spend your allowance

If your objective allows a small share of slow requests each month, that share is an error budget, like a monthly allowance. Burn rate measures how fast you are spending it. A fast burn, which would exhaust the whole budget in a day or two, deserves immediate attention. A slow burn, which would exhaust it in weeks, deserves a ticket. Alerting this way reduces false alarms from brief blips while still catching real, sustained problems quickly.

Every improvement costs something somewhere else. Wise tuning means understanding both sides before changing anything.

A rule of thumb for choosing levers

Prefer changes that reduce load over changes that increase capacity, and prefer reversible changes over irreversible ones. Removing a wasteful client, narrowing a webhook, or cleaning up old objects reduces demand permanently and usually costs nothing. Adding capacity helps only until demand catches up, and it costs money every month. When you must add capacity, do it after you have removed the waste, so you are not paying to serve work nobody needed.

Most control plane crises are not sudden surprises. They are the final result of habits that were allowed to continue.

These illustrative stories show how the ideas above combine in practice. Each one follows the same path: symptom, investigation, cause, and fix.

You do not need advanced tooling on day one. Progress in stages, and gain value at each step.

Numbers worth reviewing every month

The most common moment for performance to change is the moment you change something.

Why changes cause regressions

Every new Kubernetes release brings improvements, but it can also change defaults, add features that generate extra requests, or alter caching behavior. Third-party components that work well on one version can behave differently on the next. Even small configuration edits, such as changing a webhook’s scope or a controller’s number of workers, shift load patterns. A disciplined approach treats performance as a property that can regress, like any other quality of the system.

Follow one realistic investigation to see the method in action. The details are illustrative, but the reasoning is typical.

The report

On a Monday morning, several teams complain that their deployments take far longer than usual. Some report that command-line tools pause for many seconds before showing results. Nothing is down, applications are running, and the on-call engineer sees no alarms. The engineer begins by stating the symptom precisely: write requests and list requests at the interface server are slow, starting sometime over the weekend.

Locating the slow stage

The engineer opens the control plane dashboard. The top row shows the write latency objective violated since Saturday evening. In the second row, write latency at the interface server is high, but the database’s own request durations look normal, and its disk sync times are healthy. That rules out storage. Admission step durations show one webhook taking much longer than before, but not enough to explain everything. In-flight requests are high and the fairness queue for general clients shows long waits and a growing count of rejected requests.

Finding the source

Audit records for the weekend reveal that a new reporting job, started Saturday, makes frequent cluster-wide list requests for pods and configuration objects, with no filters. Its request count is not extreme, but its response sizes are enormous. Memory use of the interface server instances has climbed in step. The job belongs to a data analytics team that did not realize its tool lists everything every thirty seconds.

sequenceDiagram
  participant J as Reporting job
  participant A as API server
  participant E as etcd
  participant O as Other clients
  J->>A: List all pods and configuration objects, no filters
  A->>E: Fetch large result set
  E-->>A: Very large response
  A-->>J: Build and send huge reply
  O->>A: Normal requests
  Note over A: Memory and processor busy with huge replies
  A-->>O: Slow responses and rejections
        

Confirming and fixing

The engineer asks the analytics team to pause the job, and within minutes list latency falls, memory stabilizes, and rejections stop. That confirms the hypothesis. The permanent fix has three parts. The analytics team changes the tool to watch for changes and filter by label, instead of listing everything. The platform team assigns that tool’s service account to a lower-priority fairness category, so future misbehavior is contained. And the team adds an alert on unusually large response sizes and a short guide for teams on efficient use of the cluster interface.

Keep these short definitions nearby while reading dashboards, audit records, and incident reports.

Quick answers to the questions beginners ask most often.

Compare request latency at the interface server with the symptoms. If creating, listing, or updating objects is slow, suspect the control plane. If objects are created quickly but your application responds slowly to users, the control plane is probably not involved, because running applications do not depend on it for each request.

Existing workloads generally keep running, since node agents and networking continue to function with the last known state. What stops is the ability to change things: deploy, scale, replace failed pods, or update configuration. This is why control plane trouble can go unnoticed until you try to make a change.

Start with the ninety-ninth percentile latency of requests at the interface server, split by action. It reflects what clients feel and points you toward either the front door or the database for deeper investigation.

Averages hide the slow tail. A small share of very slow requests barely moves the average but causes timeouts and retries that users notice. Always look at high percentiles.

Gentle, short profiling by authorized administrators is commonly done in production, but each capture adds some load. Avoid taking many large profiles at once from a component already under stress, restrict access to the endpoints, and prefer sampled or continuous low-rate profiling where possible.

Slow storage under the database is the most frequent cause, followed by slow admission webhooks and oversized or constantly updated objects. Checking disk sync times and webhook durations quickly separates these possibilities.

Typically a few object types: high-volume events, large configuration objects, leftover finished jobs, or custom resources written by operators. Counting objects and total size by type finds the main contributors, and cleanup plus compaction and defragmentation reclaim the space.

Only after you understand the load. Larger machines can help genuine growth, but if a wasteful client or slow webhook is the cause, extra capacity just delays the problem while raising cost. Profile first, remove waste, then add capacity for demand that remains.

Monitoring watches known numbers over time and alerts when they cross thresholds. Profiling digs into why a number behaves as it does, using audit records, traces, and code-level samples. Monitoring raises the question; profiling answers it.

Partly. You can usually collect the metrics and audit logs the provider exposes and fully profile your own clients, webhooks, and workloads, but you typically cannot capture code-level profiles of the provider’s components. Bring measured evidence to the provider when deeper investigation is needed.

Before significant changes such as upgrades or new platform components, before major launches that will change load, and periodically as the cluster grows. Treat the results as a recorded baseline for later comparison.

They may be waiting on their own client-side rate limits, or their work queues may be backed up by expensive processing. Compare client-side wait times and queue metrics with server latency to tell these cases apart.

There is no single answer. Fewer, larger clusters share resources efficiently but have larger blast radius and scaling limits; several moderate clusters limit the blast radius at the cost of more management effort. Choose based on your scale, isolation needs, and operational capacity.

The Kubernetes control plane is a small group of cooperating components organized around one hub, the interface server, backed by one durable store, etcd. Because everything flows through those two places, almost every performance problem eventually appears there, even when the true cause lives in a heavy client, a slow webhook, a slow disk, or a controller waiting on its own limits. Profiling means measuring where time and resources go, moving from cheap broad signals to expensive precise ones, and following the evidence to the source instead of guessing. Used together with clear objectives, fair queuing, careful extensions, regular testing, and disciplined change management, profiling turns the control plane from an invisible risk into a well-understood, predictable part of your platform.