Caching in Kubernetes is not a performance trick added at the end. The whole design depends on it from the beginning.

Kubernetes in sixty seconds

A cluster is a group of computers working together, and each computer is a node. The brain of the cluster is the control plane. At its centre sits the API server (the program named kube-apiserver), which is the front door for every request. Behind it sits etcd, a small, strongly consistent database that stores the official record of every object: every Pod, Service, Secret and configuration.

Kubernetes works by comparing “what the record says should exist” with “what actually exists,” and fixing any difference. The programs that do this fixing are called controllers. The scheduler, which picks nodes for Pods, is one. The kubelet, the agent on each node that starts containers, is another. Each of these programs needs to know about changes the moment they happen, so it does not just ask once. It keeps listening.

The crowd at the front door

Count how many programs are listening in a medium cluster of 1,000 nodes. Every kubelet watches the Pods assigned to its node. Every kube-proxy watches Services and their endpoints. The scheduler watches Pods and nodes. Dozens of built-in controllers watch their own kinds of objects. Add monitoring agents, ingress controllers, service meshes, operators and CI tools. Easily tens of thousands of long-lived connections are open at once, and each one wants to hear about every relevant change within a second or two.

If each of those listeners caused its own query against etcd, the database would drown. etcd is built for correctness, not for being asked the same question ten thousand times a second. It keeps strict order on every write and agrees with its peers before answering. That is exactly what you want for a source of truth, and exactly what you do not want for a crowd.

The solution in one sentence

The API server opens one watch on etcd for each kind of object, keeps a full in-memory copy of those objects plus a short history of recent changes, and serves all clients from that memory. Ten thousand listeners cost etcd the same as one.

A dozen words appear again and again in this topic. Learn them once and the rest reads easily.

List and watch: the two basic verbs

Almost everything a controller does starts with two verbs. List returns a full snapshot, such as “all Pods in this namespace.” Watch returns a stream: it keeps the connection open and sends a small event whenever something changes. A smart client lists once to get the starting picture, then watches from that point on, so it never has to list again unless something breaks. This simple pairing is the heart of Kubernetes, and the cache exists to make it cheap.

resourceVersion: the clock of the cluster

Every object carries a resourceVersion, a text value the server sets whenever the object changes. It is closely tied to the internal revision counter of etcd, which ticks upward for every write anywhere in the database. Lists carry one too, meaning “this snapshot is as of this version.” Watches use it to say “start streaming from after this version.” It works like a bookmark in a long book, letting a client resume exactly where it stopped.

The watch cache is not a separate server. It lives inside every API server process, between the web endpoints and etcd.

flowchart LR
  ETCD["etcd: durable source of truth"] -->|one watch per resource type| REF["Reflector inside the API server"]
  REF --> ST["In memory store: latest copy of every object"]
  REF --> RB["Event window: ring buffer of recent changes"]
  ST --> LS["Serve LIST from memory"]
  RB --> WS["Fan out WATCH events to many clients"]
  LS --> C1["kubectl and controllers"]
  WS --> C2["kubelets, scheduler, controllers"]
    

Read the diagram from left to right. At the start sits etcd. For every kind of object, such as Pods or Services, the API server runs a reflector that lists everything once from etcd and then keeps one watch open. Whatever arrives is written into two places. The first is the store, an in-memory table holding the latest version of every object of that kind. The second is the event window, a fixed-size ring of the most recent changes. Clients never talk to etcd for these requests. Lists are answered from the store, and watches are fed from the event window and from live events.

One cache per API server

A production cluster runs several API servers for reliability, usually three or more, behind a load balancer. Each API server has its own watch cache, filled from the same etcd. The caches are not shared and not synchronised with each other. They all converge to the same truth, but at any instant one may be a fraction of a second behind another. This fact explains several surprising behaviours later in the guide.

What gets cached, and what does not

The cache holds the objects that clients list and watch through the main API: built-in kinds and custom resources. It does not cache the results of arbitrary requests, and it does not change write behaviour. Every create, update and delete goes straight to etcd, which remains the only place where the truth is decided. The cache hears about the result a moment later through its watch.

Let us follow one change, from the moment it is written to the moment a distant client hears about it.

The ring buffer and why it is short

The event window holds only recent changes, not the full history. This is deliberate. Memory is limited, and a client that has been away for a long time is better off taking a fresh snapshot than replaying thousands of old events. The window starts small and grows when changes arrive quickly, up to a large upper limit, so busy resource types such as Pods get a longer memory than quiet ones such as Namespaces. Self-managed clusters can also set sizes by hand.

Think of the ring buffer as the pinned change list on the library desk. It holds the last few dozen or hundred edits. If a visitor says “I last saw version 4000,” and the oldest line on the list is version 4020, the librarian cannot tell them what happened between 4000 and 4020. They must take a new copy of the whole record.

Slow listeners are cut off

Each watcher has a small buffer for events waiting to be sent. If a client is too slow to read its connection, perhaps because it is busy or the network is poor, its buffer fills up. The cache waits briefly, then closes that watch rather than letting one slow client hold up everyone else. The client sees its connection drop and must reconnect. This is a feature: it protects the many from the few.

Watches do not last forever

Every watch has a time limit chosen randomly in a window of roughly half an hour to an hour by default. When the limit expires, the server ends the watch, and the client reconnects and resumes from its last version. The randomness is important. If all watches expired at the same moment, every client in the cluster would reconnect at once, causing a stampede. Spreading the expiry times smooths the load.

Starting up

When an API server starts, its caches are empty. Each reflector must list its resource from etcd before the cache can serve anyone. For a few seconds, requests for a still-filling cache may be slow, or may get a “try again soon” reply (HTTP 429). The server’s readiness check stays unhealthy until its caches are ready, so a load balancer should not send traffic to it too early.

The same list request can be answered from etcd or from the cache. A single parameter decides which.

Every list or get request may include a resourceVersion setting, together with an optional resourceVersionMatch setting. Together they say how fresh the answer must be. The server uses them to decide where to read from. Understanding this table is one of the most valuable things you can learn about the API server, because it explains both great performance and nasty surprises.

flowchart TD
  R["LIST request arrives"] --> Q1{"resourceVersion given?"}
  Q1 -->|"zero"| A["Served from the watch cache, any age"]
  Q1 -->|"not set"| B["Needs the latest state"]
  B --> Q2{"Consistent read from cache enabled?"}
  Q2 -->|yes| C["Check latest etcd revision, wait for the cache, serve from cache"]
  Q2 -->|no| D["Quorum read from etcd"]
  Q1 -->|"specific number"| Q3{"Match mode"}
  Q3 -->|"NotOlderThan"| E["Cache serves once it has caught up"]
  Q3 -->|"Exact"| F["Served from etcd"]
    

What this means in practice

A request with no version is the safest-looking choice, because it promises the latest data. But it is also the most expensive, because in the traditional design it forces a trip to etcd, which then has to read and return every object. A request with version zero is cheap, because the cache already has the answer in memory. For a client that is about to start watching, slightly stale data is fine: the watch will deliver any change that happened since the snapshot, so the client catches up within moments.

That is exactly why well-written controllers ask for version zero on their first list. A thousand controllers starting together after a restart cost almost nothing to the cache. A thousand controllers asking for the latest from etcd could knock the database over.

Pagination and the cache

Large lists can be requested in pages using limit and a continue token. This keeps memory use low because the server returns, say, 500 objects at a time. Traditionally, however, a list with version zero ignores the limit and returns everything from the cache in one piece, while a paged list is answered from etcd. Also, a continue token points into a consistent view in etcd, and because etcd compacts old versions every few minutes by default, a slow client may find its token expired and receive a 410 error. That is a signal to restart the listing, not a bug.

Watch has its own rules

A good client lives in a loop: list, watch, reconnect, and only occasionally start over. Here is that loop in detail.

flowchart TD
  S["Informer starts"] --> L["LIST: get full state and its resourceVersion"]
  L --> W["WATCH from that resourceVersion"]
  W -->|events arrive| U["Update local cache, call handlers"]
  U --> W
  W -->|connection drops| W2["Resume watch from last seen version"]
  W2 -->|version still in window| W
  W2 -->|410 Gone: too old| L
    

Follow the diagram from the top. The informer starts by listing the full state and noting the list’s version. It then opens a watch from exactly that version, so no change falls into the gap between the two calls. Events arrive and update the local copy. If the connection drops, whether because of a timeout, a network blip or an API server restart, the informer reconnects and asks to resume from the last version it saw. If that version is still inside the server’s event window, the stream picks up as if nothing happened. If it is too old, the server answers 410 Gone, and the informer returns to the top and lists again.

Bookmarks: keeping the marker fresh

Imagine you are watching a quiet resource where nothing changes for hours. Your saved version stays old, while the rest of the cluster moves on and the event window slides forward. When your watch finally drops, your marker is far behind, the server answers 410 Gone, and you must perform a full, expensive list for no good reason.

Bookmarks fix this. A client that asks for them periodically receives a tiny event containing nothing but the newest version. The informer updates its saved marker, so a later reconnect starts from a recent point and nearly always lands inside the window. Bookmarks cost almost nothing, but they save a great deal of unnecessary full lists in large clusters.

The pieces of a client informer

Two caches, one story

It helps to see that there are two layers. The API server’s watch cache protects etcd from clients. The client’s informer cache protects the API server from the program’s own code. A controller that reads a thousand times a second does so from its local memory, and only the one watch connection touches the API server. Together the two layers turn a potentially huge amount of query traffic into one steady stream of changes.

The basic design has served Kubernetes for years, but two newer ideas close its biggest gaps. Check which versions of your cluster support them.

Consistent reads served from the cache

Recall the awkward choice from Chapter 5. A request for “the latest state” had to go to etcd, which is slow and heavy, while a request for “any state” came from the cache but could be stale. Newer Kubernetes releases offer a clever middle path, usually described as consistent reads from the cache.

It works like this. When a request asks for the latest data, the API server asks etcd only one small question: “what is your newest revision number right now?” That is a tiny, cheap query compared with reading all objects. The API server then waits until its own watch cache has caught up to that revision, which normally takes a few milliseconds, and answers the full list from memory. The result has the same freshness guarantee as a direct etcd read, but the heavy lifting is done by the cache. Large lists that used to load etcd now load only the API server’s memory.

Streaming lists

A normal list request builds the entire response in memory before sending the first byte. For a very large resource, such as a hundred thousand Pods, this means the API server must hold a gigantic temporary buffer, and a handful of simultaneous lists can push it out of memory. After a restart, when every client relists at once, the spike can be dramatic.

The newer approach, often called streaming list or watch-list, replaces the big list with a special watch. The client opens a watch and asks the server to send the initial state as a series of “added” events, followed by a marker event saying “the initial state is complete.” The server sends objects one by one, straight from the cache, without building a huge buffer. Memory use stays flat, and clients can begin processing objects as soon as the first ones arrive.

The watch cache gets the headlines, but a request passes through several other small caches, each saving work in its own corner.

flowchart LR
  CL["Client side: discovery cache, informer cache"] --> AU["API server: authentication token cache"]
  AU --> AZ["Authorization decision cache"]
  AZ --> AD["Admission and schema validators, compiled once"]
  AD --> WC["Watch cache and shared serialization"]
  WC --> ET["etcd"]
    

Look at the diagram from left to right. Each layer exists because the step it protects is repeated thousands of times. Asking a remote identity service “who is this token?” for every request would be wasteful, so the answer is remembered briefly. Compiling a validation schema for every write would be wasteful, so it is compiled once and reused. The farther right a cache sits, the closer it is to the data, and the more it matters for scale.

Why the short times matter for security

A cache that remembers “allowed” for five minutes means that if you remove someone’s access, they may still get through for up to five minutes, but only when the decision came from an authorization webhook. The built-in role-based access control (RBAC) checker runs in the API server using its own up-to-date in-memory view of roles, so it reacts almost at once. Know which authorizer your cluster uses before you promise anyone “instant revocation.”

The cache-bypassing request: write and delete

No cache sits in front of writes. Every create, update, patch and delete is validated, passes through admission, and is committed to etcd immediately. This is what keeps the cluster’s truth in one place. The only subtlety is that after your write returns, your next read from a cache may not yet show it, because the cache learns of the change through its own watch a moment later. The API server’s reply to a write includes the new object and its version, so a client that needs to see its own change can use that reply.

If you write or run controllers, your own program has a cache, and it behaves differently from what many people expect.

Reading from your own memory

Frameworks that help people write controllers and operators, such as controller-runtime, give you a client whose reads come from the local informer cache by default and whose writes go to the API server. This is wonderful for performance: a controller can read the objects it cares about thousands of times per second without sending a single request. It also hides a trap.

The read-after-write trap

The controller was not wrong. It simply read from a cache that is eventually consistent, meaning it will become correct soon, but not instantly. This is the same kind of lag you see between regional depots in the newspaper analogy.

How careful programs deal with it

The cost of a cache you never wanted

A controller that reads Secrets or ConfigMaps through its cache will, by default, cache all of them cluster-wide, even if it needs three. In a large cluster that can mean hundreds of megabytes of memory and a heavy initial list. The cure is to restrict what the informer watches using namespaces, labels or fields, or to fetch the few objects directly when needed.

Everything above rests on how etcd stores history. A little knowledge here explains most of the strange errors you will meet.

A database that keeps versions

etcd is a key-value database with a special property: it never overwrites a value in place. Instead, every change creates a new revision, a global counter that goes up by one for each write anywhere in the database. The older versions stay readable for a while. This design is called multi-version concurrency control, or MVCC, and it is what lets a client say “show me the state as of revision 4000,” and lets a watch say “tell me everything after revision 4000.”

The resourceVersion on a Kubernetes object is derived from these revisions. That is why a watch can resume reliably: the position in the stream is simply a place in etcd’s ordered history.

Compaction: forgetting on purpose

If etcd kept every revision forever, the database would grow without end. So old revisions are periodically removed in an operation called compaction. In a standard Kubernetes setup, the API server asks etcd to compact about every five minutes. After compaction, requests for versions older than the compaction point fail with an error that the server turns into 410 Gone. This is precisely why a paged list can expire if the client is too slow, and why the cache’s event window is short.

Compaction marks space as reusable but does not shrink the file on disk. Operators occasionally run defragmentation to give that space back. etcd also has a configured size limit; if the database reaches it, etcd stops accepting writes until space is freed, which makes the whole cluster read-only. Watching database size is a basic operational duty.

The cost of large objects and frequent updates

Because every update writes a whole new version, large objects that change often create heavy traffic. A 1 megabyte object updated ten times a minute produces a steady stream of megabytes through etcd, through the watch cache, and out to every watcher. This is why the community pushes for small objects, and why splitting data into several smaller objects is often better than one giant one.

Events deserve their own storage

The Kubernetes Event objects, which record things like “Pod scheduled” and “image pulled,” are numerous and short-lived. In large self-managed clusters, operators often store them in a separate etcd so that their churn does not slow down the main database. Event objects also expire automatically after a time-to-live, one hour by default.

Let us watch a single new Pod travel through the system and see which cache each component reads from.

Notice the pattern. Components almost never talk to each other directly. Each one writes its result to the API server and reads what others wrote by watching. The watch cache is the shared notice board that makes this whole conversation possible, and it is why the system can add components without changing existing ones.

Who watches what

Making the traffic smaller

The watch cache makes large clusters possible, but it is not free. Here is where its costs live and how to keep them small.

Memory is the main price

The watch cache keeps a copy of every cached object in the API server’s memory. The memory needed is roughly the total size of all objects of the cached kinds, plus the event window, plus working space for serving requests. A cluster with a hundred thousand Pods, each a few kilobytes, already needs gigabytes. Because every API server keeps its own cache, the cost is multiplied by the number of API servers.

This is why the size and number of your objects matters so much. A Secret that holds a large file, a ConfigMap that stores a whole data set, or a custom resource that records thousands of status entries each consumes memory on every API server, and every update is pushed to every watcher.

The three multipliers of trouble

Protecting the server from clients

The API server also limits how much any one client can ask. A mechanism called API Priority and Fairness sorts incoming requests into queues by importance and by who sent them, so a misbehaving program cannot starve the kubelets or the scheduler. A list that is cheap thanks to the cache uses less of its quota than one forcing a trip to etcd. When your program is told to slow down (HTTP 429), it should back off instead of retrying immediately.

Tuning options

A reliable cluster runs several API servers. Each has its own cache, and that creates some subtle behaviours worth understanding.

Caches that are close, not identical

All API servers read from the same etcd, so they agree on the truth. But each learns about changes through its own watch, with a delay that is normally a few milliseconds and occasionally longer. During that delay, API server A may know about a change that API server B has not yet received. A client that reads from A and then from B can see the cluster appear to move backwards in time. This is sometimes called a time travel problem.

How clients protect themselves

Well-written client libraries remember the newest version they have seen. When they relist, they ask for a snapshot “not older than” that version. A lagging API server then waits until its cache has caught up, instead of returning stale data that would make the client go backwards. This is one reason to keep client libraries current, and to avoid hand-written loops that ignore resourceVersion.

The restart stampede

When an API server restarts, or a load balancer shifts traffic, thousands of watches drop at nearly the same instant. Every client reconnects and tries to resume. Those whose saved versions are still inside the event window resume cheaply. Those who are too far behind must relist, and if they list with the latest-state setting, the combined load on etcd can be severe. Bookmarks, version zero lists, random watch timeouts and streaming lists all exist to soften this moment.

Upgrades and the human factor

During a control-plane upgrade, API servers briefly run different versions, and so may differ in which caching features they offer. A good rolling upgrade restarts servers one at a time and waits for each to become ready before moving on. Resist the urge to restart them all at once to save time. The brief outage of the whole control plane, followed by a stampede of every client in the cluster, is far worse than a slow upgrade.

A cache is a second copy of your data, and copies deserve the same protection as originals.

For a healthy security posture, give each program its own identity, with the smallest roles that let it do its job. Review which accounts hold cluster-wide list or watch rights on Secrets, ConfigMaps and nodes. And keep control-plane access separate from ordinary user access, because a person who can read the API server’s memory can read everything the cache holds.

The API server tells you how its caches are doing, if you know which numbers to read.

Metrics worth graphing

Symptoms and likely causes

A short troubleshooting routine

The lessons of years of large clusters, boiled down into habits you can adopt on day one.

Patterns worth copying

Traps to avoid

Best practices checklist

Common mistakes and quick fixes

The same few ideas show up in every big cluster. Here is where you can see them in the tools you already use.

A typical journey at a growing company

A team begins with a small cluster and a few scripts that run kubectl get in loops. As the cluster grows, the API server slows during busy hours. An engineer notices the scripts are the top callers and replaces them with a controller using a shared informer. Later, an operator is added that caches every Secret, and memory use jumps, so its scope is narrowed to a few namespaces. After a control-plane upgrade, a restart stampede causes a brief slowdown, so the team starts upgrading one server at a time and adds readiness checks. Finally they graph watch counts and object sizes monthly, and clean up old Jobs and Events. The cluster is now ten times larger, and the control plane is calmer than before.

The Kubernetes API server is the front door of the cluster, and etcd is the vault behind it. The watch cache lets thousands of clients read and listen without ever crowding the vault: one watch per resource type feeds an in-memory store and a short window of recent events, and every client is served from memory. The resourceVersion marker decides where a read is answered and how a watch resumes. Clients add their own informer caches on top, and a handful of smaller caches handle authentication, permissions and serialization. Used well, this design lets a cluster reach thousands of nodes. Used carelessly, it produces memory pressure, stale reads and stampedes.