Before we meet informers, we need to feel the pain they were invented to remove.

What polling means

Polling means asking the same question again and again on a timer. If a program wants to know which Pods exist, it can call the API server, receive the full list, compare it with the previous list, and repeat a few seconds later. This is simple to build, which is why it is the first idea almost every beginner reaches for. The trouble appears when the cluster grows or when many programs do the same thing at once.

The three costs of polling

What Kubernetes needed instead

Kubernetes is built around many small programs called controllers. Each one keeps looking at the cluster and nudges reality toward what you asked for. A typical cluster runs dozens of built-in controllers, the scheduler, a kubelet on every node, a kube-proxy on every node, plus any operators you install. All of them need an up-to-date picture of objects. The design goal was therefore clear: let every component hold a fresh local copy of the objects it cares about, and only send small updates when something changes. That goal produced two cooperating pieces. The server side offers a streaming watch. The client side offers an informer that consumes the stream and maintains the local copy.

A short tour of the parts that talk to each other, so the rest of the post has solid ground.

The cast of characters

The API server is the front door of Kubernetes. Every read and every write goes through it, whether from kubectl, the scheduler, a kubelet or your own operator. Behind it sits etcd, a reliable key-value database that stores the entire cluster state. Around it live controllers, small loops that compare desired state, which is what you declared, with actual state, which is what currently exists, and then act to close the gap.

Why the hub-and-spoke shape matters

Because all communication flows through the API server, components never talk to each other directly. The scheduler does not tell the kubelet to start a Pod. It writes a decision into the Pod object, and the kubelet, which is watching Pods assigned to its node, sees that change and acts. This indirection makes the system loosely coupled and easy to extend, but it also means the API server becomes the busiest crossroads in the cluster. Anything that reduces unnecessary traffic through that crossroads is a big win, and that is the whole job of the watch and informer design.

flowchart LR
  U["kubectl and users"] --> A["API Server"]
  A  E[("etcd")]
  A -- "watch stream" --> S["Scheduler"]
  A -- "watch stream" --> C["Controller Manager"]
  A -- "watch stream" --> K["Kubelet on each node"]
  A -- "watch stream" --> O["Your Operator"]
        

Figure 1 shows the shape. Notice that arrows from the API server to each component are streams, not repeated questions. Each component opens one long-lived connection per kind of object it cares about and then simply receives news as it happens. The sections ahead explain what travels on that stream and how each client turns it into something dependable.

Almost every Kubernetes client follows the same two-step dance: list once, then watch forever.

Step one: list

A list call returns every object of a given kind, optionally filtered by namespace, label or field. It is a snapshot, a photo of the world at one instant. The response also carries a bookmark of that instant, which we will meet shortly as the resourceVersion.

Step two: watch

A watch call tells the API server, “Keep this connection open and tell me about every change after this point.” The server responds with a continuous stream of small messages, one per change. Under the hood this is an ordinary HTTP request whose response never ends, with each event delivered as a separate chunk. Modern clusters carry this over HTTP/2, which lets many streams share a single connection to the server.

Why both steps are needed

A watch alone only reports changes. If you start watching with no starting point you would not know what already exists. A list alone only gives a snapshot that goes stale at once. Put them together and you get a complete, always-fresh picture: the list provides the base, and the watch applies every difference on top of it. This pattern is so central that the Kubernetes client library wraps it in a dedicated component, which later chapters call the reflector.

The single most important idea for understanding why watches are reliable.

What a resource version is

Every object in Kubernetes carries a field called resourceVersion. It is a string that the API server fills in, and for objects stored in etcd it is derived from etcd’s global revision counter. Each time anything in etcd changes, the counter moves forward. The object that changed receives the new number as its resource version. So a resource version is essentially a timestamp measured in changes rather than in seconds.

The golden rule: treat it as opaque

Clients must treat the value as an opaque token. You may pass it back to the server, but you must never do arithmetic on it or assume it is numeric on every storage backend. The one operation that is officially supported is equality, which is used for concurrency control. When you update an object and send back the resource version you originally read, the server accepts the write only if the object has not changed since. This is called optimistic concurrency, and it prevents two controllers from silently overwriting each other.

Two jobs, one field

List versus object versions

A list response has its own resource version in its metadata. It represents the point in time of the whole snapshot, and it is the value a client should hand to its follow-up watch. Individual objects inside the list have their own versions that reflect when each one last changed, and those can be much older. Mixing these up is a classic source of confusing bugs, so remember: the list’s version is the starting line for the watch.

Read semantics in one table

When a client lists, it can ask for different freshness guarantees by choosing how to set the resource version on the request. The table below summarises the common choices in plain language.

Informers deliberately choose the cheap, cache-served option on their first list and then rely on the watch to catch up to the present. That design is one of the main reasons large clusters survive controller restarts without the API server falling over.

What exactly travels down the connection, and how does the connection end?

The event types

Each message on the stream contains an event type and the full object that the event is about. There are only a handful of types, so they are easy to learn.

Ordering guarantees

For a single kind of object, events arrive in the order the changes happened in etcd. That means a client will never see an object’s deletion before its creation. Across different kinds there is no promise that a Pod event and a Service event appear in any particular relative order, because they come over different streams. Programs that depend on cross-object ordering must be designed to cope with seeing things in either order, which is one reason controllers are written to look at the current state rather than at event order.

How a stream ends

No stream lives forever. The API server deliberately closes watches after a randomised period, which is typically in the range of half an hour to an hour by default, so that connections are spread out over time and load balancers and restarts do not cause a thundering herd. Streams also end because of network blips, API server restarts, or rolling upgrades. A well-built client treats a closed stream as normal, waits briefly with a growing delay, and reconnects from the last version it saw.

The too-old problem

The server cannot remember history forever. etcd compacts old revisions regularly so it does not grow without bound, and the API server’s own cache holds only a limited window of recent events. If a client tries to resume from a version that has been forgotten, the server answers with a status that effectively means “that moment is gone.” The HTTP status is 410 Gone, and the reason shows up as an Expired or “too old resource version” message. The only safe response is to start over with a fresh list. We cover this recovery path in detail in Chapter 10.

sequenceDiagram
  participant C as Client
  participant S as API Server
  C->>S: List pods
  S-->>C: Snapshot plus version 100
  C->>S: Watch from version 100
  S-->>C: ADDED pod-a version 101
  S-->>C: MODIFIED pod-a version 105
  S-->>C: BOOKMARK version 110
  S-->>C: DELETED pod-a version 112
  Note over C,S: Stream closes after timeout
  C->>S: Watch from version 112
  S-->>C: Events continue
        

Read figure 2 from top to bottom. The client lists, notes version 100, and opens the watch from there. Changes arrive with growing version numbers. A bookmark confirms progress without any real change. When the stream closes, the client does not start from scratch, it simply resumes from version 112, the last number it saw. That tiny habit of remembering the last version is what makes reconnects cheap.

If every watcher talked to etcd directly, etcd would melt. The watch cache is the shield.

The fan-out problem

Imagine one thousand nodes, each running a kubelet that watches Pods. If each of those one thousand watchers created its own watch against etcd, the database would need to track one thousand separate streams for the same data and would repeat the same work for each. etcd is built for correctness and durability, not for serving thousands of identical readers cheaply.

One watch in, many watches out

The API server solves this with a component often called the cacher or watch cache. For each kind of resource, the API server opens a single watch against etcd and keeps an in-memory copy of all objects of that kind. It also keeps a rolling buffer of the most recent events. When a client opens a watch on the API server, the cacher registers it and feeds it from that shared buffer. One upstream stream becomes many downstream streams.

flowchart TD
  E[("etcd")] -- "one watch per resource kind" --> W["Watch cache in API server"]
  W --> B["Recent event buffer"]
  W --> M["In-memory object copy"]
  B --> C1["Watcher: kubelet 1"]
  B --> C2["Watcher: scheduler"]
  B --> C3["Watcher: operator"]
  M --> L["Cheap list requests"]
        

What the cache buys you

Slow watchers and back-pressure

Not every watcher keeps up. If a client reads its stream too slowly, the server has to buffer events for it, and it cannot buffer forever. When a watcher falls too far behind, the server terminates that watch rather than harm everyone else, and the client must reconnect and recover. This protects the shared cache but makes it important for clients to process events quickly. The right pattern, shown later, is to do almost no work inside the event receiver and hand heavy work to a separate queue.

An informer is not one thing. It is a small assembly line made of four cooperating parts.

The assembly line

A raw watch stream is not enough for a real program. Streams drop, events can be missed during restarts, and handlers need a tidy, consistent view of the world. The Kubernetes client library therefore builds an informer from several components, each with one clear job.

The reflector in detail

The reflector owns the list-and-watch conversation. On start it lists the objects, takes the resource version from the response, and replaces the contents of the queue with that snapshot. Then it opens a watch from that version and translates every incoming event into a queue entry. If the watch closes, it waits using a growing back-off delay and reopens it from the last version it handled. If the server says the version is too old, the reflector discards its resume point and begins again with a fresh list. The reflector deliberately knows nothing about what your program wants to do with objects. Its only duty is to keep the pipeline truthful.

The DeltaFIFO queue in detail

The queue stores work per object key, not per event. If an object changes three times before the consumer takes it, the queue keeps one entry for that key holding an ordered list of the three deltas. This grouping stops one noisy object from flooding the system with separate items and keeps the history per object in the right order. The queue also supports a special “replace” operation, used after a re-list, which lets it work out which objects vanished while the program was disconnected and generate the missing deletions.

The indexer and store in detail

The store is a thread-safe map from an object’s key, usually “namespace/name”, to the latest copy of the object. The indexer adds secondary lookup tables on top. By default there is an index by namespace, and you can add your own, for example “find all Pods on node X” or “find all objects owned by this parent.” Lookups are answered entirely from memory, so reading from the cache is typically microseconds rather than milliseconds, and costs the API server nothing.

flowchart LR
  API["API Server"] -- "list then watch" --> R["Reflector"]
  R -- "deltas" --> Q["DeltaFIFO"]
  Q -- "pop" --> L["Controller loop"]
  L -- "update" --> I[("Indexer and store")]
  L -- "notify" --> H["Event handlers"]
  H -- "enqueue key" --> W["Work queue"]
  I -- "read latest" --> K["Your reconcile logic"]
  W -- "next key" --> K
        

Why one program should never open two watches for the same kind of object.

The duplication trap

A large program such as the controller manager runs dozens of controllers, and many of them care about Pods. If each controller built its own informer, the program would open dozens of identical watches, store dozens of identical copies of every Pod, and burn memory and network for no benefit. The API server would also see a pile of redundant streams from one process.

The shared informer idea

A shared informer keeps one reflector, one queue and one cache per kind of object, and lets any number of consumers attach event handlers to it. Everyone reads from the same cache and receives the same notifications. A shared informer factory is the manager that hands out these informers: when two parts of the program ask for a Pod informer, the factory returns the very same instance both times and starts it only once.

Waiting for the cache to sync

When an informer starts, its cache is empty until the first list finishes. If your logic reads the cache too early, it will conclude that nothing exists and may take harmful actions, such as creating duplicates of things that are really there. Informers therefore expose a “has synced” signal that turns true once the initial snapshot has been fully loaded and delivered. The standard practice is to start the informers, wait for every one of them to report synced, and only then start the workers. Skipping this wait is one of the most common causes of controller bugs that appear only after a restart.

Scoping what you watch

Informers can be limited by namespace, label selector or field selector, which the server applies before sending data. A kubelet, for instance, only needs Pods assigned to its own node, so it asks the server for exactly that slice. Filtering at the server reduces network traffic, memory use and CPU on both sides. The rule of thumb is simple: never cache more than you need.

The informer tells you something changed. The work queue decides how and when you react.

The three handler callbacks

When you register with an informer, you provide up to three small functions: one called when an object is added, one when it is updated, and one when it is deleted. The update callback receives both the old and the new copy of the object, which lets you compare them and skip changes you do not care about. The delete callback sometimes receives a special wrapper called a tombstone instead of the object itself. This happens when the client missed the real delete event, for example during a disconnect, and only discovered the deletion later during a re-list. A careful handler checks for that wrapper before using the object.

Why handlers must be tiny

Handlers run on the informer’s own delivery path. If a handler does slow work, such as calling a remote service, every later notification waits behind it, and the server may eventually drop the lagging watch. The proven pattern is therefore to do almost nothing inside the handler except work out a short key for the object, typically “namespace/name”, and place that key into a separate work queue. Real processing happens in worker routines that pull keys from that queue.

What the work queue adds

Keys, not objects

A subtle but powerful design choice is that the work queue carries keys rather than whole objects. When a worker picks up a key, it looks up the freshest copy from the local cache at that moment. If five updates happened while the key waited, the worker simply sees the latest state and does one round of work. Carrying copies instead would risk acting on stale versions. This trick turns a bursty event stream into calm, bounded, always-current work.

Let us follow one real change from a user’s command to a controller’s reaction.

A day in the life of a change

Suppose you scale a Deployment from three replicas to five. The command reaches the API server, which validates it and writes the new desired state to etcd. etcd assigns a new revision. The API server’s watch on Deployments receives that change and stores it in the cache and event buffer. Then the fan-out begins: every watcher of Deployments gets the event, including the Deployment controller inside the controller manager.

sequenceDiagram
  participant U as User
  participant A as API Server
  participant E as etcd
  participant R as Reflector
  participant Q as DeltaFIFO
  participant I as Indexer
  participant H as Handler
  participant W as Work Queue
  participant X as Worker
  U->>A: Scale deployment to 5
  A->>E: Store new desired state
  E-->>A: Watch event with new revision
  A-->>R: MODIFIED deployment event
  R->>Q: Add delta
  Q->>I: Pop and update cache
  Q->>H: Notify update
  H->>W: Add key
  W->>X: Hand over key
  X->>I: Read latest deployment
  X->>A: Create missing pods
        

Step-by-step reading of the flow

In the controller process, the reflector receives the event and appends a delta to the queue. The informer’s loop pops it, updates the local store, and then calls the update handler. The handler adds the Deployment’s key to the work queue. A worker takes the key, reads the freshest Deployment from the cache, sees that five replicas are wanted and three exist, and asks the API server to create two more child objects. Those creations generate their own watch events, which travel through the same kind of pipeline in other controllers, for example the one that manages ReplicaSets and then the scheduler that picks nodes.

Typical timing

In a healthy cluster, the time from a database write to a handler call is usually a few milliseconds to tens of milliseconds. The dominant delays in real scenarios come later, such as image pulls or container start-up, not from the watch pipeline. That speed is exactly what polling could never deliver without punishing the server.

Networks fail, servers restart, history expires. Informers are built to heal themselves.

Three different healing actions

People often blur three separate ideas, so let us separate them carefully. They have different triggers and different costs.

Reconnect: the cheap path

Most interruptions end in a simple reconnect. The reflector remembers the last resource version it saw, waits briefly, and opens a new watch from there. The server replays any events it still has after that point from the watch cache, and the client catches up with no gaps and no duplicates. Because of bookmarks, even a watcher on a very quiet resource keeps a fresh resume point.

Relist: the safe path

If the reconnect gets a 410 Gone, the client must assume it missed an unknown number of changes. It lists again, and the queue’s replace operation compares the new snapshot against what the cache believed. Objects that disappeared in the meantime produce synthetic delete events, and those carry the tombstone wrapper mentioned earlier because the real delete was never seen. Objects that are new or changed produce add or update events. When it finishes, the cache is correct again, and the handlers have been told about every difference.

flowchart TD
  S["Watch stream closes"] --> R{"Can resume from last version?"}
  R -- "Yes" --> RC["Reconnect with back-off"]
  RC --> OK["Continue receiving events"]
  R -- "No, 410 Gone" --> RL["Relist all objects"]
  RL --> RP["Replace queue contents and compute differences"]
  RP --> SD["Emit synthetic deletes with tombstones"]
  SD --> OK
        

Resync: the second chance

A resync does not involve the network at all. At a configured interval, the informer replays every object in its local cache to the update handlers as if each had just changed. Why would you want that? Because a handler or worker might have failed on an object and never retried, or a bug might have left something half-done. A periodic replay gives your logic another look at everything. Many modern controllers set the interval to zero, meaning “never,” because the work queue’s retries already cover failures, and unnecessary resyncs waste CPU in big clusters. If you do use it, remember that the update handler will be called with identical old and new objects, so a handler that treats every update as a real change will do redundant work.

Back-off and jitter

When many clients lose their connection at the same instant, such as during an API server rolling upgrade, they must not all retry at the same moment. Informers wait with exponentially growing delays and add randomness, so reconnects spread over time. Without that discipline, the very recovery attempts would overload the server that just came back.

The mental model that decides whether your controller is robust or fragile.

Two ways to react to the world

An edge-triggered system reacts to the moment of change: “a Pod was just deleted, so do X.” A level-triggered system reacts to the current state: “there should be five Pods, I count three, so make two more.” The difference is huge when events can be lost, repeated or reordered, and in distributed systems they often are.

Kubernetes chooses level

Informers deliver edge-style events, but good controllers use them only as hints to wake up. The actual decision comes from comparing desired and actual state read from the cache. If an event is lost, duplicated or merged with another, the next reconcile still computes the right answer. This is why the work queue de-duplicates keys and why workers re-read the cache instead of trusting event contents. The event says “look at this object again,” not “here is exactly what happened.”

Idempotency is the partner of level triggering

Because the same key may be processed many times, each reconcile must be idempotent, meaning that running it twice produces the same outcome as running it once. If the Deployment already has five Pods, a second pass does nothing. Designing for repeatability is the price of resilience, and it is a price worth paying every time.

The watch mechanism is not an exotic feature. It is the heartbeat of the entire platform.

Informers are efficient by design, but at scale the details start to matter.

Where the cost actually goes

The big costs of an informer are memory and the initial list. Memory grows with the number and size of cached objects. The initial list, repeated whenever a relist is needed, can be a heavy moment for both the client and the server, because every object must be encoded, sent and decoded. In a cluster with hundreds of thousands of objects, careless clients can cause noticeable pressure on the control plane just by restarting together.

Practical tuning levers

Streaming the initial list

Newer Kubernetes versions improve the start-up path with a mechanism that lets a client ask the server to stream the initial state through the same watch connection, followed by a marker to say the initial burst is complete. This replaces one huge list response with a steady stream and lowers peak memory on the API server. Availability depends on your cluster version and feature settings, so check the release notes for the version you run before relying on it.

Protecting the server

The API server uses a fairness system that groups requests into priority levels and queues them, so that one misbehaving client cannot starve others. Long-running watch requests are handled differently from short requests, but the initial lists and the writes your controllers make still pass through that fairness system. A client that relists in a tight loop can still harm itself by being throttled, which shows up as slow reconciles. Good back-off behaviour keeps you on the right side of the limits.

A watch is a read permission with a long reach. Treat it with the care it deserves.

Watching needs explicit permission

Role-based access control treats list and watch as separate verbs, and informers need both, plus get in some cases. Granting a controller the right to list and watch Secrets across the cluster means it can read every Secret in every namespace. That is a very powerful permission, and the stream will keep delivering updated values as they change. Always scope controllers to the narrowest kinds, namespaces and verbs they truly need.

Identity and transport

Each client authenticates to the API server, usually with a service account token for in-cluster programs or a client certificate for node components. The connection is encrypted in transit. Because watch connections are long-lived, short-lived tokens are refreshed in the background, and a client that mishandles refresh may find its watch ended with an authentication error, which should trigger a clean reconnect rather than a crash loop.

Cache hygiene

The local cache holds a full copy of sensitive objects in process memory. Take care not to log whole objects, avoid writing the cache to disk, and keep Secret informers out of programs that do not need them. If a process is compromised, everything in its caches is exposed, so the principle of least data applies just like least privilege.

Auditing and visibility

Audit logs record watch requests, including who opened them and for which resource. Reviewing unusual watch patterns, such as a new service account watching all Secrets, is a practical way to catch both mistakes and malicious use. Watch events themselves are not logged one by one, but the changes that cause them are recorded as the original writes.

You cannot fix what you cannot see. Informers expose plenty of signals if you know where to look.

Watching the work queue

Controllers built with the standard libraries publish metrics about their work queues. The most useful ones tell a simple story. Queue depth shows how many keys are waiting. The add counter shows how fast new work arrives. Queue latency shows how long keys wait before a worker picks them up. Work duration shows how long each reconcile takes. Retry counters show how often reconciles fail and come back. A healthy controller has a shallow queue, short waits and few retries.

Watching the server side

The API server exports metrics about registered watchers per resource kind, the number of watch events sent, the size of those events, and the number of watches it terminated. A jump in registered watchers can indicate a buggy client opening streams without closing old ones. A rise in terminated watches can point to slow consumers. etcd’s own metrics for database size, compaction and request latency complete the picture, because slow storage eventually slows the entire event pipeline.

Logging with purpose

Log the key being reconciled, the outcome and the duration, rather than the whole object. Log reconnects and relists at a level where you can count them, since a sudden rise in relists is one of the clearest signs of trouble. Include the resource kind in each message so that logs from different informers can be told apart in a busy process.

A simple health routine

Habits that experienced controller authors apply almost automatically.

Patterns that work

Best practices at a glance

Real production incidents, distilled into a few memorable lessons.

No design is free. Here is an honest look at what informers give and what they ask in return.

Informer cache versus direct read

Sometimes you need the strictest possible freshness, for example before a destructive action. A direct read from the API server can be made strongly consistent, while the informer cache is allowed to be a little behind. Most controllers accept the small lag because they are level-triggered and will correct themselves on the next event, but critical one-off checks may justify the extra cost of asking the server directly. Choose deliberately, not by habit.

Questions that beginners and experienced engineers alike keep asking.

No. A watch is the server feature that streams changes over one connection. An informer is a client-side assembly that uses a list and a watch, adds a queue and a local cache, handles reconnects and relists, and notifies your handlers. The watch is the ingredient, the informer is the finished dish.

Because reading from the API server’s in-memory cache is far cheaper than a fresh read through etcd, and the watch that follows will immediately fill in any tiny gap. The result is correct and much lighter on the control plane, especially when many clients start together.

When it comes back, the informer lists the current state, compares it with its empty or old cache, and delivers add events for everything that exists. Because the logic is level-based, the controller then reconciles reality with the spec regardless of what it missed. No event history is required.

It means the client tried to resume from a resource version that the server no longer remembers, usually because the connection was down for too long or the stream was very quiet without bookmarks. The client recovers by relisting. Occasional occurrences are normal, constant ones suggest an unstable network path, a very slow consumer, or heavy write churn compacting history quickly.

Often not. The work queue’s retry behaviour already handles failures. A resync replays cached objects to handlers, which helps if your logic might leave something half-done without retrying, but it also adds load. Use it only when you can name the problem it solves.

Yes, and they should. Through a shared informer factory they attach separate handlers to the same cache and stream. This saves memory and API traffic. Just remember that they share the same object copies, so neither may modify cached objects in place.

Keys let the worker read the freshest version from the cache at processing time and let the queue merge duplicates. Storing objects would risk acting on out-of-date copies and would multiply memory use during bursts.

Do not rely on that. Within a continuous stream they arrive in order, but across reconnects, relists and resyncs your handlers may see repeats, merged changes or synthetic deletes. Design for at-least-once and idempotent handling rather than exactly-once.

They are small progress markers that update the client’s resume point even when nothing relevant changed. A quiet watcher can then reconnect later from a recent version instead of an ancient one, which avoids unnecessary relists.

Not immediately. The change must travel from the server through the watch back into your cache, which takes a short moment. Design your logic so that the next event brings the new state, instead of expecting it instantly.

It is a wrapper that holds the last known state of an object whose delete event was missed and later discovered during a relist. Handlers should check for it before treating the value as a normal object, otherwise they may fail on an unexpected type.

Yes. Custom resources are served by the same API server with the same list and watch behaviour, so the same informer machinery applies, whether through generated typed clients or generic dynamic ones.

Kubernetes avoids the trap of constant polling by splitting the job in two. The API server offers a list that gives a snapshot and a watch that streams every later change, positioned by an opaque resource version. Inside the server, a watch cache turns one expensive upstream stream into many cheap downstream ones. On the client, an informer combines a reflector, a DeltaFIFO queue, an indexed local store and event handlers into a self-healing pipeline, and a work queue turns noisy events into calm, de-duplicated, retryable work. Level-based, idempotent reconcile logic ties everything together so that lost or repeated events never leave the cluster permanently wrong.

If you remember one picture, remember the kitchen from the start: a bell that rings only when something changes, and a whiteboard that always shows the current truth. The watch is the bell, the informer is the whiteboard, and your controller is the cook who looks at the board and does whatever is needed to make reality match the order.