Before learning the solution, it helps to feel the problem it solves.

Kubernetes is a shared workspace. The things inside it, such as Pods, Deployments, Services, ConfigMaps and Nodes, are called objects, and no single person or program owns them exclusively. A developer might change a Deployment from a terminal. A deployment tool might push a new version of that same Deployment a few seconds later. An autoscaler might adjust the number of copies while both of them are still working. A controller might write progress information back onto the object. All of these writers talk to one central program, the API server, which keeps every object safe in a database called etcd.

When many writers share one object, a famous bug becomes possible. Computer scientists call it the lost update problem. It happens in three small steps. First, writer A reads the object. Second, writer B reads the same object. Third, each writer changes its own private copy and sends it back. The writer who saves last wins, and the earlier change disappears without any warning. Nobody did anything wrong. Each writer performed a perfectly normal read, edit and save. The danger hides in the gap of time between the read and the write.

sequenceDiagram
    participant A as Writer A
    participant B as Writer B
    participant S as API Server
    A->>S: Read the object
    S-->>A: Copy 1
    B->>S: Read the object
    S-->>B: Copy 1
    A->>S: Save my edited copy
    S-->>A: Saved
    B->>S: Save my edited copy
    S-->>B: Saved, silently replacing A's work
        

Why this matters even more in Kubernetes

In an ordinary website, two people rarely edit the same record in the same second. Kubernetes is different because automation never sleeps. Controllers run in endless loops, always comparing the world as it is with the world as it should be, and fixing any difference. Several of them may care about the same object at once. That means collisions are not rare accidents. They are a normal, daily part of how the system works, and the design must handle them calmly, thousands of times a day, without human help.

A small piece of text on every object that works like a “last changed” stamp.

Every Kubernetes object has a section called metadata. Inside it you will find the name, the namespace, labels, annotations and several other bookkeeping fields. One of those fields is resourceVersion. It is a short string that the server changes every time the object is stored in a new form. You can think of it as a stamp that says “this is the exact edition of the object you are looking at.”

The way it is used is beautifully simple. When you read an object, the server hands you the object together with its current stamp. Later, when you want to save a change, you send the object back with the stamp you received. The server compares your stamp with the stamp on the stored object. If they match, nobody touched the object while you were working, so your change is accepted and a new stamp is created. If they do not match, somebody else got there first, so your change is refused and you are told to look again. No waiting, no locking, just one comparison.

Three facts to remember

First, resourceVersion is a string, not a number you should do arithmetic on. Second, it is opaque, which means its inside meaning belongs to the server and clients should only copy it and compare it for equality. Third, it is owned by the server. A client cannot make up a clever value to get its way. A client can only echo back a value it was previously given.

Its neighbours in the metadata

Several other metadata fields are easy to confuse with resourceVersion. Each answers a different question, so it is worth meeting them side by side.

The stamp is borrowed from a counter inside etcd, the database underneath Kubernetes.

Kubernetes keeps all of its objects in etcd, a distributed key-value database. etcd has a feature that makes concurrency control easy: it keeps one global counter called the revision. Every successful write anywhere in the database moves that counter forward by exactly one. Each stored key also remembers two special numbers, the revision at which it was created and the revision at which it was last modified.

In the usual setup, the API server uses the last-modified revision of an object’s key as that object’s resourceVersion. This explains why the stamps look like large numbers that seem to jump around. If you write a ConfigMap, then a Pod, then the same ConfigMap again, the revisions might be 101, 102 and 103. The ConfigMap ends up with stamp 103 and the Pod keeps stamp 102. Each object only changes its stamp when that object itself is written, but the counter underneath moves for everyone.

flowchart LR
    W1["Write ConfigMap
counter becomes 101"] --> W2["Write Pod
counter becomes 102"]
    W2 --> W3["Write ConfigMap again
counter becomes 103"]
    W3 --> R["ConfigMap stamp: 103
Pod stamp: 102"]
        

What the numbers do and do not tell you

Because the counter is shared, the stamp does not tell you how many times your object has changed. A stamp that jumped by a thousand simply means the cluster was busy with other objects. Likewise, a small difference between two stamps does not mean “almost the same.” The only safe question is whether two stamps are equal.

Lists have a stamp too

When you ask for a list of objects, the answer carries one stamp for the whole list. It describes the moment the snapshot was taken. As you will see later, that stamp is the starting line for watching changes without missing or repeating anything.

Two very different philosophies for sharing something that several people want to change.

There are two classic ways to stop people from overwriting each other. The first is called pessimistic concurrency. It assumes trouble is likely, so it makes every writer take a lock before editing. While one writer holds the lock, everybody else must wait their turn. The second is called optimistic concurrency. It assumes trouble is unlikely, so it lets everyone work freely and checks only at the very end whether somebody else changed the thing in the meantime. If someone did, the late writer simply starts that one edit again.

Why Kubernetes chose optimistic

Several reasons point in the same direction. Clients in a cluster come and go constantly, and any of them can crash at any moment. A crashed client holding a lock would block everyone else until some timeout rescued the situation, and picking good timeouts for thousands of different clients is hard. The API is a stateless HTTP interface, so holding a lock across several separate requests would be awkward. Most objects are read far more often than they are written, and two writers rarely hit the same object in the same instant, so the common case is “no conflict, no cost.” Finally, etcd natively supports a transaction that says “write this only if the key is still at revision N,” which is exactly the check that optimistic concurrency needs.

Follow one update from the moment a client reads an object to the moment it succeeds or is turned away.

Understanding a conflict is easiest when you walk through it slowly. Suppose a deployment tool wants to add a label to a Deployment while an autoscaler is changing its replica count. Here is the whole story, one step at a time.

sequenceDiagram
    participant C1 as Deployment Tool
    participant C2 as Autoscaler
    participant S as API Server
    C1->>S: Read Deployment
    S-->>C1: Object at version 7
    C2->>S: Read Deployment
    S-->>C2: Object at version 7
    C2->>S: Update replicas, expecting version 7
    S-->>C2: Accepted, now version 8
    C1->>S: Update label, expecting version 7
    S-->>C1: 409 Conflict, stored version is 8
    C1->>S: Read Deployment again
    S-->>C1: Object at version 8
    C1->>S: Update label, expecting version 8
    S-->>C1: Accepted, now version 9
        

What the error looks like

The refusal arrives as an HTTP 409 with a reason of Conflict. The human-readable message is usually something like “the object has been modified; please apply your changes to the latest version and try again.” The message is advice as well as a complaint, and it describes exactly the right recovery. Note that a 409 can also mean something else, namely that you tried to create an object whose name already exists. The reason field, AlreadyExists, tells the two apart.

What happens if you send no stamp at all

If an update arrives without a resourceVersion, the outcome depends on the kind of object. Many built-in resources allow an unconditional update, which means the server skips the check and the last writer wins. Some resources refuse an update that has no stamp and answer with a validation error. Leaving the stamp out is therefore a way of switching the safety net off, and it should be a deliberate choice rather than an accident.

What actually happens between a request arriving and the database accepting or rejecting it.

The API server is a front door with several checkpoints. Every write request walks through the same sequence, and the version check sits at the very end, right next to the database, where it can be made truly reliable.

flowchart TD
    A["Request arrives"] --> B["Authentication: who are you?"]
    B --> C["Authorization: may you do this?"]
    C --> D["Mutating admission: adjust the object"]
    D --> E["Schema validation: is it well formed?"]
    E --> F["Validating admission: any policy objections?"]
    F --> G["Storage layer: read current, run preconditions"]
    G --> H{"etcd transaction: is stored version still the expected one?"}
    H -->|Yes| I["Write and return new stamp"]
    H -->|No| J["Re-read, then retry internally or return 409"]
        

The early checkpoints

Authentication proves who is calling. Authorization, usually through role-based access control, decides whether that caller is allowed to update this kind of object. Mutating admission may adjust the incoming object, for example by adding default values or injecting extra settings. Validation then checks that the result is well formed, and validating admission applies policy rules. None of these steps look at the version stamp for concurrency purposes, because the world could still change after they finish.

The storage layer and the atomic check

The storage layer reads the currently stored object, sometimes using a recent cached copy as a quick hint, and applies the requested change to it. It then verifies the preconditions that the request carried, such as the expected stamp or expected uid. The final and authoritative step is a single etcd transaction that says, in effect, “if this key’s last-modified revision is still the one I read, store the new value, otherwise tell me what is there now.” Because etcd runs that transaction atomically, two competing writers can never both succeed from the same starting point.

If the transaction fails, what happens next depends on the request. When the client pinned a specific version, the server cannot guess what to do, so it returns a 409 Conflict and lets the client decide. When the client did not pin a version, for example in a patch that only describes a small change, the server can safely re-read the latest object, re-apply the small change and try again by itself. The client never even notices.

No-op writes are skipped

The server also notices when an update would store exactly the same bytes that are already stored. In that case it skips the database write, the stamp does not change, and no watch events are produced. This is a quiet but valuable optimisation. Controllers that re-apply the same desired state again and again cost almost nothing, and watchers are not woken up for nothing.

Kubernetes offers several write styles, and each one treats the version stamp differently.

Many people assume that every write in Kubernetes works the same way. In reality there are three main styles, and picking the right one can reduce conflicts dramatically. They differ in how much of the object they send and in who is responsible for resolving a collision.

Patch is gentle, not magic

A patch describes what to change instead of what the whole object should look like, so it naturally avoids most collisions. If you only want to add one label, a patch leaves every other field exactly as the latest stored object has it. There are several patch formats. A JSON patch is a list of precise operations and can even include a test operation that fails if a value is not what you expect. A merge patch overlays your change on top of the object, and a strategic merge patch does the same while understanding how lists of built-in objects should be combined.

Two different kinds of 409

Server-side apply introduces a second meaning for the same status code. A version conflict says “your copy is old.” A field conflict says “another manager owns this field and you tried to change it to a different value.” The first is solved by reading again. The second is a question of ownership: you can choose to give up the field, agree with the current value, or deliberately force your value and take ownership. Telling these two situations apart in logs saves a lot of confusion.

The same “only if unchanged” idea protects more than updates.

Version stamps are not limited to saving edits. Delete requests can carry preconditions too, which are conditions that must be true or the delete is refused. Two preconditions are available: the object’s uid and its resourceVersion. They let a caller say “delete this exact thing, and only if it is still the thing I looked at.”

Why the uid matters

Names are reused. A StatefulSet may delete Pod “web-0” and create a new Pod with the very same name seconds later. A controller that decided to delete the old “web-0” could accidentally delete the healthy new one if it only used the name. Because the uid is different for every incarnation, a delete with a uid precondition fails safely with a conflict when the object has been replaced.

On reads, the stamp means something different: how fresh does my answer have to be?

So far the stamp has been a guard on writes. When you read or list objects, you can also pass a resourceVersion, but here it acts as a freshness dial. It lets the caller choose between “give me the most accurate answer even if it costs more” and “give me something quick and good enough.”

Why the cache matters

The API server keeps a watch cache of recent objects in memory. Answering from that cache protects the database from a flood of heavy list requests, which is vital in big clusters. The price is that the cache can lag the database by a small amount. In recent Kubernetes releases the server can also serve consistent reads from its cache after confirming the cache is caught up, which brings together accuracy and speed.

Pagination and the continue token

Large lists can be fetched in pages using a limit and a continue token. The first page fixes a snapshot at a particular stamp, and every later page reads from that same snapshot, so you never see a half-old, half-new list. The snapshot cannot be kept forever, though. If the database compacts away that history before you finish, the next page is refused as expired and you have to start the listing again.

How programs follow a stream of changes without polling and without missing anything.

Controllers do not repeatedly ask “what changed?” every second. Instead they open a watch, a long-lived connection through which the server pushes an event whenever an object is added, modified or deleted. A watch takes a starting stamp, and the server sends every change that happened after that point. This is the second great use of resourceVersion.

The list-then-watch pattern

The standard recipe, built into the client libraries as an informer, has two parts. First, list all objects of a kind and remember the stamp attached to the list. Second, start a watch from exactly that stamp. Because the list describes the world at stamp N and the watch begins at N, there is no gap where a change could slip through and no overlap where a change could be delivered twice. The informer keeps a local cache updated from the events, and controllers read from that cache instead of hammering the server.

flowchart TD
    A["List all objects"] --> B["Save snapshot and its stamp"]
    B --> C["Watch from that stamp"]
    C --> D["Apply each event to local cache"]
    D --> E{"Connection lost or stamp too old?"}
    E -->|Connection lost| F["Reconnect from last stamp seen"]
    F --> C
    E -->|410 Gone| A
    D --> C
        

Why 410 Gone happens

The server only remembers a limited window of recent history. The watch cache holds recent events in memory, and the database periodically compacts old revisions away, by default roughly every five minutes. If a client asks to resume from a stamp older than the remembered window, the server cannot replay the missing events honestly, so it says 410 Gone, meaning “that point in history is no longer available.” The correct response is to relist and rebuild the cache.

For a controller, a conflict is not an emergency. It is a normal signal that says “look again.”

A controller works in a loop: observe the current state, compare it with the desired state, and act to close the gap. To stay fast, it usually reads from its informer cache, which is a copy kept up to date by the watch stream. That copy is almost always right, but it can be a few moments behind. When the controller writes using a stamp taken from a slightly stale copy, the server may answer with a conflict.

This is expected, and the right reaction is calm. Kubernetes controllers are level-triggered, which means they act on the current state of the world rather than on a history of events. A failed write is therefore not a lost instruction. The controller puts the object’s key back on its work queue, waits a short and growing delay, and handles it again. By then the cache has caught up, the stamp is fresh, and the write succeeds. Because the reconcile logic is idempotent, meaning that running it twice gives the same result as running it once, repeating it is harmless.

flowchart TD
    A["Reconcile an object"] --> B["Read latest copy from cache"]
    B --> C["Decide the needed change"]
    C --> D["Send write with the stamp"]
    D --> E{"Result?"}
    E -->|Success| F["Done, forget this attempt"]
    E -->|Conflict| G["Requeue the key with a growing delay"]
    G --> A
    E -->|Other error| H["Log it and requeue"]
    H --> A
        

Short retry loops

Sometimes a controller prefers to retry immediately instead of requeueing. The client libraries provide a standard helper for this, usually called retry on conflict. It re-reads the live object, runs the caller’s change function on that fresh copy, and tries to write again, backing off between attempts and giving up after a few tries. The crucial detail is the order of operations. What is repeated is the intention, not the old object. Reapplying a stale copy over and over would never succeed.

A clever use of the version check that lets many copies of a program agree on a single boss.

For high availability, important components such as the scheduler and the controller manager run as several identical copies. But only one copy should actively do the work at a time, otherwise they would make duplicate or contradictory decisions. The copies choose a leader by competing for a small Kubernetes object called a Lease.

A Lease records who currently holds it, how long the holder may keep it, when it was last renewed, and how many times leadership has changed. The leader renews the Lease regularly by updating it. The other copies keep watching. If they see that the Lease has not been renewed for longer than its allowed duration, they assume the leader is gone and try to take over.

Where optimistic concurrency saves the day

Suppose the leader crashes and two standby copies notice at the same time. Both read the Lease at the same stamp, both prepare an update that says “I am the holder now,” and both send it with that stamp. etcd accepts the first and the server answers the second with a conflict. The loser immediately knows it did not win and goes back to waiting. No special lock server, no extra coordination, only the version check doing exactly what it does everywhere else.

sequenceDiagram
    participant X as Candidate X
    participant Y as Candidate Y
    participant S as API Server
    X->>S: Read Lease
    S-->>X: Expired, version 40
    Y->>S: Read Lease
    S-->>Y: Expired, version 40
    X->>S: Claim Lease, expecting version 40
    S-->>X: Accepted, now version 41
    Y->>S: Claim Lease, expecting version 40
    S-->>Y: 409 Conflict
    Note over X: X is the leader and renews regularly
    Note over Y: Y keeps waiting and watching
        

The clock question

You might wonder whether different computers disagreeing about the time could break this. The design is careful about that. Each candidate does not trust the timestamp written by the leader. Instead it notes, using its own clock, how long it has seen the record stay unchanged. Only when that locally measured time exceeds the lease duration does it attempt a takeover. A leader that cannot renew within its deadline steps down on its own rather than carrying on and risking two active leaders.

Conflicts become a real problem when too many writers crowd around too few objects.

The chance of a conflict grows with two things: how many writers touch the same object and how often each one writes. An object that many components update frequently is called a hot object. Hot objects waste effort because losers redo their work, and in extreme cases the retries themselves create more traffic and more conflicts. Kubernetes itself has had to redesign a few of its own features to cool down hot objects.

The one-writer principle

The most reliable way to prevent conflicts is to design so that each object, or at least each field, has exactly one writer. Spec fields are written by users and tools that express intent. Status fields are written by the controller that observes reality. Separate objects are used for separate concerns. When this principle is followed, the remaining conflicts are rare and harmless. When it is ignored, no retry strategy can save the system.

A small toolbox of habits that keeps controllers calm under concurrency.

Anyone who builds tools or operators for Kubernetes will meet conflicts sooner or later. The good news is that a handful of well-known patterns handle nearly every situation. Treat the following cards as a checklist when designing a new controller.

A safe update recipe

Putting the patterns together gives a simple recipe that works for most controllers.

Five habits that look harmless and cause real trouble in production.

Most conflict-related bugs come from a short list of misunderstandings. Studying them in advance is cheaper than discovering them during an outage.

Symptom to cause quick guide

Version stamps protect integrity, not secrecy, and they leave useful footprints you can measure.

Security thinking

It is important to be clear about what the stamp is not. It is not a password, a signature or an access control. It does not decide who may change an object. That job belongs to authentication and role-based access control. The stamp only answers the narrow question “has this changed since I looked?” Preconditions on delete protect against accidents and races, such as removing a replacement object by mistake, but they do not stop a person who has permission from doing something harmful on purpose. Policies, such as admission rules that make certain fields immutable, are the right tool for that.

Opacity also has a small security benefit. Because clients are told never to interpret the stamp, nobody builds trust decisions on top of its structure, and the platform stays free to change how it is produced.

What to watch

Conflicts leave clear traces in logs, metrics and audit records. A few conflicts per hour in a busy cluster are normal and prove the safety net is working. What deserves attention is a sustained high rate, a rate that keeps climbing, or conflicts on the same object that never resolve.

A fair look at what the design buys and what it costs.

No concurrency strategy is perfect, and Kubernetes made a conscious bet. Seeing the whole picture helps you decide when to lean on the built-in behaviour and when to design around it.

How it compares with other approaches

Kubernetes actually uses a blend. The coarse version stamp guards whole-object writes, while server-side apply adds field-level ownership on top for shared objects. This combination keeps the cheap, crash-safe core and softens its biggest weakness, which is that unrelated edits to the same object can collide.

Where these ideas appear in everyday cluster life.

You do not need to build a controller to meet the version stamp. It quietly shapes everyday tools and platform designs, and recognising it makes puzzling behaviour easier to explain.

Short answers to the questions that beginners ask most often.

No. The stamp changes on every stored change, including status and labels. Generation only increases when the desired state in the spec changes, so it moves far more slowly.

You can only send back a value you received. Sending a different value simply causes a conflict. Leaving it out may allow an unconditional update, but that removes your protection against overwriting other changes.

Usually not. It means someone else changed the object first, and it is the normal way the system keeps writers honest. It becomes a concern only when it is constant or never resolves.

If the new object is identical to what is stored, the server skips the write, so nothing changes and no watch event is sent. The same thing happens when you re-apply an unchanged definition.

The format is opaque and may differ between storage setups. The official guidance allows equality checks only. Event order should be taken from the order of a single watch stream.

Mostly, because the server applies the small change to the latest object. A patch can still conflict if it includes a stamp, and some patch types replace whole lists, which can overwrite another tool’s entries.

The point in history you asked to resume from has been forgotten, usually because it is older than the retained window. The right response is to list everything again and start a new watch from the new list’s stamp.

Reading from a local cache is fast and spares the API server from enormous load. Because controllers are level-triggered and idempotent, the occasional stale read just causes a harmless conflict and a retry.

A version conflict says your copy is outdated. A field conflict says another manager owns a field you tried to set to a different value. The first is fixed by re-reading, the second by deciding who should own the field.

No. It is purely a freshness check. Access control belongs to authentication and authorization, and policy rules belong to admission control.

Kubernetes is a crowded whiteboard where people and programs constantly read and write the same objects. Instead of making everyone queue behind locks, it stamps every object with a resourceVersion, lets writers work freely, and checks at save time that nobody changed the object in the meantime. A mismatch produces a clear 409 Conflict, and the loser simply reads the latest copy and tries again. The same stamp gives reads a freshness dial, gives watches an exact starting line, powers leader election, and underpins safe deletes through preconditions. Once you see the pattern, a great deal of Kubernetes behaviour, from controller retries to informer relists, becomes easy to explain.