Server-Side Apply in Kubernetes
How the cluster itself keeps track of who owns every field of every object, why that ends the old fights between tools and teams, and how to use it safely, explained from zero.
Imagine a shared document that ten people edit at the same time. One person fixes the title. Another changes a paragraph. A third adds a table. Now imagine that every time someone saves, they upload their whole copy and overwrite everyone else’s work. By lunchtime, nobody can trust what the document says, and everyone blames everyone else. Sensible tools solve this by tracking who changed which part, so that two people editing different paragraphs never collide, and two people editing the same sentence are told about it before anything is lost.
A Kubernetes object is exactly such a shared document. A developer edits it by hand. A deployment pipeline updates it. An autoscaler changes the replica count. A security tool adds a sidecar. An Operator writes the status. For many years, Kubernetes had no good answer for how all of these hands should share one object. Server-Side Apply, often shortened to SSA, is the answer. It moves the job of tracking ownership and merging changes from each client tool into the Kubernetes API server itself. This post explains the idea from the very beginning, assuming you know nothing yet. By the end, you will understand field ownership, conflicts, merge rules, migration, and the way modern tools use SSA, and you will be able to explain it clearly in design reviews and interviews.
1The Problem: Many Hands on One Object
To appreciate the solution, we first need to feel the problem. Let us start with what an object is and who touches it.
What a Kubernetes object looks like
Everything in Kubernetes is stored as an object: a Deployment, a Service, a ConfigMap, a custom resource of your own. Each object is a structured document made of fields. Some fields describe what you want, such as the container image or the number of copies. Other fields describe what the cluster observed, such as how many copies are ready. The whole document is kept in the cluster’s database, and anyone with permission can read or change it.
Who edits the same object
In a real cluster, a single object is rarely touched by just one party. Consider an ordinary Deployment for a web application. A developer commits a change to the image version, and a pipeline applies it. A horizontal autoscaler adjusts the replica count when traffic rises. A platform tool might add a standard label or a monitoring sidecar. An engineer, in the middle of an incident, uses a quick command to bump a resource limit. Each of these actors is correct to change its own part. The trouble starts when their changes overlap or overwrite each other.
Think of a kitchen shared by several cooks. One owns the soup, one owns the bread, one owns the salad. Everyone works fine as long as each cook keeps to their own station. Chaos begins when a new cook walks in, throws out the whole counter to “start clean”, and rebuilds it from a recipe card. The soup is gone, and nobody knows why.
The classic symptoms
Reset replicas
A pipeline applies the full file, which says “three replicas”, and undoes the autoscaler’s decision to run twenty.
Vanishing sidecars
A tool that added a helper container finds it deleted by the next deployment because the pipeline did not know about it.
Hotfix lost
An engineer fixes a limit by hand during an outage, and the next pipeline run silently reverts it.
Stale deletions
A field removed from the source file stays on the cluster forever, because the tool never knew it should delete it.
Server-Side Apply lets the cluster remember which party owns which field, so that parties can safely share one object, collisions are detected instead of hidden, and each party can cleanly remove only what it added.
2How Kubernetes Changes Objects Today
Before Server-Side Apply makes sense, you need a clear picture of the ordinary ways to change an object, and what each one risks.
The API server and the database
All changes go through the API server, the front door of the cluster. It checks who you are, checks you are allowed to do the action, validates the change, and stores the result in the cluster database. Every client, whether it is a command-line tool, a pipeline, or a controller, speaks to the same API server, using ordinary web requests.
The basic ways to write
| Operation | Plain-English meaning | Main risk |
|---|---|---|
| Create | Make a new object | Fails if it already exists |
| Update (replace) | Send the whole object and overwrite what is stored | Wipes out fields you did not know about |
| Patch | Send only the pieces you want to change | Merge rules vary by patch type and may surprise you |
| Delete | Remove the object | Obvious, but often combined with ownership and cleanup rules |
Optimistic concurrency
Every object carries a version stamp called the resource version. When you replace an object, you send back the version you last read. If someone else changed the object in the meantime, the stamp no longer matches, and the API server rejects your write with a conflict. You then re-read and try again. This protects against lost updates but only at the level of the whole object. It cannot tell the difference between a harmless change to a different field and a harmful overwrite of the same field.
The resource version is like a “last edited” number written on the top of a paper form. If two people take the form from the office and bring it back, the second person is told “the form has changed since you took it”. That is helpful, but it does not say which line changed, so the second person must start over even when the changes would not have clashed at all.
The types of patches
Kubernetes supports several ways to patch an object. A JSON patch is a list of precise edits by position, which is fragile if the list changes underneath you. A JSON merge patch overlays a partial document, but replaces whole lists in one go. A strategic merge patch understands Kubernetes lists by using special keys, for example merging containers by name, but it only works for built-in types and needs special client knowledge. Each is useful, yet none of them records who wrote which field.
sequenceDiagram
participant A as Pipeline
participant S as API server
participant H as Autoscaler
A->>S: Read Deployment (version 100, replicas 3)
H->>S: Set replicas to 20 (version becomes 101)
A->>S: Replace whole object using version 100
S-->>A: Conflict: the object changed
A->>S: Read again (replicas 20) and retry
A->>S: Replace whole object with replicas 3
S-->>A: Accepted, autoscaler decision lost
None of these basic operations know who owns a field. The API server treats each write as an anonymous change. So the burden of being polite falls on every client, and clients differ in how careful they are.
3Client-Side Apply and Its Limits
Before Server-Side Apply, almost everyone used a different “apply” that ran entirely inside the command-line tool. Understanding how it worked explains exactly why a new design was needed.
What “apply” means
The command many people know as kubectl apply is built around a declarative idea: you keep your desired configuration in a file, and you tell the cluster, “make it look like this”. If the object does not exist, apply creates it. If it exists, apply works out the differences and updates it. You can run the same command again and again safely.
How the old method worked
The original method is called client-side apply, because the comparison happens on the client. To work out what to change, the client uses a three-way merge with three inputs: the new file you provided, the live object currently in the cluster, and a saved copy of the file you applied the last time. That saved copy is stored inside the object itself, in an annotation named kubectl.kubernetes.io/last-applied-configuration. The client compares the old file with the new file to detect what you deleted, compares the new file with the live object to detect what to add or change, and then sends a patch.
flowchart TD
NEW["New file from the user"] --> M["Client computes three-way merge"]
LIVE["Live object read from the cluster"] --> M
LAST["Last-applied copy stored in an annotation"] --> M
M --> P["Client builds a patch"]
P --> S["API server stores the result"]
S --> A["Annotation updated with the new file"]
The limits of the client-side design
What it did well
- Made declarative files the normal way to work.
- Detected removed fields using the saved copy.
- Worked on any cluster without server support.
Where it struggled
- The saved annotation duplicated the file and had a size limit.
- Only one tool, kubectl, understood the annotation; other tools ignored it.
- Merge logic lived in the client, so different versions behaved differently.
- Poor support for custom resources, which lack the special merge hints.
- No notion of ownership, so overlapping edits silently overwrote each other.
The annotation size problem
Annotations on an object have a combined size limit of 256 kilobytes. Because the saved copy of the whole file lives in one annotation, large objects, such as big ConfigMaps or long custom resources, could hit this limit and fail to apply. Teams worked around it with awkward tricks. This was one of the first practical pains that motivated change.
Custom resources and lists
The client-side method relied on the strategic merge patch for built-in types, which knows that a list of containers should be merged by name. For custom resources, no such hints existed, so it fell back to a simpler merge that treats lists as replace-everything. Two teams editing different items in the same list of a custom resource could erase each other’s entries. As custom resources and Operators became central to Kubernetes, this was a growing problem.
Client-side apply is like each cook carrying a private notebook describing what they did last time. It only helps that one cook. A different cook, with a different notebook or none at all, has no idea what the first cook put on the counter.
Production example: pipelines clashing with autoscalers
A very common real-world story involves a deployment pipeline that applies a complete file containing a fixed replica count, while an autoscaler adjusts the same field. Teams discovered, usually during a traffic spike, that every release reset the replicas to a small number and caused brief overload. This is exactly the sort of problem that field ownership is designed to prevent.
4Meet Server-Side Apply
Server-Side Apply keeps the declarative spirit of the old apply but moves the intelligence to the one place every client shares: the API server.
The main idea
With Server-Side Apply, a client sends its desired configuration to the API server as a special kind of patch called an apply request, along with a name for itself, called the field manager. The API server then does the merging. It knows which fields each manager has said it cares about, records that information on the object, and refuses or reports changes that would trample someone else’s fields.
A short history
Alpha
Server-Side Apply first appeared as an experimental feature around Kubernetes 1.14.
Beta
It became enabled by default around 1.16, as the field-tracking design matured.
General availability
It reached GA in Kubernetes 1.22, meaning it is stable and supported for production use.
Ecosystem adoption
Tools such as GitOps controllers, Operator libraries, and cloud add-on managers have since adopted it for their own writes.
The three promises
Ownership is tracked
The cluster records which manager set each field, so there is a shared answer to “who owns this?”.
Conflicts are surfaced
If you try to change a field someone else owns to a different value, you are told, instead of overwriting silently.
Removal is clean
If you stop mentioning a field you used to own, the cluster removes it for you, as long as nobody else also owns it.
flowchart LR
C["Client with a field manager name"] -->|"apply request with desired fields"| API["API server"]
API --> MERGE["Server merges using the schema and ownership records"]
MERGE --> OWN["Ownership data stored with the object"]
MERGE --> DB["Updated object stored in the database"]
MERGE -. "if clash" .-> ERR["Conflict reported to the client"]
Declare intent, not the whole object
One of the most important mindset shifts is this: with Server-Side Apply, you should send only the fields you care about, not a complete copy of everything. You are saying, “these are the fields I own, and these are the values I want”. Fields you do not mention are left alone, unless you used to own them and have now dropped them. Other parties remain free to own the remaining fields.
Imagine the shared kitchen now has labels on every station: soup, bread, salad. Each cook tells the manager, “I am responsible for the soup, and here is what it should look like”. The manager makes sure no other cook rewrites the soup station without a conversation. And if the soup cook leaves the soup out of his next plan, the manager clears that station because it was his.
What stays the same
Server-Side Apply still uses the same objects, the same permissions system, and the same validation as every other write. You are still sending a request to the same API server. What changes is the rule for combining your request with what is already there, and the extra bookkeeping the server keeps.
Apply is a kind of write that declares full intent for the fields you own. Update is the name for every other kind of write, such as replace and ordinary patch. Both kinds are tracked, but they behave differently, as the next chapters explain.
5Field Ownership and managedFields
The heart of Server-Side Apply is a record of ownership. Once you understand this record, everything else follows naturally.
Where the record lives
Every Kubernetes object has a metadata section, and inside it there is a list called managedFields. Each entry in the list describes one field manager and the exact set of fields that manager owns. By default, command-line tools hide this list to keep the output readable, but you can ask to see it.
What each entry contains
| Part | Meaning | Example idea |
|---|---|---|
| Manager | The name of the party that made the change | A pipeline name, a controller name, or a tool name |
| Operation | Whether the change was an Apply or an Update | Apply for declared intent; Update for ordinary writes |
| API version | Which version of the object type the manager used | For example the stable version of Deployment |
| Time | When this manager last changed any of its fields | A timestamp |
| Fields set | A tree listing every field this manager owns | Image of a container, replica count, a label |
| Subresource | Whether the change was made through a special part of the object, such as status or scale | Status subresource for controllers |
What “owning a field” really means
When a manager owns a field, three things are true. First, the manager has told the cluster that it cares about the value of that field. Second, if the manager later stops mentioning the field in an apply request, the field is removed, provided no other manager also owns it. Third, if some other manager tries to set the field to a different value, the cluster treats it as a clash instead of a silent overwrite. Ownership is a mix of responsibility and protection.
flowchart TD
OBJ["Deployment object"] --> SPEC["spec"]
SPEC --> REP["replicas: owned by the autoscaler"]
SPEC --> TPL["template"]
TPL --> CON["containers (merged by name)"]
CON --> IMG["image: owned by the pipeline"]
CON --> LIM["resources limits: owned by the platform team"]
OBJ --> META["metadata labels"]
META --> LAB["team label: owned by the pipeline"]
META --> MON["monitoring label: owned by the monitoring tool"]
Shared ownership
Ownership is not always exclusive. If two managers apply the same value to the same field, both become co-owners. There is no conflict, because they agree. If one of them later changes the value, the other is told about the disagreement the next time it applies. If one stops mentioning the field, the other still owns it and the field stays. This shared ownership is what lets teams safely express “I need this to be true” without fighting over who set it first.
Naming your manager
The field manager name is chosen by the client. Command-line tools use a default name based on the program. When you use kubectl with server-side apply, the default manager name is kubectl. You can pick a more meaningful name, such as the name of your pipeline or your controller. Good names make ownership readable. Poor names, such as the same generic name for every pipeline, make it impossible to tell parties apart and can cause teams to accidentally share ownership.
Because the client chooses the manager name, a name is a label for coordination, not proof of identity. Any caller with permission to write the object could claim any manager name. Never use field ownership as an access control. Use the normal permission system for that.
The ownership record is like the sign-off sheet on a shared checklist. Next to each line is the name of the person who filled it in. You can see at a glance who is responsible for what, and if you try to change someone else’s line, the sheet tells you whose it is before you do.
6Conflicts: When Two Owners Disagree
Conflicts are a feature, not a failure. They are the cluster’s way of telling you, politely, that you are about to change something another party cares about.
When a conflict happens
A conflict arises when an apply request sets a field to a value different from the current value, and that field is owned by a different manager. The API server refuses the request with a conflict error and lists the fields in dispute, along with the names of the managers that own them. The client can then decide what to do.
flowchart TD
A["Apply request arrives"] --> B{"For each field: is it owned by another manager?"}
B -- "No" --> OK["Accept the field and record ownership"]
B -- "Yes" --> C{"Is the requested value the same as the current value?"}
C -- "Yes" --> SH["Accept and share ownership"]
C -- "No" --> F{"Is force enabled?"}
F -- "No" --> CON["Reject with a conflict listing owners and fields"]
F -- "Yes" --> TAKE["Take ownership and overwrite the value"]
Three ways to resolve a conflict
Overwrite with force
You tell the server to take ownership anyway. The previous owner loses that field. Use this only when you are sure your claim is the correct one.
Do not apply those fields
You remove the disputed fields from your configuration and let the other owner keep control. This is the right answer when the field was never really yours.
Agree on the value
You change your value to match the current one. Both of you then co-own the field, and the conflict disappears.
Who should force, and who should not
The right choice depends on who you are. A controller that fully manages an object, such as an Operator that owns the Deployments it creates, should normally force ownership, because those fields are its responsibility by design. If someone else edits them, the controller is meant to correct that. A human or a pipeline sharing an object with others should generally not force. A conflict is valuable information: it tells you that someone else has a stake in that field, and you should talk to them or reconsider.
Force is appropriate when
- You are a controller that owns the whole object.
- The object is created and managed only by your tool.
- You deliberately want to reclaim fields after drift.
Force is risky when
- Other tools or teams legitimately edit the same object.
- You are using it just to make an error go away.
- You do not know who owns the disputed fields.
The most common mistake with Server-Side Apply is adding the force option to every command the first time a conflict appears. This silently recreates the old problem of tools overwriting each other, and merely hides the warnings. Treat a conflict as a question to answer, not an obstacle to bypass.
Conflicts only happen for apply
An important detail: conflicts are reported only for apply requests. An ordinary update, such as a replace or a normal patch, never fails because of ownership. Instead, it simply takes ownership of the fields it changed. This is by design, for backward compatibility with all the tools that existed before Server-Side Apply. It also means that a stray update can quietly steal a field from an applier, and the applier will see a conflict the next time it runs.
Think of the two kinds of writers as polite and impatient. The apply writer is polite: it checks the sign-off sheet and asks before changing someone else’s line. The update writer is impatient: it simply writes what it wants and puts its own name on the line. The sheet remains accurate, but the polite writer may find, next time, that the line is no longer where it left it.
7Merge Rules: How Maps and Lists Combine
Ownership is tracked per field, but objects contain nested maps and lists. The way the server merges these shapes decides whether two teams can safely edit different parts of the same list.
The shapes inside an object
An object is built from a few basic shapes. A scalar is a single value such as a number or a text string. A map (also called an object) is a set of named fields. A list is an ordered collection of items. Merging scalars and maps is simple: each named field is handled on its own. Lists are the hard part, because items have positions that can shift when something is added or removed.
Three kinds of lists
| List type | How items are identified | Merge behavior | Typical use |
|---|---|---|---|
| Atomic | The list is treated as one single value | Whoever applies last replaces the entire list; no partial ownership | Short lists where the order or complete set matters |
| Set | Each item is its own identity, like a unique tag | Different managers can own different items; duplicates are not allowed | Lists of simple unique values |
| Map (keyed list) | Each item is identified by one or more key fields, such as a name | Items are merged by key; different managers can own different items or different fields inside an item | Containers, environment variables, volumes, ports |
An example in words
A Deployment holds a list of containers. In the built-in type, that list is merged by the container’s name. Suppose the pipeline owns the “web” container’s image, and a platform tool owns a second container named “metrics-agent”. When the pipeline applies its configuration, which mentions only the “web” container, the “metrics-agent” container is left alone, because the pipeline never owned it. That is exactly what the platform tool needs. Likewise, the platform tool can add or change its container without touching the pipeline’s.
flowchart LR
subgraph L["Containers list merged by name"]
W["web: image and port owned by the pipeline"]
M["metrics-agent: owned by the platform tool"]
end
P["Pipeline apply mentions only web"] --> W
T["Platform apply mentions only metrics-agent"] --> M
P -. "does not touch" .-> M
Maps with atomic behavior
Maps can also be marked as atomic, which means they are treated as a single value, similar to an atomic list. This is useful for small structured values that only make sense together, such as a pair of values that must change as a unit. A granular map, the default, treats each key independently.
How the server knows the rules
For built-in types like Deployment and Service, the rules are already part of Kubernetes. For custom resources, the rules come from the schema in the custom resource definition. Authors of the definition mark each list with its type, using special annotations in the schema, such as x-kubernetes-list-type and x-kubernetes-list-map-keys, and mark maps with x-kubernetes-map-type. If the author says nothing, lists default to atomic, which is the safe but coarse choice.
If you build a custom resource where several teams or controllers should edit different items of a list, mark that list as a map with clear keys, or as a set. Leaving it atomic means any apply to the list replaces it entirely and ownership of individual items cannot be tracked.
Benefits of precise list types
- Several managers can safely share one list.
- Removing your item does not remove others.
- Ownership shows exactly who added each entry.
Costs of getting them wrong
- Atomic lists cause whole-list overwrites.
- A wrong key choice can merge unrelated items.
- Changing list types later is a compatibility risk.
8Declarative Deletion: Removing by Leaving Out
One of the most useful and most surprising behaviors of Server-Side Apply is how removal works. You do not say “delete this field”. You simply stop mentioning it.
The rule
When you apply a configuration, the server compares the fields in your request with the fields you owned after your previous apply. Any field you used to own but no longer mention is removed from the object, unless another manager also owns it. If another manager owns the same field, the field stays and you simply give up your share of ownership.
sequenceDiagram
participant Y as You (manager A)
participant S as API server
Y->>S: Apply: label env=prod and label tier=web
S-->>Y: Stored. You own both labels
Y->>S: Apply: label env=prod only
S->>S: You owned tier but no longer mention it
S->>S: No other owner of tier, so remove it
S-->>Y: Stored. Only env remains
Why this is powerful
Without this rule, configuration files could only add or change things. To remove a label, you would need to run a separate delete command. With declarative deletion, your file is the complete truth about your fields. Delete a line from the file, apply again, and the cluster follows. This makes configuration review simple: the file in version control always matches what you own.
Why this can be dangerous
The same rule means that careless edits can delete real settings. If a team member trims a file and removes a field that the cluster actually needs, the next apply removes it. This is also why switching a field from “owned by me” to “owned by another tool” needs a deliberate hand-over, as we will see with autoscalers later.
Removing a label you added
You were the only owner, so it disappears exactly as intended.
Removing a value two managers set
The field remains because another manager still owns it. You merely give up your claim.
Removing a field with a default
If nothing else owns it, the server clears it and the default value returns, which may not be what the cluster was running.
Handing a field to another tool
You must make sure the other tool owns it first, or the value can briefly reset.
Declarative deletion removes fields, not whole objects. If you stop applying a Deployment file, the Deployment stays in the cluster. Removing complete objects needs a separate action, such as a delete command or a tool that tracks object inventories and prunes them.
Imagine you hang several photos on a shared wall and keep a list of which photos are yours. If you take a photo off your list, the helper takes it down for you. But if another person has also pinned the same photo, the helper leaves it up, because that person still wants it there.
9Apply Versus Update: Living Together
Real clusters mix both styles of writing. Understanding how apply and update interact helps you predict surprises before they happen.
Two operations, two behaviors
| Question | Apply | Update |
|---|---|---|
| What does the writer send? | Only the fields it wants to own | A full object or a patch of changes |
| Are conflicts reported? | Yes, unless forced | No, it silently takes ownership of changed fields |
| What does ownership include? | Everything in the applied configuration | Only the fields that actually changed |
| Are omitted fields removed? | Yes, if previously owned and not shared | No, unless the update itself removes them |
| Typical users | Pipelines, GitOps tools, controllers that declare state | Manual edits, scripts, quick patches, older tools |
How common commands appear
Many familiar commands use update-style writes and therefore show up with their own manager names in the ownership record. An interactive edit is recorded under an edit-related manager name. A scale command, a label command, or a set-image command each leave a trace. The old client-side apply appears under a client-side-apply manager name. This record is extremely handy for answering the question “who changed this field and how?” long after the event.
A realistic sequence
Pipeline applies
The pipeline declares the image and labels. It now owns those fields as an Apply manager.
Autoscaler updates
The autoscaler changes the replica count through the scale endpoint. It owns replicas as an Update manager.
Engineer edits by hand
During an incident, someone changes the image. The edit takes ownership of the image field from the pipeline.
Pipeline applies again
The pipeline sees a conflict on the image. It must decide: force to restore its version, or investigate the manual change.
The conflict in step four turns an invisible override into a visible event. Instead of the pipeline silently reverting the hotfix, someone is prompted to decide whether the manual change should be kept or reverted. That is exactly the kind of coordination the old system lacked.
Mixing styles: guidelines
Healthy mixing
- Declarative tools apply the fields they own.
- Autoscalers and controllers update the fields meant for them.
- Emergency manual edits are rare and followed by cleanup.
Unhealthy mixing
- Several tools applying the same fields under the same name.
- Manual edits that never get reconciled with the source files.
- Scripts that replace whole objects.
10Using Server-Side Apply from the Command Line
You do not need special tools to use Server-Side Apply. The familiar kubectl command supports it through a few options. This chapter explains them in words, so you know what each one means before you type it.
The key options
| Option | What it does | When to use it |
|---|---|---|
--server-side | Sends the file as a server-side apply request instead of computing the patch on the client | Whenever you want ownership tracking and server merging |
--field-manager | Sets the name under which your changes are recorded | Always, in pipelines, so every pipeline has its own clear name |
--force-conflicts | Takes ownership of any conflicting fields | Rarely, and only when you are sure the fields should be yours |
--dry-run=server | Asks the server to process the request fully but not save it | To preview what would change and whether any conflicts exist |
diff with the server-side option | Shows the difference the server would produce | In review steps before a real apply |
--show-managed-fields | Includes the ownership record in the output of a get command | When debugging who owns what |
A typical safe workflow
Choose a manager name
Pick a short, descriptive name for your pipeline, such as the application name plus the word “deploy”.
Preview with a server dry run
Run the apply in server dry-run mode. It reveals conflicts and validation errors without changing anything.
Review the diff
Look at exactly what would change, in the same review step where humans approve the release.
Apply for real
Run the actual apply with the same manager name. If a conflict appears, stop and investigate before forcing.
Verify ownership
Inspect the ownership record occasionally to confirm that your fields belong to your manager as expected.
The dry run is more honest than the old one
The older client-side dry run only simulated the merge in the client. The server dry run asks the real API server, including its validation rules, defaulting, and admission checks, to do all the work without saving the result. It is therefore much closer to the truth of what will happen. Many teams make it a mandatory step in their pipelines.
If you leave the manager name at its default, every user and every pipeline that runs the same tool shares one identity. The ownership record then cannot tell them apart. A unique name per tool or pipeline is a tiny habit that makes the whole system explainable.
Permissions needed
An apply request is a kind of patch, so the caller needs permission to patch the object. If the object may not exist yet, apply can create it, which also requires permission to create. Server-Side Apply does not require any exotic permission, which makes it easy to adopt in clusters that already use the normal permission system.
11Migrating from Client-Side Apply
Most real clusters already contain thousands of objects created with the old method. Fortunately, Kubernetes provides a smooth path across.
What happens to existing objects
Objects created or last applied with client-side apply carry the saved annotation and a record from the client-side manager. When you apply such an object for the first time using server-side apply, the system recognizes the history. It converts the old record into proper ownership data, so fields you used to manage become fields your new manager owns. You do not need to recreate anything.
flowchart LR
OLD["Objects managed by client-side apply: annotation plus client-side manager record"] --> FIRST["First server-side apply with a chosen manager name"]
FIRST --> CONV["Server upgrades ownership data for the fields you declared"]
CONV --> NEW["Objects managed by server-side apply"]
NEW -. "optional: drop old annotation over time" .-> CLEAN["Cleaner object without the saved copy"]
A careful migration plan
Check your cluster version
Make sure the cluster is at a version where Server-Side Apply is generally available.
Choose manager names
Decide on a naming scheme before you start, so that every pipeline gets a distinct and meaningful name.
Try on a test environment
Switch one pipeline in a non-production cluster and review the diffs and ownership records.
Resolve the first conflicts
Expect some conflicts with autoscalers, controllers, or manual edits. Each is a chance to clarify who should own the field.
Roll out gradually
Move production pipelines one at a time, never mixing old and new styles for the same object in the same period.
Remove old assumptions
Retire scripts that rely on the saved annotation or on client-side merge behavior.
Things that surprise teams
Fields get removed
Because the old record is converted into ownership, a field present in the old record but missing from your new file may now be removed. Review diffs closely.
Full objects in files
Files exported from a live cluster contain fields owned by other parties. Applying them as-is claims ownership of fields that should belong to others.
Mixed tools
If one team uses the old method and another the new, their writes may follow different rules on the same object.
Going back
Switching back to the old method is possible, but the ownership data and saved annotation may no longer be in step. Plan carefully before reversing.
A very common source of trouble is saving a live object to a file, editing it, and applying it. That file includes fields set by controllers, defaults, and status. Applying it declares ownership of all of them. Write clean files that contain only the fields you intend to own.
Production example: staged migration at a large platform
Teams running many clusters commonly migrate in waves: first internal tooling namespaces, then non-critical applications, then production. They monitor conflict counts during each wave and treat a surge of conflicts as a sign that ownership boundaries were unclear, which is valuable information they then fix in documentation and files before proceeding.
12Server-Side Apply in Controllers and Operators
Controllers write to the cluster all day long. Server-Side Apply gives them a safer and simpler way to say what they want.
The old way for controllers
Traditionally, a controller that managed a child object would read the object, change a few fields in memory, and send the entire object back with an update, or compute a patch by hand. This required careful code to avoid overwriting fields set by others, and it required retries when the version stamp changed. Many subtle bugs came from this read-modify-write pattern.
The new way: declare and apply
With Server-Side Apply, a controller builds a small description of just the fields it cares about, called an apply configuration, and sends it with its field manager name. The server merges it. There is no need to read first, no need to retry because of version stamps, and no risk of accidentally resetting fields owned by someone else. The client libraries for the main programming languages provide typed helpers for building such configurations, and popular controller frameworks support apply as a first-class operation.
Benefits for controllers
- Simpler code with no read-modify-write loop.
- Fewer conflicts caused by stale reads.
- Automatic cleanup of fields the controller no longer sets.
- Clear, inspectable ownership of the controller’s fields.
Things to watch
- The controller must send a complete intent each time, not a partial delta.
- It should force ownership only for fields it truly manages.
- Large objects mean larger requests.
Good manager names for controllers
A controller should use one stable field manager name that identifies it, and use the same name on every run and every replica. If the name changes between versions, the new name starts as a separate owner, and the old name’s fields are left behind, which can cause confusing ghost fields. Treat the manager name as a permanent part of the controller’s identity.
Spec and status are separate
Controllers often write two different parts of an object: the spec of the child objects they create, and the status of their own custom resource. The status is usually written through a separate subresource, and the ownership record keeps track of the subresource in each entry. A good pattern is to apply status with its own clear manager name, so that status and spec ownership never get tangled together.
flowchart TD
CR["Custom resource written by a user"] --> CTRL["Controller reconcile loop"]
CTRL -->|"apply child objects with the controller's manager name"| CHILD["Deployments, Services, ConfigMaps"]
CTRL -->|"apply status through the status subresource"| ST["Status of the custom resource"]
OTHER["Other tools, autoscalers, humans"] -->|"update their own fields"| CHILD
If you write an Operator, think about which fields on the child objects you want to own, and which you want to leave free for users or other tools, such as replica counts managed by an autoscaler or labels added by a platform. Apply only the first group. This single design choice is what makes your Operator a good citizen.
Production example: Operator frameworks adopt apply
Popular Operator tooling now offers apply as a normal way to write objects, with options for naming the manager and forcing ownership. Authors who switch from read-modify-write to apply often delete large chunks of defensive code, and report fewer bugs caused by two controllers fighting over the same field.
13Sharing Fields Safely: Autoscalers and Hand-Overs
The replica count is the most famous shared field in Kubernetes. It is the perfect lesson in how to hand a field from one owner to another without breaking anything.
The classic conflict
A Deployment has a replica count in its spec. A horizontal pod autoscaler is designed to change that same number as load rises and falls. Meanwhile, your deployment file might also state a replica count. If your pipeline applies a fixed number every release, it fights the autoscaler. The autoscaler raises replicas to twenty, the next release sets them back to three, and the service struggles.
The simple solution
With Server-Side Apply, the clean answer is for the deployment file to not mention the replica count at all, once an autoscaler manages it. The pipeline then does not own the field, so it never overwrites the autoscaler’s choice. The autoscaler, which writes through the scale part of the object, owns the field. The two live together without conflict.
sequenceDiagram
participant P as Pipeline (apply)
participant H as Autoscaler (update)
participant S as API server
P->>S: Apply: image, labels (no replicas)
H->>S: Set replicas to 12 through scale
S-->>H: Stored, autoscaler owns replicas
P->>S: Apply new image on next release (no replicas)
S-->>P: Stored, replicas untouched at 12
The hand-over problem
The trap appears when you already have a file that mentions replicas, and you want to switch to autoscaling. If you simply delete the line from the file and apply, the server sees a field you used to own that you no longer mention. If nobody else owns it, the field is removed, and the default value takes over, usually one replica. Your service shrinks suddenly to a single copy before the autoscaler reacts. This is the sharp edge of declarative deletion.
A safe hand-over pattern
Let the autoscaler start owning the field first
Create the autoscaler and let it set the replica count, so that it becomes an owner of the field while your pipeline still shares it.
Confirm the ownership record
Inspect the object and check that the autoscaler appears as an owner of the replica field.
Remove the line from your file
Apply the updated file. Because another owner exists, the field stays, and your pipeline simply gives up its share.
Watch for stability
Observe the next releases. The replica count should stay under the autoscaler’s control without sudden jumps.
The exact steps for transferring ownership of a field are described in the Kubernetes documentation and have been refined over time. Treat the pattern here as the idea, and confirm the details against the documentation for your cluster version before running it on production.
Other fields that are commonly shared
Added by several tools
Pipelines, cost tools, and security scanners may each add their own labels. Distinct keys keep them apart.
Hints for controllers
Ingress controllers, meshes, and backup tools read annotations that different teams may set separately.
Sidecars from platforms
A platform may add helper containers to every workload. Keyed list merging lets them coexist with yours.
Vertical autoscaling
A tool that adjusts CPU and memory may own those fields. Your file should then not set them, or should coordinate with the tool.
14Server-Side Apply and GitOps Tools
GitOps tools continuously compare the cluster with a Git repository and apply the difference. They are among the heaviest users of Server-Side Apply, and they show its strengths very clearly.
What GitOps does
A GitOps tool, such as Flux or Argo CD, watches a repository where your configuration files live. When the files change, the tool applies them to the cluster. It also periodically checks whether the cluster has drifted from the repository, and can correct the drift. This is exactly the repeated, declarative apply that Server-Side Apply was built for.
Why GitOps benefits
Gains from server-side apply
- No giant saved annotation on every object.
- Fewer false drift reports caused by fields set by controllers.
- Better handling of custom resources with proper list rules.
- Clear conflict messages when someone edits by hand.
New things to manage
- Ownership conflicts with Operators, autoscalers, and meshes.
- Need to decide which fields Git should own.
- Larger ownership records on objects.
The ownership split that works
The healthiest GitOps setups follow a simple rule: Git declares only the fields humans decide, and everything else is left to the cluster’s other actors. That means files should contain the intended spec, and not fields generated by controllers, not status, and not values owned by autoscalers. When a conflict appears, the right fix is usually to remove the disputed field from Git rather than forcing it in.
flowchart TD
GIT["Git repository: human-decided fields only"] --> GO["GitOps tool applies with its own manager name"]
GO --> OBJ["Cluster object"]
AS["Autoscaler updates replicas"] --> OBJ
OP["Operator applies generated fields and status"] --> OBJ
MESH["Platform injects sidecar fields"] --> OBJ
OBJ --> DRIFT["Drift check compares only fields Git owns"]
Handling drift without fighting
Drift means the cluster differs from Git. With ownership tracking, a GitOps tool can limit its drift checks to the fields it owns. A change to a field owned by someone else is not drift from Git’s point of view, because Git never claimed it. This greatly reduces noisy alerts and pointless corrections. Where a tool does still report differences in fields it should not manage, you can configure ignore rules for those specific paths.
When your GitOps tool reports a conflict, ask: “Whose field is this really?” If it is the Operator’s or autoscaler’s, remove it from Git. If it is Git’s, find and stop the other writer. Avoid the shortcut of forcing ownership everywhere.
Production example: Flux and Argo CD options
Flux uses server-side apply for its reconciliations, and Argo CD offers a server-side apply option that teams enable per application, especially for very large objects or custom resources. Both tools document how to ignore specific fields and how to handle conflicts, which makes them practical places to learn the ownership model on real workloads.
15Helm, Kustomize, and CI Pipelines
Most teams do not write raw files by hand. They generate them with templating and overlay tools, then push them through pipelines. Server-Side Apply fits into each of these, but each has its own habits worth understanding.
Where the tools fit
| Tool | How it normally applies changes | Relationship to server-side apply |
|---|---|---|
| Helm | Computes its own three-way comparison between the previous release, the new release, and the live object, then sends patches | Helm keeps its own release history and does not use field ownership by default. Newer versions of Helm have added options to use server-side apply |
| Kustomize | Builds final files from a base and overlays, then hands them to kubectl | Works well, because the output is simply a set of files you can apply server-side |
| Plain kubectl in a pipeline | Applies files directly | Fully supports server-side apply with a unique manager name |
| GitOps controllers | Apply from Git on a loop | Often use server-side apply, as the last chapter explained |
Imagine a print shop. Kustomize and templating tools are the designers who prepare the final artwork. The apply step is the printing press. Server-Side Apply is a press that stamps a small signature in the corner of every section it prints, so anyone later can see which designer’s work each section came from.
Helm and ownership
Helm tracks what it installed through release records. When Helm updates a release, it compares the old and new versions of the chart output with the live object, and sends changes. This works well for many cases, but Helm does not give each field an owner in the way Server-Side Apply does. If another controller changes a field that Helm also manages, Helm may overwrite it during the next upgrade, or leave it, depending on the details. Because Helm’s behavior around server-side apply has been evolving, check the documentation for your version of Helm before relying on any specific option.
If one pipeline installs an object with Helm, and another applies changes to the same object with Server-Side Apply, the two do not share a common idea of who owns what. Choose one tool per object, or define very clearly which fields each tool may touch.
Kustomize and overlays
Kustomize creates the final files by layering small changes on top of a shared base. Since it only produces files, it does not conflict with Server-Side Apply. In fact, the combination is pleasant: each environment’s overlay declares only the fields that differ, and the apply step records ownership of exactly those fields. Be careful, though, that overlays do not accidentally include fields that should belong to another system, such as the replica count when an autoscaler is in use.
Designing a pipeline that behaves well
Generate clean files
Render the final files from templates or overlays, and keep only the fields your pipeline intends to own.
Validate
Run schema validation and a server dry run, which also reports ownership conflicts early.
Show the diff for approval
Present the exact difference to a human reviewer for sensitive environments.
Apply with a stable manager name
Use the same name for the same pipeline every time, and a different name for each separate pipeline.
Fail loudly on conflicts
Do not force by default. A conflict is a signal that two parties disagree, and a human should look at it.
Production example: separate pipelines for separate concerns
Larger organizations often split their delivery into several pipelines: one for application workloads, one for platform add-ons, and one for security policy. Giving each its own manager name makes the ownership record read like an organization chart. When something unexpected changes in production, an engineer can read the record and immediately know which team’s pipeline to ask.
16Custom Resources and Merge Hints
Built-in objects come with merge rules already attached. Custom resources, the objects defined by Operators and extensions, need those rules to be declared by their authors. This is where Server-Side Apply shines most.
Why custom resources need hints
The server understands the shape of built-in objects and knows, for instance, that the container list in a pod is keyed by container name. For a custom resource, the server only knows what the author wrote in the schema. If the author says nothing about a list, the server treats the whole list as one unit, which means that two owners cannot manage different items. Authors can add small hints to their schema that describe how lists and maps should be merged.
The three hints in plain words
Atomic list
The whole list is one value. Whoever applies last replaces the entire list. Safe, simple, and the default.
Set list
A list of simple values, like names, where each value is unique. Different owners can add different values.
Map list
A list of objects identified by one or more key fields. Each item is merged by its key, so owners can manage different items and different fields in the same item.
Granular or atomic map
An object whose fields can be owned separately, or one that is treated as a single value. Authors choose depending on whether the fields make sense on their own.
Think of a shared recipe card. An atomic list is like a card where the ingredients are written on a single sticker. Anyone who changes it replaces the whole sticker. A map list is like a card where each ingredient has its own labeled line, so one cook can change the salt line while another changes the butter line, with no clash.
How authors choose
Choose a keyed list when
- Items have a natural unique name, such as a port name or a rule name.
- Different tools may add or edit different items.
- Items are independent of each other.
Choose an atomic list when
- The order matters, or the items only make sense together.
- There is no stable key.
- Partial edits would produce invalid combinations.
flowchart TD
Q1{"Do items have a stable unique key?"} -->|"yes"| MAP["Use a keyed map list"]
Q1 -->|"no"| Q2{"Are items simple unique values?"}
Q2 -->|"yes"| SET["Use a set list"]
Q2 -->|"no"| ATOM["Use an atomic list"]
MAP --> NOTE["Owners can edit different items and fields"]
SET --> NOTE2["Owners can add different values"]
ATOM --> NOTE3["Last applier replaces the whole list"]
Switching a list from atomic to keyed after users already depend on it can change how existing ownership records are interpreted. Decide early, test with two writers, and read the documentation for CRD schema changes before altering a published API.
Production example: shared lists in platform APIs
Gateway and networking APIs in the Kubernetes ecosystem have lists of listeners and routes, where different teams need to add their own entries to one shared object. Keyed lists make this possible, so an application team can add a route entry without touching entries owned by the platform team.
17Admission Webhooks, Defaults, and Mutation
Not every change to an object comes from a user or a controller. Some come from the API server itself, as it fills in defaults, and from webhooks that modify objects on the way in. Understanding these helps explain surprising ownership entries.
The journey of a request
When an apply request arrives, the server authenticates and authorizes it, merges it with the stored object, applies defaults for unset fields, runs any mutating webhooks, validates the result, runs validating webhooks, and finally stores it. The ownership record is updated through this process, so it is useful to know where each field’s value came from.
sequenceDiagram
participant C as Client
participant A as API server
participant M as Mutating webhook
participant V as Validating webhook
participant E as Storage
C->>A: Apply request with manager name
A->>A: Authenticate and authorize
A->>A: Merge with stored object, update ownership
A->>A: Apply defaults
A->>M: Send object for mutation
M-->>A: Return changes
A->>A: Validate schema
A->>V: Send object for approval
V-->>A: Allow or deny
A->>E: Store object
A-->>C: Return result
Defaults and ownership
Default values are not normally owned by the person who sent the request. A defaulted field simply appears in the stored object without a user claiming it. If you later write a value for that field, you become its owner. This is why a freshly applied object can contain many fields that your file never mentioned, and why those fields do not appear under your manager’s name.
Mutating webhooks and the sidecar story
A mutating webhook can change an object as it passes through. A well-known example is a service mesh adding a helper container to every pod. These changes happen inside the server’s own flow. When a webhook changes a field, the ownership record is typically updated so that the change is attributed to the request’s manager, unless the webhook operates in a way that avoids it. Since behavior can depend on version and webhook design, it is wise to test with a small object and read the ownership record before depending on any particular outcome.
A webhook that rewrites a field on every request can cause your pipeline to see a difference on every run. If your file sets a value and the webhook changes it, each apply will appear to change the object again. The fix is to stop declaring that field, or to adjust the webhook so it respects values that are already set.
Idempotent mutation
A good mutating webhook is idempotent, meaning that running it twice gives the same result as running it once. Idempotent webhooks keep apply requests stable, because repeated applies do not keep adding new items or flipping values. This is a quality bar worth checking in any webhook your platform installs.
Production example: sidecar injection
Service meshes such as Istio and Linkerd add proxy containers to workloads through webhooks. Teams that move their pipelines to Server-Side Apply commonly find that their files should never list the injected container, because the keyed container list lets the mesh’s entry sit next to the application’s own entry without conflict.
18Security, Permissions, and Auditing
Because Server-Side Apply records who changed what, it is tempting to treat the record as a security feature. It is helpful, but it is not a security boundary. This chapter draws that line carefully.
The manager name is a label, not a lock
The manager name is chosen by the caller. Anyone who can send an apply request can use any name they like. The server does not check that the name matches the real identity of the caller. Therefore, ownership records are useful for coordination and debugging, but they cannot be trusted to prove who really made a change. For proof, use the audit log, which records the authenticated identity of each request.
| Question | Best source | Why |
|---|---|---|
| Which tool probably owns this field? | Ownership record | Fast, visible on the object, designed for coordination |
| Who really sent this request? | Audit log | Records the authenticated user or service account |
| Who is allowed to change this object? | Role-based access control | Enforced by the server before any merge happens |
| Which values are allowed? | Admission policy | Enforced on the final merged object |
The manager name is like the name someone writes on a sticky note they leave on a shared fridge. It is handy for knowing who to ask about the leftover soup, but anyone can write any name. The building’s security camera, which is the audit log, shows who actually opened the door.
Permissions
Apply is sent as a patch, so permission to patch an object is what authorizes it. Roles that grant patch on an object type therefore also permit apply. Some clusters use fine-grained rules that limit patching to certain object names or namespaces. Review roles for automation accounts, since a pipeline that can patch can also change any field of the objects it can reach, even when its own file mentions only a few.
The force option as a privilege
Forcing ownership is not a separate permission. Anyone who can apply can also force. This means that a careless script could take fields from a security controller or an autoscaler. Control this through review of pipeline definitions, through admission policies that protect critical fields, and through alerts on unexpected ownership changes.
Good security habits
- Treat the audit log as the source of truth for identity.
- Give each automation its own service account and manager name.
- Use admission policy to protect fields that must never change.
- Alert when ownership of critical fields moves.
Common mistakes
- Trusting the manager name as proof of identity.
- Sharing one service account across many pipelines.
- Granting broad patch rights to every automation.
- Forcing conflicts by default in scripts.
During an investigation, use the ownership record to find which pipeline probably wrote a field, then use the audit log to confirm the real identity and time. Together they give a fast and reliable picture.
Production example: protecting critical fields
Security-conscious teams use policy engines such as Kyverno or Open Policy Agent Gatekeeper to refuse changes to critical fields, such as privileged settings on pods, no matter which manager name or tool sends the request. The policy checks the final object, so a forced apply cannot slip past it.
19Performance, Size, and Scale
Ownership tracking is not free. Every object carries extra data, and every apply request requires extra work on the server. For most clusters the cost is small, but at large scale it deserves attention.
Where the cost comes from
The ownership record lists every field a manager owns, so objects with many fields and many managers carry larger records. For a small configuration object, this is negligible. For a very large custom resource with thousands of fields, or an object touched by dozens of different managers, the record can become a noticeable share of the object’s size. The storage system behind Kubernetes has a limit on the size of any single object, so extremely large objects need care.
Many managers, many entries
Each distinct manager name creates its own entry. A problem appears when tools generate a new manager name on every run, for example by including a random suffix or timestamp. The record then fills with abandoned entries that no longer correspond to any active tool. Always use stable names, so that the same tool reuses the same entry.
A pipeline that includes a build number in its manager name creates a new owner on every run. Fields stay owned by the old names, and new runs cannot remove them. Over time the object grows and conflicts become confusing. Keep names stable and meaningful.
Request cost
Because the server does the merging, large apply requests cost server work. A GitOps tool that applies thousands of objects on a short loop can add noticeable load. Practical remedies include longer reconcile intervals, skipping objects that have not changed, and splitting very large objects. These are tuning choices, and the right values depend on your cluster size and tools.
A short checklist for scale
Use stable manager names
Prevent endless growth of ownership entries.
Keep files minimal
Declare only the fields you own, which keeps both requests and records small.
Watch object sizes
Track the size of your largest custom resources and alert as they approach storage limits.
Tune reconcile loops
Avoid applying unchanged objects at high frequency.
Measure before optimizing
Use the API server’s own metrics to see whether apply requests are actually a bottleneck.
Production example: avoiding the annotation limit
One of the original motivations for Server-Side Apply was that very large custom resources could not be applied with the old method, because the saved copy in the annotation exceeded its size limit. Teams that manage large resources, such as big monitoring or policy definitions, often adopt server-side apply precisely to escape that limit.
20Debugging and Observability
When something unexpected happens to an object, you want answers quickly: who changed it, when, and why did my change not stick? The ownership record is the best place to begin.
Reading the ownership record
By default, most commands hide the ownership record to keep the output readable. You can ask to see it. For each entry you will find the manager name, whether the operation was an apply or an update, the time of the last change, the version of the API used, the part of the object it covers if it is a subresource, and the list of fields owned. Read it from top to bottom like a ledger.
| Symptom | Likely cause | First thing to check |
|---|---|---|
| Apply returns a conflict | Another manager owns a field with a different value | The conflict message names the manager and the field |
| My change disappears after a while | Another controller or pipeline overwrites it | The ownership entries and their timestamps |
| A field I removed from my file is still there | Another manager also owns it | Whether a second owner is listed for that field |
| A field vanished after I removed it | I was the only owner | Apply history, and whether a hand-over was needed |
| Object keeps changing with no human action | Two automations fight in a loop | Timestamps and manager names that alternate |
| Object seems very large | Many stale manager entries | Count of entries and whether names are stable |
A debugging method
Reproduce with a server dry run
Run the apply in server dry-run mode to see the conflict or validation message without changing anything.
Read the ledger
Display the ownership record and find the entry that owns the field in question.
Match the name to a tool
Use your naming scheme to identify which pipeline or controller the manager name belongs to.
Confirm with the audit log
If the identity matters, check the audit log for the real caller and time of the last change.
Decide the true owner
Choose which party should own the field, then change the other party’s file or code to stop declaring it.
Debugging with the ownership record is like checking the edit history of a shared spreadsheet. You do not argue about who is right. You open the history, see which colleague last changed the cell and when, and then talk to that person.
Spotting a tug of war
A telltale sign of two systems fighting is an object whose modification time keeps changing, while the same fields flip between values. If the ownership record shows two managers alternating, you have found the pair. Fix the root cause by assigning each field to exactly one of them.
What to monitor
Conflict rate
Count of apply requests that were rejected for conflicts. A sudden rise suggests a new overlap.
Request latency
Time taken by apply requests, especially for large objects.
Object size
Size of the biggest custom resources, as a share of the storage limit.
Forced applies
Pipelines using the force option, since each is a possible ownership takeover.
Debugging is easy when manager names follow a rule such as team, tool, and purpose. It is painful when names are random. Document the rule in your platform handbook and enforce it in pipeline templates.
21A Worked Example: Team, Autoscaler, and Mesh
Let us walk through a realistic story from beginning to end, using no code, only words. It brings together ownership, sharing, and conflicts.
The cast
- The application team deploys a web service named “shop” through a pipeline with the manager name “shop-deploy”.
- The autoscaler adjusts the replica count according to traffic.
- The service mesh adds a proxy container through a webhook.
- The platform team adds a cost-center label through a separate pipeline named “platform-labels”.
The story in five scenes
First release
The team applies a file with the application container, its image, its port, and a replica count of three. The pipeline owns all those fields. The server also fills in many defaults, which no one owns.
The mesh joins in
The mesh webhook adds a proxy container to the pod template. Because containers are a keyed list, the proxy entry sits beside the application entry, and each has its own owner.
The platform adds a label
The platform pipeline applies a small file with only one label key. It now owns that single label, and no one else’s labels are touched.
Autoscaling arrives
The team adds an autoscaler. They first let it take part in ownership of the replica field, then remove the replica line from their file. The service continues running at the autoscaler’s chosen size, with no sudden drop.
A new image
A later release changes only the image. The pipeline’s file mentions the image and port, the server merges, and every other owner’s fields remain untouched.
flowchart LR
subgraph OBJ["Deployment: shop"]
F1["image, port: owned by shop-deploy"]
F2["replicas: owned by autoscaler"]
F3["proxy container: owned by mesh webhook"]
F4["cost-center label: owned by platform-labels"]
F5["defaults: no owner until someone sets them"]
end
P1["Application pipeline"] --> F1
P2["Autoscaler"] --> F2
P3["Mesh webhook"] --> F3
P4["Platform pipeline"] --> F4
What would have gone wrong before
With the old method, the team’s release would have overwritten the replica count with three on every deployment, undoing the autoscaler’s work. The mesh’s proxy container might have been removed on the next apply, because the saved copy did not mention it. And the platform label would have required a risky manual edit. Server-Side Apply turns each of these hazards into a calm, explainable situation.
The one conflict that could still happen
Suppose a developer, in a hurry, edits the cost-center label by hand with an update command. Updates do not raise conflicts. They silently take ownership. The platform pipeline’s next apply would then report a conflict, because the developer and platform now disagree on the label’s value. This is the system working properly: it raises a flag instead of silently choosing a winner. The platform team can then decide whether to force the label back or discuss with the developer.
Server-Side Apply does not remove disagreements. It makes them visible, specific, and local to one field, so they can be resolved calmly.
22An Incident Walkthrough: Two Pipelines at War
The following story is a composite of the kind of incident many teams experience in their first months with Server-Side Apply. It is fictional, and its purpose is to teach the pattern of diagnosis.
The symptom
A company runs a payment service. Every few minutes, the number of running copies drops from twelve to three and then climbs back. Customers see slow responses during the dips. The on-call engineer finds that nothing is crashing. The service is simply being resized over and over.
The investigation
Look at the object’s history
The engineer displays the ownership record and notices two managers that both list the replica field: the autoscaler and a GitOps tool.
Check timestamps
The two entries show alternating times, a few minutes apart. The pattern matches the dips exactly.
Read the GitOps configuration
The repository still contains a replica count of three, left over from before autoscaling was introduced. Someone had also enabled the force option to silence a conflict message weeks earlier.
Understand the loop
Every reconcile, the GitOps tool forced the value three. The autoscaler then noticed low capacity and raised the value. The cycle repeated.
sequenceDiagram
participant G as GitOps tool (force on)
participant S as API server
participant H as Autoscaler
G->>S: Apply replicas 3 (forced)
S-->>G: Accepted, GitOps owns replicas
H->>S: Set replicas 12 (update)
S-->>H: Accepted, autoscaler owns replicas
G->>S: Apply replicas 3 again (forced)
S-->>G: Accepted, ownership flips back
H->>S: Set replicas 12 again
The fix
The team removed the replica count from the repository, following the hand-over pattern from the earlier chapter. They first confirmed that the autoscaler appeared as an owner, then removed the line. They also removed the force option from the GitOps configuration, so that any future overlap would surface as a visible conflict rather than a silent tug of war.
What the team learned
Lessons that worked
- Read the ownership record first.
- Alternating timestamps reveal a fight.
- Remove the disputed field from the party that should not own it.
- Keep the force option off by default.
Mistakes to avoid
- Forcing to silence a conflict message without asking why it exists.
- Leaving old values in the repository after adopting an autoscaler.
- Assuming a stable system when no pod is crashing.
When a tool reports a conflict, it is telling you that two parties disagree. Silencing it with force removes the message but not the disagreement. The disagreement then shows up later as strange behavior in production.
23Best Practices
The lessons of the previous chapters can be gathered into a short list of habits. Teams that follow them report that Server-Side Apply becomes almost invisible, working quietly in the background.
One stable name per tool
Use a clear, consistent manager name for every pipeline and controller. Never include run numbers or random strings.
Declare only what you own
Keep files small. Do not paste in exported objects with defaults, status, or fields set by others.
Do not force by default
Treat each conflict as a conversation to have. Force only when the takeover is deliberate and understood.
Always dry-run on the server
Include a server dry run and a diff in every pipeline, before the real apply.
Plan hand-overs
When a field moves from one owner to another, make sure the new owner is in place before the old one lets go.
Add merge hints to custom resources
If you write an API, mark lists with the right strategy and test with two writers.
Use audit logs for identity
Never rely on manager names as proof of who did something.
Monitor conflicts and size
Watch conflict rates, forced applies, and object sizes, and review them regularly.
A one-page team agreement
Many teams find it useful to write a short agreement and keep it with their platform documentation. It might say: every automation has a unique manager name following a documented pattern; every file declares only owned fields; no automation forces conflicts without a review; and every shared field has a named primary owner. A one-page agreement prevents most of the disputes that would otherwise occur.
Production example: ownership maps in platform handbooks
Platform teams at larger companies often keep an ownership map for their most important objects, such as the standard Deployment template. The map lists which fields belong to application teams, which to the platform, and which to automation. New engineers learn the map in their first week, and it matches what the ownership record shows in the live cluster.
24Anti-Patterns and Decision Records
An anti-pattern is a habit that looks sensible but leads to trouble. Here are the most common ones, written as short decision records so you can see the problem, the reason people fall into it, and the better choice.
Why teams do it
A conflict message blocks a release, and forcing makes the message go away immediately.
What goes wrong
Ownership flips between parties, and hidden fights begin. The real disagreement is never solved.
Better choice
Investigate each conflict, decide the true owner, and change the other party’s file or code.
Why teams do it
Exporting a working object feels like the quickest way to get a correct file.
What goes wrong
The file claims ownership of defaults, status, and other parties’ fields, creating conflicts and fights later.
Better choice
Write minimal files by hand or from templates, listing only fields you intend to own.
Why teams do it
Names are generated automatically from build numbers or tool defaults.
What goes wrong
Old names keep ownership of fields, new runs cannot remove them, and the record grows without limit.
Better choice
Pick one stable, documented name per pipeline and never change it casually.
Why teams do it
Different teams adopt the new method at different speeds.
What goes wrong
The two methods follow different rules about merging and removal, so changes appear or vanish unexpectedly.
Better choice
Migrate each object completely, one pipeline at a time, and retire the old method for that object.
Why teams do it
The record looks like an audit trail, so it is easy to treat as one.
What goes wrong
Any caller can choose any name, so the record can be misleading in an investigation.
Better choice
Use the audit log for identity and the ownership record for coordination.
Why teams do it
It seems natural to delete a line from a file when you no longer want to manage it.
What goes wrong
If you were the only owner, the server removes the field, and the default value returns, such as one replica.
Better choice
Make sure another owner holds the field before you stop declaring it, or accept the removal knowingly.
25Managed Kubernetes and Cloud Add-ons
On managed services from the big cloud providers, many components inside your cluster are installed and updated by the provider. Those components are writers too, and Server-Side Apply is how you and the provider avoid stepping on each other.
Who else writes to your cluster
On a managed cluster, the provider often installs networking, DNS, and storage components, and keeps them up to date. When you customize one of them, for example by adjusting the resources given to the cluster’s DNS pods, you become a second writer on an object the provider also manages. Without ownership tracking, the next provider upgrade could silently erase your change, or your change could silently break the upgrade.
How providers use ownership
Amazon’s managed Kubernetes service, for example, offers managed add-ons and lets you choose how conflicts are resolved when you have customized an add-on’s settings. Typical choices are to overwrite your changes with the provider’s values, to preserve your changes, or to do nothing and report the problem. Behind these options is the same idea: fields are owned by a party, and the provider’s tool must decide what to do when you own a field it wants to change. Other providers offer similar behavior for their own managed components. Option names and defaults vary by provider and version, so confirm them in the current documentation.
| Provider policy | What happens to your customized fields | Best used when |
|---|---|---|
| Overwrite | The provider’s values replace yours | You want the provider’s standard setup and have no real customizations |
| Preserve | Your values are kept, and the provider changes only other fields | You have deliberate customizations that must survive upgrades |
| Fail or report | The upgrade stops and shows the conflict | You want a human to review every disagreement |
Think of renting a furnished apartment. The landlord maintains the building, but you hang your own pictures. When the landlord repaints, a good agreement says whether your pictures should be taken down, left in place, or whether the painter should stop and ask you first.
Practical advice for managed clusters
Know what the provider owns
Read the ownership record on provider-installed objects before changing them.
Customize through supported settings
Prefer the provider’s configuration options over raw edits of the objects.
Use a named manager for your changes
If you must edit, apply with a clear name so your fields can be recognized later.
Test upgrades in a staging cluster
Confirm what happens to your customizations before the same upgrade reaches production.
Production example: customizing a cluster DNS add-on
A common case is raising the resources or replica count of the cluster’s DNS component for a busy cluster. Teams that apply this change under a named manager, and choose the preserve policy, find that provider upgrades keep their tuning. Teams that edit by hand and choose overwrite sometimes find their tuning reset after an upgrade, and the cluster slows down again.
26A Decision Guide: Should You Use It, and How?
Server-Side Apply is the recommended direction for new work, but a few questions help you decide how to adopt it and how to configure each writer.
flowchart TD
S["Who is writing this object?"] --> Q1{"Is it a controller or Operator?"}
Q1 -->|"yes"| C1["Apply with one stable manager name; apply only the fields it manages; force only its own fields"]
Q1 -->|"no"| Q2{"Is it an automated pipeline?"}
Q2 -->|"yes"| C2["Apply with a unique manager name; dry-run first; do not force by default"]
Q2 -->|"no"| Q3{"Is it a person making a one-off change?"}
Q3 -->|"yes"| C3["Prefer a change through the pipeline; if urgent, apply with your own name and tell the owner"]
Q3 -->|"no"| C4["Review who the writer is and why"]
C1 --> E["Does another writer share a field?"]
C2 --> E
C3 --> E
E -->|"yes"| H["Decide one primary owner and plan a hand-over"]
E -->|"no"| OK["Proceed"]
A comparison of the two apply methods
| Question | Client-side apply | Server-side apply |
|---|---|---|
| Where is the merge computed? | In the client tool | On the API server |
| How is ownership tracked? | Through a saved copy in an annotation | Through a field-level record in the object’s metadata |
| What happens on disagreement? | The last writer wins silently | A conflict is reported to the applier |
| Handling of custom resources | Weak for lists | Strong, guided by schema hints |
| Size limits from tracking | Annotation limit applies | No annotation, but ownership data adds size |
| Best fit | Simple, single-writer setups | Shared objects, controllers, GitOps |
When the old method is still fine
If a small team manages a handful of simple objects, with a single pipeline that is the only writer, the old method may serve perfectly well, and there is little pressure to move. The strongest reasons to adopt Server-Side Apply are: several writers on the same objects, controllers that need safe writes, custom resources with lists, very large objects, and a wish for clear, field-level accountability.
Do not migrate everything on day one. Begin with the objects that already cause confusion, such as those shared with an autoscaler or an Operator, and learn from them before moving the rest.
27A Learning Path for Your Team
Reading about ownership is useful, but the ideas become natural only through practice. This short path turns the chapters above into a plan any team can follow over a few weeks.
Week one: observe
Display the ownership record of a few ordinary objects in a test cluster. Learn to read manager names, operations, and field lists without changing anything.
Week two: create a harmless conflict
Apply the same object from two different manager names with different values. Watch the refusal message, then resolve it in each of the three ways: change the value, remove the field, and force.
Week three: practice a hand-over
Set up an autoscaler on a test workload and move the replica field from your file to the autoscaler without any dip in running copies.
Week four: switch one real pipeline
Pick one low-risk pipeline, choose its manager name, add a server dry run and a diff, and move it across with the team watching.
Afterwards: write the agreement
Record your naming rule, your rule about forcing, and your list of shared fields in one page of platform documentation.
Use a throwaway namespace or a small local cluster for the exercises. Seeing a conflict and a field removal with no consequences is the fastest way to build the confidence you will want when the same thing happens in production.
28Frequently Asked Questions
These are the questions people ask most often when they first meet Server-Side Apply.
29Summary and Key Takeaways
We began with a shared document that many people edit and nobody tracks. Kubernetes objects are exactly like that, and Server-Side Apply is the feature that adds the missing track changes.
What to remember
- Ownership is recorded per field. The object’s metadata lists each manager and the exact fields it owns, so the question “who set this?” always has an answer.
- The server does the merging. You send only the fields you care about, and built-in or schema-defined rules decide how maps and lists combine.
- Conflicts are a feature. An apply that disagrees with another owner is refused with a clear message, instead of silently winning. Updates are the exception, since they take ownership quietly.
- Leaving a field out means letting it go. If you were the only owner, the field is removed. Plan a hand-over before you stop declaring a shared field, such as replicas under an autoscaler.
- Use stable, unique manager names and avoid force by default. Controllers may force their own fields, but pipelines and people should investigate conflicts first.
- The manager name is not a security boundary. Use the audit log and access control for identity and protection, and the ownership record for coordination.
- Adopt it where the pain is. Controllers, GitOps, custom resources, and shared objects benefit most, and simple single-writer setups can move at their own pace.