Server-Side Apply in Kubernetes

Server-Side Apply in Kubernetes

How the cluster itself keeps track of who owns every field of every object, why that ends the old fights between tools and teams, and how to use it safely, explained from zero.

Imagine a shared document that ten people edit at the same time. One person fixes the title. Another changes a paragraph. A third adds a table. Now imagine that every time someone saves, they upload their whole copy and overwrite everyone else’s work. By lunchtime, nobody can trust what the document says, and everyone blames everyone else. Sensible tools solve this by tracking who changed which part, so that two people editing different paragraphs never collide, and two people editing the same sentence are told about it before anything is lost.

A Kubernetes object is exactly such a shared document. A developer edits it by hand. A deployment pipeline updates it. An autoscaler changes the replica count. A security tool adds a sidecar. An Operator writes the status. For many years, Kubernetes had no good answer for how all of these hands should share one object. Server-Side Apply, often shortened to SSA, is the answer. It moves the job of tracking ownership and merging changes from each client tool into the Kubernetes API server itself. This post explains the idea from the very beginning, assuming you know nothing yet. By the end, you will understand field ownership, conflicts, merge rules, migration, and the way modern tools use SSA, and you will be able to explain it clearly in design reviews and interviews.

1The Problem: Many Hands on One Object

To appreciate the solution, we first need to feel the problem. Let us start with what an object is and who touches it.

What a Kubernetes object looks like

Everything in Kubernetes is stored as an object: a Deployment, a Service, a ConfigMap, a custom resource of your own. Each object is a structured document made of fields. Some fields describe what you want, such as the container image or the number of copies. Other fields describe what the cluster observed, such as how many copies are ready. The whole document is kept in the cluster’s database, and anyone with permission can read or change it.

Who edits the same object

In a real cluster, a single object is rarely touched by just one party. Consider an ordinary Deployment for a web application. A developer commits a change to the image version, and a pipeline applies it. A horizontal autoscaler adjusts the replica count when traffic rises. A platform tool might add a standard label or a monitoring sidecar. An engineer, in the middle of an incident, uses a quick command to bump a resource limit. Each of these actors is correct to change its own part. The trouble starts when their changes overlap or overwrite each other.

Everyday analogy

Think of a kitchen shared by several cooks. One owns the soup, one owns the bread, one owns the salad. Everyone works fine as long as each cook keeps to their own station. Chaos begins when a new cook walks in, throws out the whole counter to “start clean”, and rebuilds it from a recipe card. The soup is gone, and nobody knows why.

The classic symptoms

Symptom

Reset replicas

A pipeline applies the full file, which says “three replicas”, and undoes the autoscaler’s decision to run twenty.

Symptom

Vanishing sidecars

A tool that added a helper container finds it deleted by the next deployment because the pipeline did not know about it.

Symptom

Hotfix lost

An engineer fixes a limit by hand during an outage, and the next pipeline run silently reverts it.

Symptom

Stale deletions

A field removed from the source file stays on the cluster forever, because the tool never knew it should delete it.

5+
different actors often edit one Deployment
1
shared object with no built-in memory of who did what
GA
Server-Side Apply is generally available since Kubernetes 1.22
i
The core idea in one sentence

Server-Side Apply lets the cluster remember which party owns which field, so that parties can safely share one object, collisions are detected instead of hidden, and each party can cleanly remove only what it added.

2How Kubernetes Changes Objects Today

Before Server-Side Apply makes sense, you need a clear picture of the ordinary ways to change an object, and what each one risks.

The API server and the database

All changes go through the API server, the front door of the cluster. It checks who you are, checks you are allowed to do the action, validates the change, and stores the result in the cluster database. Every client, whether it is a command-line tool, a pipeline, or a controller, speaks to the same API server, using ordinary web requests.

The basic ways to write

OperationPlain-English meaningMain risk
CreateMake a new objectFails if it already exists
Update (replace)Send the whole object and overwrite what is storedWipes out fields you did not know about
PatchSend only the pieces you want to changeMerge rules vary by patch type and may surprise you
DeleteRemove the objectObvious, but often combined with ownership and cleanup rules

Optimistic concurrency

Every object carries a version stamp called the resource version. When you replace an object, you send back the version you last read. If someone else changed the object in the meantime, the stamp no longer matches, and the API server rejects your write with a conflict. You then re-read and try again. This protects against lost updates but only at the level of the whole object. It cannot tell the difference between a harmless change to a different field and a harmful overwrite of the same field.

Everyday analogy

The resource version is like a “last edited” number written on the top of a paper form. If two people take the form from the office and bring it back, the second person is told “the form has changed since you took it”. That is helpful, but it does not say which line changed, so the second person must start over even when the changes would not have clashed at all.

The types of patches

Kubernetes supports several ways to patch an object. A JSON patch is a list of precise edits by position, which is fragile if the list changes underneath you. A JSON merge patch overlays a partial document, but replaces whole lists in one go. A strategic merge patch understands Kubernetes lists by using special keys, for example merging containers by name, but it only works for built-in types and needs special client knowledge. Each is useful, yet none of them records who wrote which field.

sequenceDiagram
  participant A as Pipeline
  participant S as API server
  participant H as Autoscaler
  A->>S: Read Deployment (version 100, replicas 3)
  H->>S: Set replicas to 20 (version becomes 101)
  A->>S: Replace whole object using version 100
  S-->>A: Conflict: the object changed
  A->>S: Read again (replicas 20) and retry
  A->>S: Replace whole object with replicas 3
  S-->>A: Accepted, autoscaler decision lost
    
Figure 1: Whole-object conflict detection catches the first collision, but a careless retry still erases another actor’s field
!
What is missing

None of these basic operations know who owns a field. The API server treats each write as an anonymous change. So the burden of being polite falls on every client, and clients differ in how careful they are.

3Client-Side Apply and Its Limits

Before Server-Side Apply, almost everyone used a different “apply” that ran entirely inside the command-line tool. Understanding how it worked explains exactly why a new design was needed.

What “apply” means

The command many people know as kubectl apply is built around a declarative idea: you keep your desired configuration in a file, and you tell the cluster, “make it look like this”. If the object does not exist, apply creates it. If it exists, apply works out the differences and updates it. You can run the same command again and again safely.

How the old method worked

The original method is called client-side apply, because the comparison happens on the client. To work out what to change, the client uses a three-way merge with three inputs: the new file you provided, the live object currently in the cluster, and a saved copy of the file you applied the last time. That saved copy is stored inside the object itself, in an annotation named kubectl.kubernetes.io/last-applied-configuration. The client compares the old file with the new file to detect what you deleted, compares the new file with the live object to detect what to add or change, and then sends a patch.

flowchart TD
  NEW["New file from the user"] --> M["Client computes three-way merge"]
  LIVE["Live object read from the cluster"] --> M
  LAST["Last-applied copy stored in an annotation"] --> M
  M --> P["Client builds a patch"]
  P --> S["API server stores the result"]
  S --> A["Annotation updated with the new file"]
    
Figure 2: How client-side apply decides what to change, entirely inside the client

The limits of the client-side design

What it did well

  • Made declarative files the normal way to work.
  • Detected removed fields using the saved copy.
  • Worked on any cluster without server support.

Where it struggled

  • The saved annotation duplicated the file and had a size limit.
  • Only one tool, kubectl, understood the annotation; other tools ignored it.
  • Merge logic lived in the client, so different versions behaved differently.
  • Poor support for custom resources, which lack the special merge hints.
  • No notion of ownership, so overlapping edits silently overwrote each other.

The annotation size problem

Annotations on an object have a combined size limit of 256 kilobytes. Because the saved copy of the whole file lives in one annotation, large objects, such as big ConfigMaps or long custom resources, could hit this limit and fail to apply. Teams worked around it with awkward tricks. This was one of the first practical pains that motivated change.

Custom resources and lists

The client-side method relied on the strategic merge patch for built-in types, which knows that a list of containers should be merged by name. For custom resources, no such hints existed, so it fell back to a simpler merge that treats lists as replace-everything. Two teams editing different items in the same list of a custom resource could erase each other’s entries. As custom resources and Operators became central to Kubernetes, this was a growing problem.

Everyday analogy

Client-side apply is like each cook carrying a private notebook describing what they did last time. It only helps that one cook. A different cook, with a different notebook or none at all, has no idea what the first cook put on the counter.

Production example: pipelines clashing with autoscalers

A very common real-world story involves a deployment pipeline that applies a complete file containing a fixed replica count, while an autoscaler adjusts the same field. Teams discovered, usually during a traffic spike, that every release reset the replicas to a small number and caused brief overload. This is exactly the sort of problem that field ownership is designed to prevent.

4Meet Server-Side Apply

Server-Side Apply keeps the declarative spirit of the old apply but moves the intelligence to the one place every client shares: the API server.

The main idea

With Server-Side Apply, a client sends its desired configuration to the API server as a special kind of patch called an apply request, along with a name for itself, called the field manager. The API server then does the merging. It knows which fields each manager has said it cares about, records that information on the object, and refuses or reports changes that would trample someone else’s fields.

A short history

1

Alpha

Server-Side Apply first appeared as an experimental feature around Kubernetes 1.14.

2

Beta

It became enabled by default around 1.16, as the field-tracking design matured.

3

General availability

It reached GA in Kubernetes 1.22, meaning it is stable and supported for production use.

4

Ecosystem adoption

Tools such as GitOps controllers, Operator libraries, and cloud add-on managers have since adopted it for their own writes.

The three promises

Promise 1

Ownership is tracked

The cluster records which manager set each field, so there is a shared answer to “who owns this?”.

Promise 2

Conflicts are surfaced

If you try to change a field someone else owns to a different value, you are told, instead of overwriting silently.

Promise 3

Removal is clean

If you stop mentioning a field you used to own, the cluster removes it for you, as long as nobody else also owns it.

flowchart LR
  C["Client with a field manager name"] -->|"apply request with desired fields"| API["API server"]
  API --> MERGE["Server merges using the schema and ownership records"]
  MERGE --> OWN["Ownership data stored with the object"]
  MERGE --> DB["Updated object stored in the database"]
  MERGE -. "if clash" .-> ERR["Conflict reported to the client"]
    
Figure 3: In Server-Side Apply the API server performs the merge and remembers who owns what

Declare intent, not the whole object

One of the most important mindset shifts is this: with Server-Side Apply, you should send only the fields you care about, not a complete copy of everything. You are saying, “these are the fields I own, and these are the values I want”. Fields you do not mention are left alone, unless you used to own them and have now dropped them. Other parties remain free to own the remaining fields.

Everyday analogy

Imagine the shared kitchen now has labels on every station: soup, bread, salad. Each cook tells the manager, “I am responsible for the soup, and here is what it should look like”. The manager makes sure no other cook rewrites the soup station without a conversation. And if the soup cook leaves the soup out of his next plan, the manager clears that station because it was his.

What stays the same

Server-Side Apply still uses the same objects, the same permissions system, and the same validation as every other write. You are still sending a request to the same API server. What changes is the rule for combining your request with what is already there, and the extra bookkeeping the server keeps.

i
Terminology to remember

Apply is a kind of write that declares full intent for the fields you own. Update is the name for every other kind of write, such as replace and ordinary patch. Both kinds are tracked, but they behave differently, as the next chapters explain.

5Field Ownership and managedFields

The heart of Server-Side Apply is a record of ownership. Once you understand this record, everything else follows naturally.

Where the record lives

Every Kubernetes object has a metadata section, and inside it there is a list called managedFields. Each entry in the list describes one field manager and the exact set of fields that manager owns. By default, command-line tools hide this list to keep the output readable, but you can ask to see it.

What each entry contains

PartMeaningExample idea
ManagerThe name of the party that made the changeA pipeline name, a controller name, or a tool name
OperationWhether the change was an Apply or an UpdateApply for declared intent; Update for ordinary writes
API versionWhich version of the object type the manager usedFor example the stable version of Deployment
TimeWhen this manager last changed any of its fieldsA timestamp
Fields setA tree listing every field this manager ownsImage of a container, replica count, a label
SubresourceWhether the change was made through a special part of the object, such as status or scaleStatus subresource for controllers

What “owning a field” really means

When a manager owns a field, three things are true. First, the manager has told the cluster that it cares about the value of that field. Second, if the manager later stops mentioning the field in an apply request, the field is removed, provided no other manager also owns it. Third, if some other manager tries to set the field to a different value, the cluster treats it as a clash instead of a silent overwrite. Ownership is a mix of responsibility and protection.

flowchart TD
  OBJ["Deployment object"] --> SPEC["spec"]
  SPEC --> REP["replicas: owned by the autoscaler"]
  SPEC --> TPL["template"]
  TPL --> CON["containers (merged by name)"]
  CON --> IMG["image: owned by the pipeline"]
  CON --> LIM["resources limits: owned by the platform team"]
  OBJ --> META["metadata labels"]
  META --> LAB["team label: owned by the pipeline"]
  META --> MON["monitoring label: owned by the monitoring tool"]
    
Figure 4: One object, many owners: each branch of the field tree can belong to a different manager

Shared ownership

Ownership is not always exclusive. If two managers apply the same value to the same field, both become co-owners. There is no conflict, because they agree. If one of them later changes the value, the other is told about the disagreement the next time it applies. If one stops mentioning the field, the other still owns it and the field stays. This shared ownership is what lets teams safely express “I need this to be true” without fighting over who set it first.

Naming your manager

The field manager name is chosen by the client. Command-line tools use a default name based on the program. When you use kubectl with server-side apply, the default manager name is kubectl. You can pick a more meaningful name, such as the name of your pipeline or your controller. Good names make ownership readable. Poor names, such as the same generic name for every pipeline, make it impossible to tell parties apart and can cause teams to accidentally share ownership.

!
Not a security boundary

Because the client chooses the manager name, a name is a label for coordination, not proof of identity. Any caller with permission to write the object could claim any manager name. Never use field ownership as an access control. Use the normal permission system for that.

Everyday analogy

The ownership record is like the sign-off sheet on a shared checklist. Next to each line is the name of the person who filled it in. You can see at a glance who is responsible for what, and if you try to change someone else’s line, the sheet tells you whose it is before you do.

6Conflicts: When Two Owners Disagree

Conflicts are a feature, not a failure. They are the cluster’s way of telling you, politely, that you are about to change something another party cares about.

When a conflict happens

A conflict arises when an apply request sets a field to a value different from the current value, and that field is owned by a different manager. The API server refuses the request with a conflict error and lists the fields in dispute, along with the names of the managers that own them. The client can then decide what to do.

flowchart TD
  A["Apply request arrives"] --> B{"For each field: is it owned by another manager?"}
  B -- "No" --> OK["Accept the field and record ownership"]
  B -- "Yes" --> C{"Is the requested value the same as the current value?"}
  C -- "Yes" --> SH["Accept and share ownership"]
  C -- "No" --> F{"Is force enabled?"}
  F -- "No" --> CON["Reject with a conflict listing owners and fields"]
  F -- "Yes" --> TAKE["Take ownership and overwrite the value"]
    
Figure 5: The decision the API server makes for every field in an apply request

Three ways to resolve a conflict

1

Overwrite with force

You tell the server to take ownership anyway. The previous owner loses that field. Use this only when you are sure your claim is the correct one.

2

Do not apply those fields

You remove the disputed fields from your configuration and let the other owner keep control. This is the right answer when the field was never really yours.

3

Agree on the value

You change your value to match the current one. Both of you then co-own the field, and the conflict disappears.

Who should force, and who should not

The right choice depends on who you are. A controller that fully manages an object, such as an Operator that owns the Deployments it creates, should normally force ownership, because those fields are its responsibility by design. If someone else edits them, the controller is meant to correct that. A human or a pipeline sharing an object with others should generally not force. A conflict is valuable information: it tells you that someone else has a stake in that field, and you should talk to them or reconsider.

Force is appropriate when

  • You are a controller that owns the whole object.
  • The object is created and managed only by your tool.
  • You deliberately want to reclaim fields after drift.

Force is risky when

  • Other tools or teams legitimately edit the same object.
  • You are using it just to make an error go away.
  • You do not know who owns the disputed fields.
!
The lazy fix

The most common mistake with Server-Side Apply is adding the force option to every command the first time a conflict appears. This silently recreates the old problem of tools overwriting each other, and merely hides the warnings. Treat a conflict as a question to answer, not an obstacle to bypass.

Conflicts only happen for apply

An important detail: conflicts are reported only for apply requests. An ordinary update, such as a replace or a normal patch, never fails because of ownership. Instead, it simply takes ownership of the fields it changed. This is by design, for backward compatibility with all the tools that existed before Server-Side Apply. It also means that a stray update can quietly steal a field from an applier, and the applier will see a conflict the next time it runs.

Everyday analogy

Think of the two kinds of writers as polite and impatient. The apply writer is polite: it checks the sign-off sheet and asks before changing someone else’s line. The update writer is impatient: it simply writes what it wants and puts its own name on the line. The sheet remains accurate, but the polite writer may find, next time, that the line is no longer where it left it.

7Merge Rules: How Maps and Lists Combine

Ownership is tracked per field, but objects contain nested maps and lists. The way the server merges these shapes decides whether two teams can safely edit different parts of the same list.

The shapes inside an object

An object is built from a few basic shapes. A scalar is a single value such as a number or a text string. A map (also called an object) is a set of named fields. A list is an ordered collection of items. Merging scalars and maps is simple: each named field is handled on its own. Lists are the hard part, because items have positions that can shift when something is added or removed.

Three kinds of lists

List typeHow items are identifiedMerge behaviorTypical use
AtomicThe list is treated as one single valueWhoever applies last replaces the entire list; no partial ownershipShort lists where the order or complete set matters
SetEach item is its own identity, like a unique tagDifferent managers can own different items; duplicates are not allowedLists of simple unique values
Map (keyed list)Each item is identified by one or more key fields, such as a nameItems are merged by key; different managers can own different items or different fields inside an itemContainers, environment variables, volumes, ports

An example in words

A Deployment holds a list of containers. In the built-in type, that list is merged by the container’s name. Suppose the pipeline owns the “web” container’s image, and a platform tool owns a second container named “metrics-agent”. When the pipeline applies its configuration, which mentions only the “web” container, the “metrics-agent” container is left alone, because the pipeline never owned it. That is exactly what the platform tool needs. Likewise, the platform tool can add or change its container without touching the pipeline’s.

flowchart LR
  subgraph L["Containers list merged by name"]
    W["web: image and port owned by the pipeline"]
    M["metrics-agent: owned by the platform tool"]
  end
  P["Pipeline apply mentions only web"] --> W
  T["Platform apply mentions only metrics-agent"] --> M
  P -. "does not touch" .-> M
    
Figure 6: A keyed list lets two managers share one list without stepping on each other

Maps with atomic behavior

Maps can also be marked as atomic, which means they are treated as a single value, similar to an atomic list. This is useful for small structured values that only make sense together, such as a pair of values that must change as a unit. A granular map, the default, treats each key independently.

How the server knows the rules

For built-in types like Deployment and Service, the rules are already part of Kubernetes. For custom resources, the rules come from the schema in the custom resource definition. Authors of the definition mark each list with its type, using special annotations in the schema, such as x-kubernetes-list-type and x-kubernetes-list-map-keys, and mark maps with x-kubernetes-map-type. If the author says nothing, lists default to atomic, which is the safe but coarse choice.

i
For custom resource authors

If you build a custom resource where several teams or controllers should edit different items of a list, mark that list as a map with clear keys, or as a set. Leaving it atomic means any apply to the list replaces it entirely and ownership of individual items cannot be tracked.

Benefits of precise list types

  • Several managers can safely share one list.
  • Removing your item does not remove others.
  • Ownership shows exactly who added each entry.

Costs of getting them wrong

  • Atomic lists cause whole-list overwrites.
  • A wrong key choice can merge unrelated items.
  • Changing list types later is a compatibility risk.

8Declarative Deletion: Removing by Leaving Out

One of the most useful and most surprising behaviors of Server-Side Apply is how removal works. You do not say “delete this field”. You simply stop mentioning it.

The rule

When you apply a configuration, the server compares the fields in your request with the fields you owned after your previous apply. Any field you used to own but no longer mention is removed from the object, unless another manager also owns it. If another manager owns the same field, the field stays and you simply give up your share of ownership.

sequenceDiagram
  participant Y as You (manager A)
  participant S as API server
  Y->>S: Apply: label env=prod and label tier=web
  S-->>Y: Stored. You own both labels
  Y->>S: Apply: label env=prod only
  S->>S: You owned tier but no longer mention it
  S->>S: No other owner of tier, so remove it
  S-->>Y: Stored. Only env remains
    
Figure 7: Leaving a field out of your next apply removes it, if you were its only owner

Why this is powerful

Without this rule, configuration files could only add or change things. To remove a label, you would need to run a separate delete command. With declarative deletion, your file is the complete truth about your fields. Delete a line from the file, apply again, and the cluster follows. This makes configuration review simple: the file in version control always matches what you own.

Why this can be dangerous

The same rule means that careless edits can delete real settings. If a team member trims a file and removes a field that the cluster actually needs, the next apply removes it. This is also why switching a field from “owned by me” to “owned by another tool” needs a deliberate hand-over, as we will see with autoscalers later.

Safe case

Removing a label you added

You were the only owner, so it disappears exactly as intended.

Shared case

Removing a value two managers set

The field remains because another manager still owns it. You merely give up your claim.

Risky case

Removing a field with a default

If nothing else owns it, the server clears it and the default value returns, which may not be what the cluster was running.

Careful case

Handing a field to another tool

You must make sure the other tool owns it first, or the value can briefly reset.

!
Whole objects are not deleted

Declarative deletion removes fields, not whole objects. If you stop applying a Deployment file, the Deployment stays in the cluster. Removing complete objects needs a separate action, such as a delete command or a tool that tracks object inventories and prunes them.

Everyday analogy

Imagine you hang several photos on a shared wall and keep a list of which photos are yours. If you take a photo off your list, the helper takes it down for you. But if another person has also pinned the same photo, the helper leaves it up, because that person still wants it there.

9Apply Versus Update: Living Together

Real clusters mix both styles of writing. Understanding how apply and update interact helps you predict surprises before they happen.

Two operations, two behaviors

QuestionApplyUpdate
What does the writer send?Only the fields it wants to ownA full object or a patch of changes
Are conflicts reported?Yes, unless forcedNo, it silently takes ownership of changed fields
What does ownership include?Everything in the applied configurationOnly the fields that actually changed
Are omitted fields removed?Yes, if previously owned and not sharedNo, unless the update itself removes them
Typical usersPipelines, GitOps tools, controllers that declare stateManual edits, scripts, quick patches, older tools

How common commands appear

Many familiar commands use update-style writes and therefore show up with their own manager names in the ownership record. An interactive edit is recorded under an edit-related manager name. A scale command, a label command, or a set-image command each leave a trace. The old client-side apply appears under a client-side-apply manager name. This record is extremely handy for answering the question “who changed this field and how?” long after the event.

A realistic sequence

1

Pipeline applies

The pipeline declares the image and labels. It now owns those fields as an Apply manager.

2

Autoscaler updates

The autoscaler changes the replica count through the scale endpoint. It owns replicas as an Update manager.

3

Engineer edits by hand

During an incident, someone changes the image. The edit takes ownership of the image field from the pipeline.

4

Pipeline applies again

The pipeline sees a conflict on the image. It must decide: force to restore its version, or investigate the manual change.

i
A useful side effect

The conflict in step four turns an invisible override into a visible event. Instead of the pipeline silently reverting the hotfix, someone is prompted to decide whether the manual change should be kept or reverted. That is exactly the kind of coordination the old system lacked.

Mixing styles: guidelines

Healthy mixing

  • Declarative tools apply the fields they own.
  • Autoscalers and controllers update the fields meant for them.
  • Emergency manual edits are rare and followed by cleanup.

Unhealthy mixing

  • Several tools applying the same fields under the same name.
  • Manual edits that never get reconciled with the source files.
  • Scripts that replace whole objects.

10Using Server-Side Apply from the Command Line

You do not need special tools to use Server-Side Apply. The familiar kubectl command supports it through a few options. This chapter explains them in words, so you know what each one means before you type it.

The key options

OptionWhat it doesWhen to use it
--server-sideSends the file as a server-side apply request instead of computing the patch on the clientWhenever you want ownership tracking and server merging
--field-managerSets the name under which your changes are recordedAlways, in pipelines, so every pipeline has its own clear name
--force-conflictsTakes ownership of any conflicting fieldsRarely, and only when you are sure the fields should be yours
--dry-run=serverAsks the server to process the request fully but not save itTo preview what would change and whether any conflicts exist
diff with the server-side optionShows the difference the server would produceIn review steps before a real apply
--show-managed-fieldsIncludes the ownership record in the output of a get commandWhen debugging who owns what

A typical safe workflow

1

Choose a manager name

Pick a short, descriptive name for your pipeline, such as the application name plus the word “deploy”.

2

Preview with a server dry run

Run the apply in server dry-run mode. It reveals conflicts and validation errors without changing anything.

3

Review the diff

Look at exactly what would change, in the same review step where humans approve the release.

4

Apply for real

Run the actual apply with the same manager name. If a conflict appears, stop and investigate before forcing.

5

Verify ownership

Inspect the ownership record occasionally to confirm that your fields belong to your manager as expected.

The dry run is more honest than the old one

The older client-side dry run only simulated the merge in the client. The server dry run asks the real API server, including its validation rules, defaulting, and admission checks, to do all the work without saving the result. It is therefore much closer to the truth of what will happen. Many teams make it a mandatory step in their pipelines.

i
Always set a manager name

If you leave the manager name at its default, every user and every pipeline that runs the same tool shares one identity. The ownership record then cannot tell them apart. A unique name per tool or pipeline is a tiny habit that makes the whole system explainable.

Permissions needed

An apply request is a kind of patch, so the caller needs permission to patch the object. If the object may not exist yet, apply can create it, which also requires permission to create. Server-Side Apply does not require any exotic permission, which makes it easy to adopt in clusters that already use the normal permission system.

11Migrating from Client-Side Apply

Most real clusters already contain thousands of objects created with the old method. Fortunately, Kubernetes provides a smooth path across.

What happens to existing objects

Objects created or last applied with client-side apply carry the saved annotation and a record from the client-side manager. When you apply such an object for the first time using server-side apply, the system recognizes the history. It converts the old record into proper ownership data, so fields you used to manage become fields your new manager owns. You do not need to recreate anything.

flowchart LR
  OLD["Objects managed by client-side apply: annotation plus client-side manager record"] --> FIRST["First server-side apply with a chosen manager name"]
  FIRST --> CONV["Server upgrades ownership data for the fields you declared"]
  CONV --> NEW["Objects managed by server-side apply"]
  NEW -. "optional: drop old annotation over time" .-> CLEAN["Cleaner object without the saved copy"]
    
Figure 8: The upgrade path from the old method to the new one

A careful migration plan

1

Check your cluster version

Make sure the cluster is at a version where Server-Side Apply is generally available.

2

Choose manager names

Decide on a naming scheme before you start, so that every pipeline gets a distinct and meaningful name.

3

Try on a test environment

Switch one pipeline in a non-production cluster and review the diffs and ownership records.

4

Resolve the first conflicts

Expect some conflicts with autoscalers, controllers, or manual edits. Each is a chance to clarify who should own the field.

5

Roll out gradually

Move production pipelines one at a time, never mixing old and new styles for the same object in the same period.

6

Remove old assumptions

Retire scripts that rely on the saved annotation or on client-side merge behavior.

Things that surprise teams

Surprise

Fields get removed

Because the old record is converted into ownership, a field present in the old record but missing from your new file may now be removed. Review diffs closely.

Surprise

Full objects in files

Files exported from a live cluster contain fields owned by other parties. Applying them as-is claims ownership of fields that should belong to others.

Surprise

Mixed tools

If one team uses the old method and another the new, their writes may follow different rules on the same object.

Surprise

Going back

Switching back to the old method is possible, but the ownership data and saved annotation may no longer be in step. Plan carefully before reversing.

!
The exported-file trap

A very common source of trouble is saving a live object to a file, editing it, and applying it. That file includes fields set by controllers, defaults, and status. Applying it declares ownership of all of them. Write clean files that contain only the fields you intend to own.

Production example: staged migration at a large platform

Teams running many clusters commonly migrate in waves: first internal tooling namespaces, then non-critical applications, then production. They monitor conflict counts during each wave and treat a surge of conflicts as a sign that ownership boundaries were unclear, which is valuable information they then fix in documentation and files before proceeding.

12Server-Side Apply in Controllers and Operators

Controllers write to the cluster all day long. Server-Side Apply gives them a safer and simpler way to say what they want.

The old way for controllers

Traditionally, a controller that managed a child object would read the object, change a few fields in memory, and send the entire object back with an update, or compute a patch by hand. This required careful code to avoid overwriting fields set by others, and it required retries when the version stamp changed. Many subtle bugs came from this read-modify-write pattern.

The new way: declare and apply

With Server-Side Apply, a controller builds a small description of just the fields it cares about, called an apply configuration, and sends it with its field manager name. The server merges it. There is no need to read first, no need to retry because of version stamps, and no risk of accidentally resetting fields owned by someone else. The client libraries for the main programming languages provide typed helpers for building such configurations, and popular controller frameworks support apply as a first-class operation.

Benefits for controllers

  • Simpler code with no read-modify-write loop.
  • Fewer conflicts caused by stale reads.
  • Automatic cleanup of fields the controller no longer sets.
  • Clear, inspectable ownership of the controller’s fields.

Things to watch

  • The controller must send a complete intent each time, not a partial delta.
  • It should force ownership only for fields it truly manages.
  • Large objects mean larger requests.

Good manager names for controllers

A controller should use one stable field manager name that identifies it, and use the same name on every run and every replica. If the name changes between versions, the new name starts as a separate owner, and the old name’s fields are left behind, which can cause confusing ghost fields. Treat the manager name as a permanent part of the controller’s identity.

Spec and status are separate

Controllers often write two different parts of an object: the spec of the child objects they create, and the status of their own custom resource. The status is usually written through a separate subresource, and the ownership record keeps track of the subresource in each entry. A good pattern is to apply status with its own clear manager name, so that status and spec ownership never get tangled together.

flowchart TD
  CR["Custom resource written by a user"] --> CTRL["Controller reconcile loop"]
  CTRL -->|"apply child objects with the controller's manager name"| CHILD["Deployments, Services, ConfigMaps"]
  CTRL -->|"apply status through the status subresource"| ST["Status of the custom resource"]
  OTHER["Other tools, autoscalers, humans"] -->|"update their own fields"| CHILD
    
Figure 9: A controller applies only its own fields to child objects, leaving room for others
i
Design with sharing in mind

If you write an Operator, think about which fields on the child objects you want to own, and which you want to leave free for users or other tools, such as replica counts managed by an autoscaler or labels added by a platform. Apply only the first group. This single design choice is what makes your Operator a good citizen.

Production example: Operator frameworks adopt apply

Popular Operator tooling now offers apply as a normal way to write objects, with options for naming the manager and forcing ownership. Authors who switch from read-modify-write to apply often delete large chunks of defensive code, and report fewer bugs caused by two controllers fighting over the same field.

13Sharing Fields Safely: Autoscalers and Hand-Overs

The replica count is the most famous shared field in Kubernetes. It is the perfect lesson in how to hand a field from one owner to another without breaking anything.

The classic conflict

A Deployment has a replica count in its spec. A horizontal pod autoscaler is designed to change that same number as load rises and falls. Meanwhile, your deployment file might also state a replica count. If your pipeline applies a fixed number every release, it fights the autoscaler. The autoscaler raises replicas to twenty, the next release sets them back to three, and the service struggles.

The simple solution

With Server-Side Apply, the clean answer is for the deployment file to not mention the replica count at all, once an autoscaler manages it. The pipeline then does not own the field, so it never overwrites the autoscaler’s choice. The autoscaler, which writes through the scale part of the object, owns the field. The two live together without conflict.

sequenceDiagram
  participant P as Pipeline (apply)
  participant H as Autoscaler (update)
  participant S as API server
  P->>S: Apply: image, labels (no replicas)
  H->>S: Set replicas to 12 through scale
  S-->>H: Stored, autoscaler owns replicas
  P->>S: Apply new image on next release (no replicas)
  S-->>P: Stored, replicas untouched at 12
    
Figure 10: When the pipeline never mentions replicas, the autoscaler’s value survives every release

The hand-over problem

The trap appears when you already have a file that mentions replicas, and you want to switch to autoscaling. If you simply delete the line from the file and apply, the server sees a field you used to own that you no longer mention. If nobody else owns it, the field is removed, and the default value takes over, usually one replica. Your service shrinks suddenly to a single copy before the autoscaler reacts. This is the sharp edge of declarative deletion.

A safe hand-over pattern

1

Let the autoscaler start owning the field first

Create the autoscaler and let it set the replica count, so that it becomes an owner of the field while your pipeline still shares it.

2

Confirm the ownership record

Inspect the object and check that the autoscaler appears as an owner of the replica field.

3

Remove the line from your file

Apply the updated file. Because another owner exists, the field stays, and your pipeline simply gives up its share.

4

Watch for stability

Observe the next releases. The replica count should stay under the autoscaler’s control without sudden jumps.

!
Check the official guidance for your version

The exact steps for transferring ownership of a field are described in the Kubernetes documentation and have been refined over time. Treat the pattern here as the idea, and confirm the details against the documentation for your cluster version before running it on production.

Other fields that are commonly shared

Labels

Added by several tools

Pipelines, cost tools, and security scanners may each add their own labels. Distinct keys keep them apart.

Annotations

Hints for controllers

Ingress controllers, meshes, and backup tools read annotations that different teams may set separately.

Containers

Sidecars from platforms

A platform may add helper containers to every workload. Keyed list merging lets them coexist with yours.

Resources

Vertical autoscaling

A tool that adjusts CPU and memory may own those fields. Your file should then not set them, or should coordinate with the tool.

14Server-Side Apply and GitOps Tools

GitOps tools continuously compare the cluster with a Git repository and apply the difference. They are among the heaviest users of Server-Side Apply, and they show its strengths very clearly.

What GitOps does

A GitOps tool, such as Flux or Argo CD, watches a repository where your configuration files live. When the files change, the tool applies them to the cluster. It also periodically checks whether the cluster has drifted from the repository, and can correct the drift. This is exactly the repeated, declarative apply that Server-Side Apply was built for.

Why GitOps benefits

Gains from server-side apply

  • No giant saved annotation on every object.
  • Fewer false drift reports caused by fields set by controllers.
  • Better handling of custom resources with proper list rules.
  • Clear conflict messages when someone edits by hand.

New things to manage

  • Ownership conflicts with Operators, autoscalers, and meshes.
  • Need to decide which fields Git should own.
  • Larger ownership records on objects.

The ownership split that works

The healthiest GitOps setups follow a simple rule: Git declares only the fields humans decide, and everything else is left to the cluster’s other actors. That means files should contain the intended spec, and not fields generated by controllers, not status, and not values owned by autoscalers. When a conflict appears, the right fix is usually to remove the disputed field from Git rather than forcing it in.

flowchart TD
  GIT["Git repository: human-decided fields only"] --> GO["GitOps tool applies with its own manager name"]
  GO --> OBJ["Cluster object"]
  AS["Autoscaler updates replicas"] --> OBJ
  OP["Operator applies generated fields and status"] --> OBJ
  MESH["Platform injects sidecar fields"] --> OBJ
  OBJ --> DRIFT["Drift check compares only fields Git owns"]
    
Figure 11: Each actor owns its own fields, and drift detection looks only at the fields Git owns

Handling drift without fighting

Drift means the cluster differs from Git. With ownership tracking, a GitOps tool can limit its drift checks to the fields it owns. A change to a field owned by someone else is not drift from Git’s point of view, because Git never claimed it. This greatly reduces noisy alerts and pointless corrections. Where a tool does still report differences in fields it should not manage, you can configure ignore rules for those specific paths.

i
A good operating rule

When your GitOps tool reports a conflict, ask: “Whose field is this really?” If it is the Operator’s or autoscaler’s, remove it from Git. If it is Git’s, find and stop the other writer. Avoid the shortcut of forcing ownership everywhere.

Production example: Flux and Argo CD options

Flux uses server-side apply for its reconciliations, and Argo CD offers a server-side apply option that teams enable per application, especially for very large objects or custom resources. Both tools document how to ignore specific fields and how to handle conflicts, which makes them practical places to learn the ownership model on real workloads.

15Helm, Kustomize, and CI Pipelines

Most teams do not write raw files by hand. They generate them with templating and overlay tools, then push them through pipelines. Server-Side Apply fits into each of these, but each has its own habits worth understanding.

Where the tools fit

ToolHow it normally applies changesRelationship to server-side apply
HelmComputes its own three-way comparison between the previous release, the new release, and the live object, then sends patchesHelm keeps its own release history and does not use field ownership by default. Newer versions of Helm have added options to use server-side apply
KustomizeBuilds final files from a base and overlays, then hands them to kubectlWorks well, because the output is simply a set of files you can apply server-side
Plain kubectl in a pipelineApplies files directlyFully supports server-side apply with a unique manager name
GitOps controllersApply from Git on a loopOften use server-side apply, as the last chapter explained
Everyday analogy

Imagine a print shop. Kustomize and templating tools are the designers who prepare the final artwork. The apply step is the printing press. Server-Side Apply is a press that stamps a small signature in the corner of every section it prints, so anyone later can see which designer’s work each section came from.

Helm and ownership

Helm tracks what it installed through release records. When Helm updates a release, it compares the old and new versions of the chart output with the live object, and sends changes. This works well for many cases, but Helm does not give each field an owner in the way Server-Side Apply does. If another controller changes a field that Helm also manages, Helm may overwrite it during the next upgrade, or leave it, depending on the details. Because Helm’s behavior around server-side apply has been evolving, check the documentation for your version of Helm before relying on any specific option.

!
Mixing Helm and apply on one object

If one pipeline installs an object with Helm, and another applies changes to the same object with Server-Side Apply, the two do not share a common idea of who owns what. Choose one tool per object, or define very clearly which fields each tool may touch.

Kustomize and overlays

Kustomize creates the final files by layering small changes on top of a shared base. Since it only produces files, it does not conflict with Server-Side Apply. In fact, the combination is pleasant: each environment’s overlay declares only the fields that differ, and the apply step records ownership of exactly those fields. Be careful, though, that overlays do not accidentally include fields that should belong to another system, such as the replica count when an autoscaler is in use.

Designing a pipeline that behaves well

1

Generate clean files

Render the final files from templates or overlays, and keep only the fields your pipeline intends to own.

2

Validate

Run schema validation and a server dry run, which also reports ownership conflicts early.

3

Show the diff for approval

Present the exact difference to a human reviewer for sensitive environments.

4

Apply with a stable manager name

Use the same name for the same pipeline every time, and a different name for each separate pipeline.

5

Fail loudly on conflicts

Do not force by default. A conflict is a signal that two parties disagree, and a human should look at it.

Production example: separate pipelines for separate concerns

Larger organizations often split their delivery into several pipelines: one for application workloads, one for platform add-ons, and one for security policy. Giving each its own manager name makes the ownership record read like an organization chart. When something unexpected changes in production, an engineer can read the record and immediately know which team’s pipeline to ask.

16Custom Resources and Merge Hints

Built-in objects come with merge rules already attached. Custom resources, the objects defined by Operators and extensions, need those rules to be declared by their authors. This is where Server-Side Apply shines most.

Why custom resources need hints

The server understands the shape of built-in objects and knows, for instance, that the container list in a pod is keyed by container name. For a custom resource, the server only knows what the author wrote in the schema. If the author says nothing about a list, the server treats the whole list as one unit, which means that two owners cannot manage different items. Authors can add small hints to their schema that describe how lists and maps should be merged.

The three hints in plain words

List

Atomic list

The whole list is one value. Whoever applies last replaces the entire list. Safe, simple, and the default.

List

Set list

A list of simple values, like names, where each value is unique. Different owners can add different values.

List

Map list

A list of objects identified by one or more key fields. Each item is merged by its key, so owners can manage different items and different fields in the same item.

Object

Granular or atomic map

An object whose fields can be owned separately, or one that is treated as a single value. Authors choose depending on whether the fields make sense on their own.

Everyday analogy

Think of a shared recipe card. An atomic list is like a card where the ingredients are written on a single sticker. Anyone who changes it replaces the whole sticker. A map list is like a card where each ingredient has its own labeled line, so one cook can change the salt line while another changes the butter line, with no clash.

How authors choose

Choose a keyed list when

  • Items have a natural unique name, such as a port name or a rule name.
  • Different tools may add or edit different items.
  • Items are independent of each other.

Choose an atomic list when

  • The order matters, or the items only make sense together.
  • There is no stable key.
  • Partial edits would produce invalid combinations.
flowchart TD
  Q1{"Do items have a stable unique key?"} -->|"yes"| MAP["Use a keyed map list"]
  Q1 -->|"no"| Q2{"Are items simple unique values?"}
  Q2 -->|"yes"| SET["Use a set list"]
  Q2 -->|"no"| ATOM["Use an atomic list"]
  MAP --> NOTE["Owners can edit different items and fields"]
  SET --> NOTE2["Owners can add different values"]
  ATOM --> NOTE3["Last applier replaces the whole list"]
    
Figure 12: Choosing a merge strategy for a list in a custom resource schema
i
Changing the strategy later is delicate

Switching a list from atomic to keyed after users already depend on it can change how existing ownership records are interpreted. Decide early, test with two writers, and read the documentation for CRD schema changes before altering a published API.

Production example: shared lists in platform APIs

Gateway and networking APIs in the Kubernetes ecosystem have lists of listeners and routes, where different teams need to add their own entries to one shared object. Keyed lists make this possible, so an application team can add a route entry without touching entries owned by the platform team.

17Admission Webhooks, Defaults, and Mutation

Not every change to an object comes from a user or a controller. Some come from the API server itself, as it fills in defaults, and from webhooks that modify objects on the way in. Understanding these helps explain surprising ownership entries.

The journey of a request

When an apply request arrives, the server authenticates and authorizes it, merges it with the stored object, applies defaults for unset fields, runs any mutating webhooks, validates the result, runs validating webhooks, and finally stores it. The ownership record is updated through this process, so it is useful to know where each field’s value came from.

sequenceDiagram
  participant C as Client
  participant A as API server
  participant M as Mutating webhook
  participant V as Validating webhook
  participant E as Storage
  C->>A: Apply request with manager name
  A->>A: Authenticate and authorize
  A->>A: Merge with stored object, update ownership
  A->>A: Apply defaults
  A->>M: Send object for mutation
  M-->>A: Return changes
  A->>A: Validate schema
  A->>V: Send object for approval
  V-->>A: Allow or deny
  A->>E: Store object
  A-->>C: Return result
    
Figure 13: Where defaults and webhooks sit in the life of an apply request

Defaults and ownership

Default values are not normally owned by the person who sent the request. A defaulted field simply appears in the stored object without a user claiming it. If you later write a value for that field, you become its owner. This is why a freshly applied object can contain many fields that your file never mentioned, and why those fields do not appear under your manager’s name.

Mutating webhooks and the sidecar story

A mutating webhook can change an object as it passes through. A well-known example is a service mesh adding a helper container to every pod. These changes happen inside the server’s own flow. When a webhook changes a field, the ownership record is typically updated so that the change is attributed to the request’s manager, unless the webhook operates in a way that avoids it. Since behavior can depend on version and webhook design, it is wise to test with a small object and read the ownership record before depending on any particular outcome.

!
Webhooks that fight your file

A webhook that rewrites a field on every request can cause your pipeline to see a difference on every run. If your file sets a value and the webhook changes it, each apply will appear to change the object again. The fix is to stop declaring that field, or to adjust the webhook so it respects values that are already set.

Idempotent mutation

A good mutating webhook is idempotent, meaning that running it twice gives the same result as running it once. Idempotent webhooks keep apply requests stable, because repeated applies do not keep adding new items or flipping values. This is a quality bar worth checking in any webhook your platform installs.

Production example: sidecar injection

Service meshes such as Istio and Linkerd add proxy containers to workloads through webhooks. Teams that move their pipelines to Server-Side Apply commonly find that their files should never list the injected container, because the keyed container list lets the mesh’s entry sit next to the application’s own entry without conflict.

18Security, Permissions, and Auditing

Because Server-Side Apply records who changed what, it is tempting to treat the record as a security feature. It is helpful, but it is not a security boundary. This chapter draws that line carefully.

The manager name is a label, not a lock

The manager name is chosen by the caller. Anyone who can send an apply request can use any name they like. The server does not check that the name matches the real identity of the caller. Therefore, ownership records are useful for coordination and debugging, but they cannot be trusted to prove who really made a change. For proof, use the audit log, which records the authenticated identity of each request.

QuestionBest sourceWhy
Which tool probably owns this field?Ownership recordFast, visible on the object, designed for coordination
Who really sent this request?Audit logRecords the authenticated user or service account
Who is allowed to change this object?Role-based access controlEnforced by the server before any merge happens
Which values are allowed?Admission policyEnforced on the final merged object
Everyday analogy

The manager name is like the name someone writes on a sticky note they leave on a shared fridge. It is handy for knowing who to ask about the leftover soup, but anyone can write any name. The building’s security camera, which is the audit log, shows who actually opened the door.

Permissions

Apply is sent as a patch, so permission to patch an object is what authorizes it. Roles that grant patch on an object type therefore also permit apply. Some clusters use fine-grained rules that limit patching to certain object names or namespaces. Review roles for automation accounts, since a pipeline that can patch can also change any field of the objects it can reach, even when its own file mentions only a few.

The force option as a privilege

Forcing ownership is not a separate permission. Anyone who can apply can also force. This means that a careless script could take fields from a security controller or an autoscaler. Control this through review of pipeline definitions, through admission policies that protect critical fields, and through alerts on unexpected ownership changes.

Good security habits

  • Treat the audit log as the source of truth for identity.
  • Give each automation its own service account and manager name.
  • Use admission policy to protect fields that must never change.
  • Alert when ownership of critical fields moves.

Common mistakes

  • Trusting the manager name as proof of identity.
  • Sharing one service account across many pipelines.
  • Granting broad patch rights to every automation.
  • Forcing conflicts by default in scripts.
i
Combine the two views

During an investigation, use the ownership record to find which pipeline probably wrote a field, then use the audit log to confirm the real identity and time. Together they give a fast and reliable picture.

Production example: protecting critical fields

Security-conscious teams use policy engines such as Kyverno or Open Policy Agent Gatekeeper to refuse changes to critical fields, such as privileged settings on pods, no matter which manager name or tool sends the request. The policy checks the final object, so a forced apply cannot slip past it.

19Performance, Size, and Scale

Ownership tracking is not free. Every object carries extra data, and every apply request requires extra work on the server. For most clusters the cost is small, but at large scale it deserves attention.

Where the cost comes from

1
ownership entry per manager and operation
+
extra bytes on each stored object
0
saved annotation needed any more

The ownership record lists every field a manager owns, so objects with many fields and many managers carry larger records. For a small configuration object, this is negligible. For a very large custom resource with thousands of fields, or an object touched by dozens of different managers, the record can become a noticeable share of the object’s size. The storage system behind Kubernetes has a limit on the size of any single object, so extremely large objects need care.

Many managers, many entries

Each distinct manager name creates its own entry. A problem appears when tools generate a new manager name on every run, for example by including a random suffix or timestamp. The record then fills with abandoned entries that no longer correspond to any active tool. Always use stable names, so that the same tool reuses the same entry.

!
Unstable names grow the record

A pipeline that includes a build number in its manager name creates a new owner on every run. Fields stay owned by the old names, and new runs cannot remove them. Over time the object grows and conflicts become confusing. Keep names stable and meaningful.

Request cost

Because the server does the merging, large apply requests cost server work. A GitOps tool that applies thousands of objects on a short loop can add noticeable load. Practical remedies include longer reconcile intervals, skipping objects that have not changed, and splitting very large objects. These are tuning choices, and the right values depend on your cluster size and tools.

A short checklist for scale

1

Use stable manager names

Prevent endless growth of ownership entries.

2

Keep files minimal

Declare only the fields you own, which keeps both requests and records small.

3

Watch object sizes

Track the size of your largest custom resources and alert as they approach storage limits.

4

Tune reconcile loops

Avoid applying unchanged objects at high frequency.

5

Measure before optimizing

Use the API server’s own metrics to see whether apply requests are actually a bottleneck.

Production example: avoiding the annotation limit

One of the original motivations for Server-Side Apply was that very large custom resources could not be applied with the old method, because the saved copy in the annotation exceeded its size limit. Teams that manage large resources, such as big monitoring or policy definitions, often adopt server-side apply precisely to escape that limit.

20Debugging and Observability

When something unexpected happens to an object, you want answers quickly: who changed it, when, and why did my change not stick? The ownership record is the best place to begin.

Reading the ownership record

By default, most commands hide the ownership record to keep the output readable. You can ask to see it. For each entry you will find the manager name, whether the operation was an apply or an update, the time of the last change, the version of the API used, the part of the object it covers if it is a subresource, and the list of fields owned. Read it from top to bottom like a ledger.

SymptomLikely causeFirst thing to check
Apply returns a conflictAnother manager owns a field with a different valueThe conflict message names the manager and the field
My change disappears after a whileAnother controller or pipeline overwrites itThe ownership entries and their timestamps
A field I removed from my file is still thereAnother manager also owns itWhether a second owner is listed for that field
A field vanished after I removed itI was the only ownerApply history, and whether a hand-over was needed
Object keeps changing with no human actionTwo automations fight in a loopTimestamps and manager names that alternate
Object seems very largeMany stale manager entriesCount of entries and whether names are stable

A debugging method

1

Reproduce with a server dry run

Run the apply in server dry-run mode to see the conflict or validation message without changing anything.

2

Read the ledger

Display the ownership record and find the entry that owns the field in question.

3

Match the name to a tool

Use your naming scheme to identify which pipeline or controller the manager name belongs to.

4

Confirm with the audit log

If the identity matters, check the audit log for the real caller and time of the last change.

5

Decide the true owner

Choose which party should own the field, then change the other party’s file or code to stop declaring it.

Everyday analogy

Debugging with the ownership record is like checking the edit history of a shared spreadsheet. You do not argue about who is right. You open the history, see which colleague last changed the cell and when, and then talk to that person.

Spotting a tug of war

A telltale sign of two systems fighting is an object whose modification time keeps changing, while the same fields flip between values. If the ownership record shows two managers alternating, you have found the pair. Fix the root cause by assigning each field to exactly one of them.

What to monitor

Metric

Conflict rate

Count of apply requests that were rejected for conflicts. A sudden rise suggests a new overlap.

Metric

Request latency

Time taken by apply requests, especially for large objects.

Signal

Object size

Size of the biggest custom resources, as a share of the storage limit.

Signal

Forced applies

Pipelines using the force option, since each is a possible ownership takeover.

i
Write a naming convention down

Debugging is easy when manager names follow a rule such as team, tool, and purpose. It is painful when names are random. Document the rule in your platform handbook and enforce it in pipeline templates.

21A Worked Example: Team, Autoscaler, and Mesh

Let us walk through a realistic story from beginning to end, using no code, only words. It brings together ownership, sharing, and conflicts.

The cast

  • The application team deploys a web service named “shop” through a pipeline with the manager name “shop-deploy”.
  • The autoscaler adjusts the replica count according to traffic.
  • The service mesh adds a proxy container through a webhook.
  • The platform team adds a cost-center label through a separate pipeline named “platform-labels”.

The story in five scenes

1

First release

The team applies a file with the application container, its image, its port, and a replica count of three. The pipeline owns all those fields. The server also fills in many defaults, which no one owns.

2

The mesh joins in

The mesh webhook adds a proxy container to the pod template. Because containers are a keyed list, the proxy entry sits beside the application entry, and each has its own owner.

3

The platform adds a label

The platform pipeline applies a small file with only one label key. It now owns that single label, and no one else’s labels are touched.

4

Autoscaling arrives

The team adds an autoscaler. They first let it take part in ownership of the replica field, then remove the replica line from their file. The service continues running at the autoscaler’s chosen size, with no sudden drop.

5

A new image

A later release changes only the image. The pipeline’s file mentions the image and port, the server merges, and every other owner’s fields remain untouched.

flowchart LR
  subgraph OBJ["Deployment: shop"]
    F1["image, port: owned by shop-deploy"]
    F2["replicas: owned by autoscaler"]
    F3["proxy container: owned by mesh webhook"]
    F4["cost-center label: owned by platform-labels"]
    F5["defaults: no owner until someone sets them"]
  end
  P1["Application pipeline"] --> F1
  P2["Autoscaler"] --> F2
  P3["Mesh webhook"] --> F3
  P4["Platform pipeline"] --> F4
    
Figure 14: Four parties share one object, each owning a distinct set of fields

What would have gone wrong before

With the old method, the team’s release would have overwritten the replica count with three on every deployment, undoing the autoscaler’s work. The mesh’s proxy container might have been removed on the next apply, because the saved copy did not mention it. And the platform label would have required a risky manual edit. Server-Side Apply turns each of these hazards into a calm, explainable situation.

The one conflict that could still happen

Suppose a developer, in a hurry, edits the cost-center label by hand with an update command. Updates do not raise conflicts. They silently take ownership. The platform pipeline’s next apply would then report a conflict, because the developer and platform now disagree on the label’s value. This is the system working properly: it raises a flag instead of silently choosing a winner. The platform team can then decide whether to force the label back or discuss with the developer.

i
The lesson

Server-Side Apply does not remove disagreements. It makes them visible, specific, and local to one field, so they can be resolved calmly.

22An Incident Walkthrough: Two Pipelines at War

The following story is a composite of the kind of incident many teams experience in their first months with Server-Side Apply. It is fictional, and its purpose is to teach the pattern of diagnosis.

The symptom

A company runs a payment service. Every few minutes, the number of running copies drops from twelve to three and then climbs back. Customers see slow responses during the dips. The on-call engineer finds that nothing is crashing. The service is simply being resized over and over.

The investigation

1

Look at the object’s history

The engineer displays the ownership record and notices two managers that both list the replica field: the autoscaler and a GitOps tool.

2

Check timestamps

The two entries show alternating times, a few minutes apart. The pattern matches the dips exactly.

3

Read the GitOps configuration

The repository still contains a replica count of three, left over from before autoscaling was introduced. Someone had also enabled the force option to silence a conflict message weeks earlier.

4

Understand the loop

Every reconcile, the GitOps tool forced the value three. The autoscaler then noticed low capacity and raised the value. The cycle repeated.

sequenceDiagram
  participant G as GitOps tool (force on)
  participant S as API server
  participant H as Autoscaler
  G->>S: Apply replicas 3 (forced)
  S-->>G: Accepted, GitOps owns replicas
  H->>S: Set replicas 12 (update)
  S-->>H: Accepted, autoscaler owns replicas
  G->>S: Apply replicas 3 again (forced)
  S-->>G: Accepted, ownership flips back
  H->>S: Set replicas 12 again
    
Figure 15: A forced apply and an autoscaler flip ownership of the replica field in a loop

The fix

The team removed the replica count from the repository, following the hand-over pattern from the earlier chapter. They first confirmed that the autoscaler appeared as an owner, then removed the line. They also removed the force option from the GitOps configuration, so that any future overlap would surface as a visible conflict rather than a silent tug of war.

What the team learned

Lessons that worked

  • Read the ownership record first.
  • Alternating timestamps reveal a fight.
  • Remove the disputed field from the party that should not own it.
  • Keep the force option off by default.

Mistakes to avoid

  • Forcing to silence a conflict message without asking why it exists.
  • Leaving old values in the repository after adopting an autoscaler.
  • Assuming a stable system when no pod is crashing.
!
A conflict message is information

When a tool reports a conflict, it is telling you that two parties disagree. Silencing it with force removes the message but not the disagreement. The disagreement then shows up later as strange behavior in production.

23Best Practices

The lessons of the previous chapters can be gathered into a short list of habits. Teams that follow them report that Server-Side Apply becomes almost invisible, working quietly in the background.

Naming

One stable name per tool

Use a clear, consistent manager name for every pipeline and controller. Never include run numbers or random strings.

Files

Declare only what you own

Keep files small. Do not paste in exported objects with defaults, status, or fields set by others.

Conflicts

Do not force by default

Treat each conflict as a conversation to have. Force only when the takeover is deliberate and understood.

Preview

Always dry-run on the server

Include a server dry run and a diff in every pipeline, before the real apply.

Sharing

Plan hand-overs

When a field moves from one owner to another, make sure the new owner is in place before the old one lets go.

Schemas

Add merge hints to custom resources

If you write an API, mark lists with the right strategy and test with two writers.

Security

Use audit logs for identity

Never rely on manager names as proof of who did something.

Operations

Monitor conflicts and size

Watch conflict rates, forced applies, and object sizes, and review them regularly.

A one-page team agreement

Many teams find it useful to write a short agreement and keep it with their platform documentation. It might say: every automation has a unique manager name following a documented pattern; every file declares only owned fields; no automation forces conflicts without a review; and every shared field has a named primary owner. A one-page agreement prevents most of the disputes that would otherwise occur.

The best ownership system is one where every field has exactly one obvious owner, and everyone knows who it is.

Production example: ownership maps in platform handbooks

Platform teams at larger companies often keep an ownership map for their most important objects, such as the standard Deployment template. The map lists which fields belong to application teams, which to the platform, and which to automation. New engineers learn the map in their first week, and it matches what the ownership record shows in the live cluster.

24Anti-Patterns and Decision Records

An anti-pattern is a habit that looks sensible but leads to trouble. Here are the most common ones, written as short decision records so you can see the problem, the reason people fall into it, and the better choice.

ANTI-PATTERN 01 · FORCE EVERYTHING TO MAKE PIPELINES GREENAvoid
Why teams do it

A conflict message blocks a release, and forcing makes the message go away immediately.

What goes wrong

Ownership flips between parties, and hidden fights begin. The real disagreement is never solved.

Better choice

Investigate each conflict, decide the true owner, and change the other party’s file or code.

ANTI-PATTERN 02 · APPLYING EXPORTED LIVE OBJECTSAvoid
Why teams do it

Exporting a working object feels like the quickest way to get a correct file.

What goes wrong

The file claims ownership of defaults, status, and other parties’ fields, creating conflicts and fights later.

Better choice

Write minimal files by hand or from templates, listing only fields you intend to own.

ANTI-PATTERN 03 · CHANGING MANAGER NAMES OFTENAvoid
Why teams do it

Names are generated automatically from build numbers or tool defaults.

What goes wrong

Old names keep ownership of fields, new runs cannot remove them, and the record grows without limit.

Better choice

Pick one stable, documented name per pipeline and never change it casually.

ANTI-PATTERN 04 · MIXING OLD AND NEW APPLY ON ONE OBJECTAvoid
Why teams do it

Different teams adopt the new method at different speeds.

What goes wrong

The two methods follow different rules about merging and removal, so changes appear or vanish unexpectedly.

Better choice

Migrate each object completely, one pipeline at a time, and retire the old method for that object.

ANTI-PATTERN 05 · TRUSTING THE MANAGER NAME FOR SECURITYAvoid
Why teams do it

The record looks like an audit trail, so it is easy to treat as one.

What goes wrong

Any caller can choose any name, so the record can be misleading in an investigation.

Better choice

Use the audit log for identity and the ownership record for coordination.

ANTI-PATTERN 06 · DROPPING A FIELD WITHOUT A HAND-OVERAvoid
Why teams do it

It seems natural to delete a line from a file when you no longer want to manage it.

What goes wrong

If you were the only owner, the server removes the field, and the default value returns, such as one replica.

Better choice

Make sure another owner holds the field before you stop declaring it, or accept the removal knowingly.

25Managed Kubernetes and Cloud Add-ons

On managed services from the big cloud providers, many components inside your cluster are installed and updated by the provider. Those components are writers too, and Server-Side Apply is how you and the provider avoid stepping on each other.

Who else writes to your cluster

On a managed cluster, the provider often installs networking, DNS, and storage components, and keeps them up to date. When you customize one of them, for example by adjusting the resources given to the cluster’s DNS pods, you become a second writer on an object the provider also manages. Without ownership tracking, the next provider upgrade could silently erase your change, or your change could silently break the upgrade.

How providers use ownership

Amazon’s managed Kubernetes service, for example, offers managed add-ons and lets you choose how conflicts are resolved when you have customized an add-on’s settings. Typical choices are to overwrite your changes with the provider’s values, to preserve your changes, or to do nothing and report the problem. Behind these options is the same idea: fields are owned by a party, and the provider’s tool must decide what to do when you own a field it wants to change. Other providers offer similar behavior for their own managed components. Option names and defaults vary by provider and version, so confirm them in the current documentation.

Provider policyWhat happens to your customized fieldsBest used when
OverwriteThe provider’s values replace yoursYou want the provider’s standard setup and have no real customizations
PreserveYour values are kept, and the provider changes only other fieldsYou have deliberate customizations that must survive upgrades
Fail or reportThe upgrade stops and shows the conflictYou want a human to review every disagreement
Everyday analogy

Think of renting a furnished apartment. The landlord maintains the building, but you hang your own pictures. When the landlord repaints, a good agreement says whether your pictures should be taken down, left in place, or whether the painter should stop and ask you first.

Practical advice for managed clusters

1

Know what the provider owns

Read the ownership record on provider-installed objects before changing them.

2

Customize through supported settings

Prefer the provider’s configuration options over raw edits of the objects.

3

Use a named manager for your changes

If you must edit, apply with a clear name so your fields can be recognized later.

4

Test upgrades in a staging cluster

Confirm what happens to your customizations before the same upgrade reaches production.

Production example: customizing a cluster DNS add-on

A common case is raising the resources or replica count of the cluster’s DNS component for a busy cluster. Teams that apply this change under a named manager, and choose the preserve policy, find that provider upgrades keep their tuning. Teams that edit by hand and choose overwrite sometimes find their tuning reset after an upgrade, and the cluster slows down again.

26A Decision Guide: Should You Use It, and How?

Server-Side Apply is the recommended direction for new work, but a few questions help you decide how to adopt it and how to configure each writer.

flowchart TD
  S["Who is writing this object?"] --> Q1{"Is it a controller or Operator?"}
  Q1 -->|"yes"| C1["Apply with one stable manager name; apply only the fields it manages; force only its own fields"]
  Q1 -->|"no"| Q2{"Is it an automated pipeline?"}
  Q2 -->|"yes"| C2["Apply with a unique manager name; dry-run first; do not force by default"]
  Q2 -->|"no"| Q3{"Is it a person making a one-off change?"}
  Q3 -->|"yes"| C3["Prefer a change through the pipeline; if urgent, apply with your own name and tell the owner"]
  Q3 -->|"no"| C4["Review who the writer is and why"]
  C1 --> E["Does another writer share a field?"]
  C2 --> E
  C3 --> E
  E -->|"yes"| H["Decide one primary owner and plan a hand-over"]
  E -->|"no"| OK["Proceed"]
    
Figure 16: Choosing how each kind of writer should use Server-Side Apply

A comparison of the two apply methods

QuestionClient-side applyServer-side apply
Where is the merge computed?In the client toolOn the API server
How is ownership tracked?Through a saved copy in an annotationThrough a field-level record in the object’s metadata
What happens on disagreement?The last writer wins silentlyA conflict is reported to the applier
Handling of custom resourcesWeak for listsStrong, guided by schema hints
Size limits from trackingAnnotation limit appliesNo annotation, but ownership data adds size
Best fitSimple, single-writer setupsShared objects, controllers, GitOps

When the old method is still fine

If a small team manages a handful of simple objects, with a single pipeline that is the only writer, the old method may serve perfectly well, and there is little pressure to move. The strongest reasons to adopt Server-Side Apply are: several writers on the same objects, controllers that need safe writes, custom resources with lists, very large objects, and a wish for clear, field-level accountability.

i
Start where the pain is

Do not migrate everything on day one. Begin with the objects that already cause confusion, such as those shared with an autoscaler or an Operator, and learn from them before moving the rest.

27A Learning Path for Your Team

Reading about ownership is useful, but the ideas become natural only through practice. This short path turns the chapters above into a plan any team can follow over a few weeks.

1

Week one: observe

Display the ownership record of a few ordinary objects in a test cluster. Learn to read manager names, operations, and field lists without changing anything.

2

Week two: create a harmless conflict

Apply the same object from two different manager names with different values. Watch the refusal message, then resolve it in each of the three ways: change the value, remove the field, and force.

3

Week three: practice a hand-over

Set up an autoscaler on a test workload and move the replica field from your file to the autoscaler without any dip in running copies.

4

Week four: switch one real pipeline

Pick one low-risk pipeline, choose its manager name, add a server dry run and a diff, and move it across with the team watching.

5

Afterwards: write the agreement

Record your naming rule, your rule about forcing, and your list of shared fields in one page of platform documentation.

i
Practice in a disposable place

Use a throwaway namespace or a small local cluster for the exercises. Seeing a conflict and a field removal with no consequences is the fastest way to build the confidence you will want when the same thing happens in production.

28Frequently Asked Questions

These are the questions people ask most often when they first meet Server-Side Apply.

Q1What is Server-Side Apply in one sentence?
It is a way of changing Kubernetes objects where you send only the fields you care about, the server merges them into the stored object, and the server records which writer owns each field.
Q2Is it a replacement for kubectl?
No. You still use the same command line tool. You simply ask it to send requests in the server-side style by adding an option. Controllers and other programs can use it directly through the API.
Q3When did it become stable?
To my knowledge it appeared as an early preview around version 1.14, moved to beta around 1.16, and became generally available in 1.22. Check the release notes for your own version to be certain.
Q4What is a field manager?
It is the name attached to a writer’s changes. The server uses it to record who owns each field. You choose it with an option on the command line, or a controller sets it in its code.
Q5What is a conflict?
A conflict happens when you apply a value for a field that another manager owns, and your value is different. The server refuses the request and tells you which field and which manager. If the values are the same, ownership is simply shared.
Q6Should I use the force option?
Rarely. It takes ownership of the disputed fields from the other writer. It is appropriate for a controller that truly owns its fields, but for pipelines and people it usually hides a disagreement that should be solved properly.
Q7Why did a field disappear when I removed it from my file?
Because you were its only owner. With server-side apply, leaving out a field you used to own means you no longer want it, so the server removes it if nobody else owns it. If another owner exists, the field stays.
Q8Does an ordinary update raise conflicts too?
No. An update silently takes ownership of the fields it changes. Only apply requests are checked for conflicts. This is why occasional hand edits can later cause a conflict for a pipeline.
Q9Is the manager name a security feature?
No. Any caller can choose any name. Use it for coordination and debugging, and use the audit log and access control for security.
Q10How does it help with custom resources?
Authors can mark lists and objects in their schema as atomic, set-like, or keyed. The server then merges them correctly, so different writers can manage different items in the same list.
Q11Can I migrate objects created with the old method?
Yes. The first server-side apply on such an object converts the earlier record into ownership data. Review the diff carefully, because fields from the old record that are missing from your new file may be removed.
Q12Does it make my objects bigger?
Yes, a little. The ownership record adds data to each object, though it removes the old saved-copy annotation. Keep manager names stable and files minimal to keep the size under control.

29Summary and Key Takeaways

We began with a shared document that many people edit and nobody tracks. Kubernetes objects are exactly like that, and Server-Side Apply is the feature that adds the missing track changes.

What to remember

  • Ownership is recorded per field. The object’s metadata lists each manager and the exact fields it owns, so the question “who set this?” always has an answer.
  • The server does the merging. You send only the fields you care about, and built-in or schema-defined rules decide how maps and lists combine.
  • Conflicts are a feature. An apply that disagrees with another owner is refused with a clear message, instead of silently winning. Updates are the exception, since they take ownership quietly.
  • Leaving a field out means letting it go. If you were the only owner, the field is removed. Plan a hand-over before you stop declaring a shared field, such as replicas under an autoscaler.
  • Use stable, unique manager names and avoid force by default. Controllers may force their own fields, but pipelines and people should investigate conflicts first.
  • The manager name is not a security boundary. Use the audit log and access control for identity and protection, and the ownership record for coordination.
  • Adopt it where the pain is. Controllers, GitOps, custom resources, and shared objects benefit most, and simple single-writer setups can move at their own pace.