To understand any tool, first understand the pain that made someone invent it.

A quick refresher

Kubernetes is a system that runs your applications across many computers. Each computer in the group is called a node, and the whole group is a cluster. Your applications run inside pods, which are small wrappers around one or more containers. Normally, you tell Kubernetes “run three copies of my web application,” and it picks any nodes it likes.

But some programs are node helpers

Certain programs are not about your application at all. They are about the machine itself. Think about what a busy operations team needs from every node in the cluster.

Why ordinary tools fall short

A Deployment, the usual manager for everyday applications, lets you say “keep N copies.” But it decides freely where they go. It might put two copies on one node and none on another. Even if you set N equal to the current number of nodes, the moment a node is added or removed, the count is wrong, and somebody has to remember to change it.

Doing this by hand is a recipe for gaps. The one node without a log collector is the one where the problem happens, and nobody sees it.

A DaemonSet is a Kubernetes object that ensures a copy of a pod runs on all, or a chosen group of, nodes in the cluster.

Let us unpack that. A Kubernetes object is a record you store in the cluster describing something you want. A copy of a pod means one pod built from a template you provide. All nodes, or a chosen group means the default is every node, but you can narrow it using labels and rules, such as only nodes with graphics cards.

The word daemon comes from the Unix world. A daemon is a background program that quietly does a supporting job without any user interacting with it, like a print spooler or a clock updater. A DaemonSet is a set of such background programs, one per machine.

The behaviour in four sentences

flowchart TB
  DS[DaemonSet: log collector]
  DS --> N1[Node 1 with one collector pod]
  DS --> N2[Node 2 with one collector pod]
  DS --> N3[Node 3 with one collector pod]
  NEW[New Node 4 joins] -.->|pod added automatically| DS
        

Kubernetes offers several managers for pods. Choosing the right one is half the skill.

DaemonSet versus Deployment

A Deployment cares about how many copies exist. A DaemonSet cares about which nodes have a copy. If your question is “how many web servers do I need for my traffic?” you want a Deployment. If your question is “which programs must exist on every machine?” you want a DaemonSet.

DaemonSet versus static pods

Static pods are defined as files on a particular node and are run directly by that node’s manager program, the kubelet. They are tied to one machine and are not managed through the cluster’s normal controllers. A DaemonSet, in contrast, is a proper cluster object: you manage it centrally, update it with rolling updates and watch it through the API.

Eight simple terms make the rest of the story easy.

Taints and tolerations in plain words

Some nodes are deliberately kept for special work. For example, the nodes that run the cluster’s control plane, its brain, are normally marked so that ordinary application pods stay away. That mark is a taint. A pod that wants to run there must carry a matching toleration. DaemonSets often need such tolerations, because a monitoring agent that skips the control plane nodes leaves blind spots.

The DaemonSet controller is a tireless comparer of two lists: the nodes that should have a pod and the nodes that do.

Kubernetes works on the idea of desired state and actual state. You describe what you want, and controllers keep comparing it with reality. The DaemonSet controller lives in the control plane, the cluster’s brain, and continuously performs this simple loop.

flowchart LR
  A[Watch nodes and DaemonSets] --> B[List nodes that should run the pod]
  B --> C[List nodes that already have the pod]
  C --> D{Any difference?}
  D -->|Node missing a pod| E[Create a pod for that node]
  D -->|Pod on wrong node| F[Delete that pod]
  D -->|No difference| G[Do nothing]
  E --> A
  F --> A
  G --> A
        

What the loop checks

Every DaemonSet pod is created for one specific node. The interesting part is how it gets there.

In older Kubernetes versions, the DaemonSet controller placed pods on nodes directly, bypassing the scheduler, the matchmaker that normally chooses a home for each pod. In modern versions, the controller instead creates each pod with a rule that says “this pod must run on node X,” and lets the ordinary scheduler carry out the placement. This means DaemonSet pods enjoy the same features as any other pods, such as priorities and preemption.

sequenceDiagram
  participant U as You
  participant A as API Server
  participant D as DaemonSet Controller
  participant S as Scheduler
  participant K as kubelet on Node 3
  U->>A: Create DaemonSet
  D->>A: Watch and see DaemonSet
  D->>A: Create one pod per eligible node
  Note over D,A: Each pod carries a rule pinning it to one node
  S->>A: Bind each pod to its pinned node
  K->>A: Any pods assigned to me?
  A-->>K: Yes, start this pod
  K->>A: Report pod running
        

Why pinning matters

Each DaemonSet pod carries an instruction, in the form of node affinity, that matches exactly one node’s name. That guarantees that the pod meant for Node 3 cannot end up on Node 5. The scheduler still checks that the node has enough room, but the choice of node is not open.

If it must be on every machine, it is a DaemonSet candidate. Here are the classic jobs.

“Every node” is the default, but you can narrow the group in several ways.

Node selector: the simple filter

A node selector says “only nodes carrying this label.” If you label your machines that have graphics cards, a DaemonSet with that selector will run only on them. Add the label to a new node and a pod appears. Remove the label and the pod is deleted.

Node affinity: the richer filter

Node affinity offers more expressive rules, such as “nodes whose zone is one of these three” or “prefer nodes with fast disks.” For DaemonSets, the hard “required” form is the one that matters, because a DaemonSet pod either belongs on a node or does not.

Tolerations: reaching reserved nodes

Nodes reserved for special purposes carry taints. To include them, add tolerations to the pod template. The most common case is the control plane nodes. A monitoring or logging agent that cannot run there leaves the cluster’s most important machines unobserved.

flowchart TB
  DS[DaemonSet with node selector gpu equals true]
  DS --> A[Node A labelled gpu: pod runs]
  DS --> B[Node B labelled gpu: pod runs]
  DS -. skipped .-> C[Node C no label: no pod]
  DS -. skipped .-> D[Node D no label: no pod]
        

Changing a node helper on hundreds of machines at once is risky. Update strategies control the pace.

When you change the pod template, for example to use a newer version of an agent, Kubernetes must replace the pods on every node. It offers two update strategies, chosen in the DaemonSet’s description.

Surge: starting the new before stopping the old

Recent Kubernetes versions also support a maximum surge setting for DaemonSet rolling updates. With surge, the new pod is started on a node before the old one is stopped, so the node is never without its helper. This helps for programs such as network agents or ingress-style helpers where a gap would hurt. There is a caution: for a short time, two pods of the same DaemonSet coexist on one node, so they must not fight over shared resources such as a fixed port.

flowchart LR
  V1[All nodes on version 1] --> R1[Update node 1]
  R1 --> R2[Update node 2]
  R2 --> R3[Update node 3]
  R3 --> V2[All nodes on version 2]
  R2 -. problem found .-> RB[Roll back to version 1]
        

Because each pod serves its own node, the usual “send traffic to any copy” approach often does not fit.

A web application behind a Service can be reached at any of its pods, and the Service spreads traffic. A node helper is different: a pod collecting logs on Node 3 has meaning only for Node 3. So how do other programs reach it? Several patterns exist.

flowchart LR
  subgraph N1[Node 1]
    APP1[Application pod] -->|local call| AG1[Agent pod]
  end
  subgraph N2[Node 2]
    APP2[Application pod] -->|local call| AG2[Agent pod]
  end
  AG1 -->|pushes data| CENTRAL[Central server]
  AG2 -->|pushes data| CENTRAL
        

Node helpers often need to see things that ordinary pods are kept away from. That power needs care.

Normal pods live in a sealed space with little view of the machine around them. But a log collector must read the log folder on the node, a monitoring agent must read the machine’s own statistics, and a security tool must see other programs’ activity. DaemonSet pods therefore often ask for special access.

One pod per node sounds small, but it is multiplied by every node in the cluster.

Suppose an agent asks for a small slice of processor and memory. On a cluster of five nodes, this is barely noticeable. On a cluster of five hundred nodes, the same small request becomes five hundred slices reserved across the fleet. And that reservation is taken from every node before your own applications get their share.

Requests and limits

Every container can declare a request, the amount it needs and that the scheduler reserves, and a limit, the maximum it may use. For DaemonSets, both should be chosen with care. Too small, and the agent is starved or killed for exceeding memory. Too large, and every node loses capacity you could have used for applications.

Priority: helpers must survive pressure

When a node runs short of resources, Kubernetes may remove low-priority pods to protect itself. Node helpers that the machine depends on, such as networking agents, should not be the first to go. Kubernetes provides built-in high-priority classes for system-level components, and critical DaemonSets can use them so that they are protected and, if needed, can even push out less important pods.

A helper that fails silently defeats its purpose. Kubernetes gives several safety nets.

Restart policy

DaemonSet pods must use the restart policy Always. If a container crashes, the kubelet restarts it in place, with growing delays between attempts if it keeps failing. This matches the purpose: a node helper should always be running.

Probes

Probes are small regular health checks. A liveness probe restarts the container if it seems stuck. A readiness probe tells Kubernetes whether the pod is ready, which also controls whether a rolling update may move to the next node. A startup probe gives slow programs time to start. Well-chosen probes make updates safer and failures easier to spot.

Surviving troubled nodes

Since node helpers exist to support troubled machines, Kubernetes automatically gives DaemonSet pods tolerations for common node problems, such as running low on memory or disk, or becoming unreachable. Regular application pods would be moved away in these situations, but DaemonSet pods are designed to stay.

stateDiagram-v2
  [*] --> Pending
  Pending --> Running: node has room
  Running --> Restarting: container crashes
  Restarting --> Running: restart succeeds
  Running --> Removed: node deleted or no longer matches
  Removed --> [*]
        

A pod that runs on every node, often with special powers, is a high-value target and a high-impact mistake.

Ironically, the programs that watch everything also need watching.

Reading the DaemonSet’s own numbers

Every DaemonSet reports a handful of counts in its status. Learning to read them is the fastest way to spot trouble.

The three pillars for DaemonSet pods

When a DaemonSet misbehaves, compare its numbers, then follow the chain from node to pod.

A DaemonSet is the right tool for one job, and the wrong tool for most others.

When a DaemonSet is a good fit

Choose one when the program is tied to the machine, not to your application’s load: logs, node metrics, networking, storage plumbing, security monitoring and hardware exposure. Also choose it when you want new nodes to be covered automatically.

When to choose something else

If you need a certain number of copies, use a Deployment. If you need stable identities and disks, use a StatefulSet. If the task finishes, use a Job. And if the helper could instead be added as a sidecar inside each application’s pod, consider whether per-pod coverage suits better than per-node coverage.

Habits that experienced teams follow, and traps they learned to avoid.

Helpful patterns

Traps to avoid

Simple habits that prevent most early pain.

You may have used a cluster for months without noticing that DaemonSets were quietly working.

No. There is no replica setting. The number of pods equals the number of eligible nodes. To change it, change which nodes are eligible using labels, affinity or tolerations.

Not by default, because those nodes normally carry a taint that repels ordinary pods. Add a matching toleration to the pod template if you want the pod there.

If the node matches the DaemonSet’s rules, the controller creates a pod for it automatically within moments of the node registering. No manual action is needed.

The pod on that node is cleaned up. It is not moved to another node, because every other eligible node already has its own pod.

Normally no. The design is one pod per eligible node. The exception is a brief overlap during a rolling update that uses surge, where the new pod starts before the old one stops.

Change the pod template, for example the image version. With the default rolling update strategy, Kubernetes replaces pods node by node. With the OnDelete strategy, pods update only when you delete them.

Yes. Kubernetes stores revisions of the DaemonSet, so you can return to an earlier version using the rollout undo feature of the command-line tool.

By default all its pods are deleted too. If you delete with the orphan option, the pods are left running without an owner, which is sometimes useful when replacing one DaemonSet with another.

Most often that node has too little free processing power or memory for the pod’s requests, or a fixed host port is already taken. Describing the pod shows the reason in its events.

No. A sidecar is a helper container inside each application pod, one per pod. A DaemonSet pod is one per node and serves all pods on that node. Choose based on whether the helper belongs to an application or to a machine.

Often yes. Kubernetes automatically adds tolerations for common node conditions, such as not ready or resource pressure, so that node helpers stay in place even when ordinary pods would be moved away.

A DaemonSet is a Kubernetes object that keeps exactly one pod running on every eligible node, or on a chosen group of nodes. It exists because some programs, such as log collectors, monitoring agents, network plugins, storage helpers and security tools, belong to the machine, not to your application’s load. A DaemonSet controller constantly compares the list of nodes with the list of pods, creating pods for new nodes and cleaning up after departed ones.

Around this simple idea sits a rich set of controls: node selectors, affinity and tolerations to choose where pods run, rolling updates and rollbacks to change versions safely, probes and priorities to keep helpers healthy, and careful security settings because these pods often reach deep into the host. The costs are multiplied by node count, and so are the risks, which makes measured resources and strict permissions especially important.

Keep the hotel chain picture in mind. Head office does not chase each branch about its fire alarm. It sets one rule: every branch, always, exactly one. New branches get one automatically, and a watchful manager keeps checking that the rule holds. That is the DaemonSet.