A scheduler is a matchmaker. The framework is the set of stages the matchmaker goes through, and plugins are the specialists at each stage.

A Pod is a small package that holds one or more containers. A node is a machine, physical or virtual, that can run Pods. A new Pod starts life with no node assigned. The scheduler is the control plane component that looks at every unassigned Pod, picks the best node for it, and writes that choice back into the cluster. After that, the node’s own agent, called the kubelet, takes over and actually starts the containers.

The scheduling framework is the way the scheduler organizes its thinking. Instead of one giant function, the work is split into named stages. Each stage is called an extension point, which is simply a spot where plugins can attach. A plugin is a small piece of logic that registers itself at one or more extension points. When the scheduler reaches a stage, it calls every plugin registered there, collects the answers, and moves on.

Two big questions the scheduler answers

Every scheduling decision answers two questions in order. The first is “Which nodes are even allowed to run this Pod?” That is called filtering. The second is “Among the allowed nodes, which one is best?” That is called scoring. Almost every built-in plugin exists to help answer one of these two questions, and understanding this split is the key to understanding everything else in this guide.

Understanding the problem makes the design feel obvious.

In the early versions of Kubernetes, the scheduler used two lists of rules. The first list, called predicates, were yes-or-no tests. The second list, called priorities, were scoring functions. These were written directly into the scheduler’s source code. If your company needed a special rule, such as “never place a database Pod on a machine that also runs a web server owned by another team,” you had two bad choices. You could change the scheduler’s source code and keep your own modified copy forever, or you could attach an outside web service to the scheduler.

The outside web service approach was called a scheduler extender. The scheduler would send a network request carrying the Pod and a list of nodes, and the extender would reply with a shorter list or with scores. Extenders worked, but they had real limits. They were slow because every call crossed the network. They could only join at a few fixed points. They could not see the scheduler’s internal cache. And they could not take part in the later steps, such as reserving space or holding a Pod until a group of Pods was ready.

The Kubernetes community introduced the scheduling framework to solve these problems. It rebuilt the scheduler so that even its own built-in behavior is made of plugins. This is an important point: the default scheduler is not special. It is simply a framework running a default list of plugins. When you tune it, you are changing which plugins run and how they are weighted. When you extend it, you are adding one more plugin to the same list the built-in ones use.

Eight short words that unlock every scheduler conversation.

Requests, not real usage

A beginner surprise: the scheduler does not look at how busy a node is right now. It looks at the sum of the resource requests of the Pods already placed there and compares that with what the node can offer. A node may be using only ten percent of its real CPU yet be “full” for scheduling because the Pods on it have asked for all of it. This is deliberate. Requests are stable promises, while live usage jumps up and down every second.

Status codes

When a plugin finishes, it returns a status, which is a small answer with a code. The important codes are easy to remember. Success means all is well. Unschedulable means “this node does not work for this Pod, but things might change.” UnschedulableAndUnresolvable means “no change can ever fix this, so do not bother trying other fixes such as preemption.” Error means something broke inside the plugin. Wait means “hold the Pod for a while.” Skip means “I have nothing to do for this Pod, so skip my later stages.” These codes control what the scheduler does next.

The scheduler is one program that listens to the API server and speaks only to it.

Kubernetes has a control plane, which is the brain, and many worker nodes, which are the muscle. Inside the control plane, the API server is the single front door and the etcd database is the single source of truth. The kube-scheduler never reads the database directly. It watches the API server, which means it keeps a long-lived connection and receives a message each time a Pod or a node changes.

flowchart TB
  U["User or controller creates a Pod"] --> API["kube-apiserver"]
  API  ETCD[("etcd datastore")]
  API -->|"watch Pods Nodes Volumes"| INF["Scheduler informers"]
  INF --> CACHE["Scheduler cache and node snapshot"]
  INF --> QUEUE["Scheduling queues"]
  QUEUE --> FW["Framework with plugins"]
  CACHE --> FW
  FW -->|"bind Pod to node"| API
  API -->|"watch Pods on this node"| KL["kubelet on chosen node"]
  KL --> CR["Container runtime"]
    

A Pod that cannot be placed does not vanish. It waits in the right line until things change.

The scheduler does not simply keep one list. It keeps three, and a Pod moves between them depending on what happens. Understanding the three lines explains why a Pod sometimes gets placed within milliseconds and why another one seems to sleep for minutes.

flowchart TD
  NP["New Pod without a node"] --> PE{"PreEnqueue gates pass?"}
  PE -->|"No"| GT["Held back as gated"]
  PE -->|"Yes"| AQ["activeQ ordered by QueueSort"]
  AQ --> CY["Scheduling cycle"]
  CY -->|"Node chosen"| BD["Binding cycle"]
  CY -->|"No node fits"| UP["Unschedulable pool"]
  UP -->|"Cluster event may help"| BQ["backoffQ"]
  UP -->|"Waited too long"| BQ
  BQ -->|"Backoff time over"| AQ
  BD -->|"Binding failed"| BQ
  BD -->|"Bound"| DN["Pod running on node"]
    

Why the unschedulable pool matters

Imagine a Pod that asks for eight GPUs, and the cluster has none free. Trying it again every second would waste the scheduler’s time and block everyone else. So the scheduler parks it in the unschedulable pool. Plugins can tell the scheduler which kinds of events could possibly help that Pod. If a node is added, a Pod is removed, or a label changes, the scheduler checks whether the event matters to the parked Pod and moves it out if so. As a safety net, any Pod that has waited in the pool for about five minutes is moved to be tried again anyway.

Scheduling gates

A scheduling gate is a label-like marker in the Pod’s specification that says “do not try to schedule me yet.” An outside controller can attach a gate when a Pod is created and remove it later, once some condition is true, such as “the quota system approved this job.” A built-in plugin called SchedulingGates works at the PreEnqueue extension point and keeps gated Pods out of the active queue entirely, so they cost the scheduler nothing while they wait.

Twelve named stages, split into two cycles. Learn this map once and every plugin makes sense.

The work of placing one Pod is split into two parts. The scheduling cycle chooses a node. It runs for one Pod at a time, in strict order, because each decision changes the picture for the next Pod. The binding cycle then carries out the decision. It can run for many Pods at once, because it mostly waits on the network. Together they form one scheduling attempt.

flowchart LR
  subgraph SC["Scheduling cycle - one Pod at a time"]
    PE["PreEnqueue"] --> QS["QueueSort"]
    QS --> PF["PreFilter"]
    PF --> F["Filter"]
    F --> PoF["PostFilter when no node fits"]
    F --> PS["PreScore"]
    PS --> S["Score"]
    S --> N["NormalizeScore"]
    N --> R["Reserve"]
    R --> P["Permit"]
  end
  subgraph BC["Binding cycle - can run in parallel"]
    PB["PreBind"] --> B["Bind"]
    B --> PO["PostBind"]
  end
  P --> PB
    

Filtering is the stage where impossible nodes are thrown out.

PreFilter: do the heavy thinking once

Suppose a Pod has a complicated rule that says “do not place me on a node that already has a Pod from the same team.” To check one node, you must look at all Pods across the cluster. Doing that scan once per node, for thousands of nodes, would be wasteful. A PreFilter plugin does the heavy scan a single time per Pod and stores the result. Later, the Filter plugin simply reads the stored result for each node.

PreFilter plugins can also act as early guards. If a Pod asks for something that is impossible in the whole cluster, such as a volume claim that does not exist, the PreFilter plugin can fail the Pod right away, saving the effort of testing every node. A PreFilter plugin can even hand back a shorter list of candidate nodes. For example, if a Pod names a specific node, the scheduler only needs to test that node.

Filter: the yes-or-no test on every node

The Filter stage asks every Filter plugin about every node. A node is kept only if every plugin says yes. The scheduler runs these checks in parallel, using many workers at once, because the same question is asked about thousands of nodes. For safety, a Filter plugin must not change anything. It only looks and answers.

The shortcut: not every node is examined

In a huge cluster, testing every single node would take too long. The scheduler therefore stops searching once it has found enough nodes that pass the filters. How many are enough is controlled by a setting called percentageOfNodesToScore. When left at its default, the scheduler uses a sliding rule: in small clusters it looks at about half of the nodes, and the share shrinks as the cluster grows, down to a minimum of five percent. It also always tries to find at least one hundred suitable nodes. In a cluster with fewer than a hundred nodes it simply examines all of them.

To avoid always favoring the same nodes at the start of the list, the scheduler remembers where it stopped last time and begins the next search from there, going round in a circle. This keeps the load spread across the cluster.

PostFilter: the rescue stage

If the Filter stage leaves no node at all, the Pod is not simply rejected. The scheduler calls the PostFilter plugins, which are the rescue team. The built-in one, called DefaultPreemption, tries to make room by removing less important Pods. You will meet it fully in Chapter 12. PostFilter only runs when filtering finds nothing, so on a healthy cluster it is rarely used.

When several nodes are acceptable, scoring decides which one wins.

After filtering, usually many nodes remain. Now the scheduler must choose one. It does this by asking every Score plugin to give each remaining node a number from 0 to 100. Higher is better. A node’s final score is the sum of what each plugin said, after each plugin’s number is multiplied by a weight that you can configure. The node with the highest total wins. If two nodes tie, the scheduler picks one of them at random, which helps spread load fairly.

PreScore and NormalizeScore

PreScore works like PreFilter: it does shared preparation once, using the list of nodes that survived filtering, so that scoring each node is quick. NormalizeScore is an optional step that belongs to a Score plugin. Some plugins compute raw numbers in odd ranges, such as the count of matching Pods. Normalization rescales those raw numbers into the 0 to 100 range so that plugins can be fairly added together.

The main scoring plugins

Hard rule versus soft rule: one plugin, two stages

Several plugins appear in both lists. The reason is that Kubernetes lets you write the same kind of rule in two strengths. A required rule becomes a Filter: break it and the node is out. A preferred rule becomes a Score: break it and the node just loses points. Topology spread shows this clearly. Setting it to “do not schedule” when the spread cannot be kept makes it a hard Filter. Setting it to “schedule anyway” turns it into a soft Score.

Choosing a node on paper is not the same as placing the Pod there. These last stages close the gap safely.

Reserve: hold it before someone else takes it

As soon as a node is chosen, the scheduler immediately assumes the Pod is on that node in its own cache, even though the real binding has not happened yet. This is so that the very next Pod’s decision already sees the space as taken. Reserve plugins take part in this step. A plugin that tracks something stateful, such as which storage volume was matched to which claim, records its choice during Reserve. Every Reserve plugin also has an Unreserve step, which undoes the hold if anything later fails. Unreserve runs in the reverse order, like unwinding a stack.

Permit: the final gatekeeper

The Permit stage is the last chance to stop or delay a Pod before binding. A Permit plugin can answer in three ways. It can approve, which lets the binding continue. It can deny, which sends the Pod back to the queue and triggers Unreserve. Or it can tell the Pod to wait, with a time limit that is at most fifteen minutes. A waiting Pod stays reserved on its node and sits in a holding area until another plugin approves it or the time runs out.

The wait option is what makes gang scheduling possible, which is the rule “start all of these Pods together or none of them.” You will see this in Chapter 15.

PreBind: last preparation

PreBind plugins do work that must finish before the Pod is attached to the node. The best known example is the VolumeBinding plugin. If a Pod needs a storage volume that has to be created or attached on that particular node, the plugin triggers that work here and waits until it is done. If PreBind fails, the whole reservation is undone and the Pod returns to the queue.

Bind and PostBind: the finish line

A Bind plugin performs the actual write that links the Pod to the node. By default, the plugin called DefaultBinder sends a binding request to the API server. Several Bind plugins can exist, and they are tried in order. A plugin that does not want to handle a particular Pod returns the Skip status, and the next plugin gets its turn. This allows, for example, a custom binder for special devices. After a successful bind, PostBind plugins run for information only, such as sending a notice or cleaning up temporary data. They cannot change the result.

A map of which built-in plugin works at which stage.

Look carefully at this table. A plugin like NodeResourcesFit touches three stages. That tells you something about the design: the stages are the vocabulary, and each plugin is a sentence made from those words. When you understand which stages exist, you can read the whole table like a story.

Let us follow a single web-server Pod from creation to running, with the plugins visible at every step.

sequenceDiagram
  participant U as User
  participant API as API Server
  participant Q as Scheduler Queue
  participant FW as Plugins
  participant K as Kubelet
  U->>API: Create Pod with no node
  API-->>Q: Watch event for unscheduled Pod
  Q->>FW: Pop highest priority Pod
  FW->>FW: Filter nodes then score survivors
  FW->>FW: Reserve and Permit on best node
  FW->>API: Bind Pod to chosen node
  API-->>K: Watch event for Pod on this node
  K->>API: Report Pod Running
    

Sometimes the cluster is full and an important Pod arrives. Preemption is the fair way to make room.

Every Pod can have a priority, a whole number taken from an object called a PriorityClass. A payment service might have a high priority and an overnight report job a low one. Priority has two effects. First, the queue’s PrioritySort plugin puts higher priority Pods ahead of others. Second, if a high priority Pod cannot be placed anywhere, the DefaultPreemption plugin may remove lower priority Pods to make space. Removing them is called preemption.

flowchart TD
  A["Pod cannot fit on any node"] --> B["PostFilter runs DefaultPreemption"]
  B --> C{"Does Pod outrank some running Pods?"}
  C -->|"No"| X["Pod stays pending"]
  C -->|"Yes"| D["For each node pick lowest priority victims"]
  D --> E["Choose the node with least damage"]
  E --> F["Delete victims and set nominated node"]
  F --> G["Pod is scheduled once space frees"]
    

You change the scheduler’s behavior without touching its source code, by editing a configuration file.

The scheduler reads a configuration object called KubeSchedulerConfiguration when it starts. Inside, you describe one or more profiles. A profile has a name, a list of which plugins are enabled or disabled at each extension point, and optional settings for individual plugins, called plugin arguments. Pods choose a profile by naming it in their schedulerName field. If they do not name one, they use the default profile.

Several scheduling personalities in one process

Because a single scheduler can run many profiles, one running program can behave in different ways for different workloads. A common setup defines a default profile that spreads web services widely, and a second profile, perhaps named “bin-packing,” that packs batch jobs tightly. Teams then choose by setting a scheduler name on their Pods. A restriction to remember is that all profiles in one scheduler must use the same QueueSort plugin, since there is only one queue.

When the built-in plugins are not enough, you can add your own, and the scheduler treats it exactly like a built-in one.

Custom plugins are written in the Go language, the same language as Kubernetes itself. A plugin is a small component that declares its name and implements one or more of the extension point interfaces. You then build your own copy of the scheduler program, called an out-of-tree build because the plugin lives outside the main Kubernetes source tree. In that build, you register the plugin with the framework. Finally you list it in the scheduler configuration so that it is switched on at the right stages. Because the plugin is compiled into the scheduler, it runs in the same process and has no network cost.

CycleState: a notebook for one scheduling attempt

The scheduling cycle is split into many stages, but plugins often need to pass information from one stage to another. For this, the framework gives every attempt a small storage area called CycleState. A PreFilter plugin writes its precomputed result into it, and the Filter plugin for the same topic reads it later. The notebook is thrown away when the attempt ends. Because Filter runs on many nodes in parallel, whatever is stored must be safe to read from many workers at once.

Do not start from zero: the scheduler-plugins project

A community project under the Kubernetes organization, called scheduler-plugins, collects ready-made out-of-tree plugins. It includes a Coscheduling plugin for gang scheduling, a CapacityScheduling plugin for elastic resource quotas, plugins that understand the internal topology of a node such as NUMA zones, and plugins that place Pods based on real usage measurements. Many teams install this project’s scheduler build and simply turn on the plugins they need.

Plugins versus extenders versus a whole new scheduler

The framework truly shines in the cases that the old scheduler could never handle well.

Gang scheduling with Permit

Some jobs only make sense if all their Pods start together. A distributed machine learning training job with eight workers is useless if only five are running, and worse, those five hold expensive GPUs while waiting for the other three. The classic fix is gang scheduling, also called coscheduling: place all or none. A Coscheduling plugin implements this using the Permit stage.

sequenceDiagram
  participant P1 as Worker Pod 1
  participant P2 as Worker Pod 2
  participant P3 as Worker Pod 3
  participant PM as Permit plugin
  P1->>PM: Reserved on node A, ask permission
  PM-->>P1: Wait, group has 1 of 3
  P2->>PM: Reserved on node B, ask permission
  PM-->>P2: Wait, group has 2 of 3
  P3->>PM: Reserved on node C, ask permission
  PM-->>P3: Allow, group complete
  PM-->>P1: Allow waiting Pod
  PM-->>P2: Allow waiting Pod
    

Each Pod in the group passes filtering and scoring on its own and gets reserved on a node. At Permit, the plugin counts how many group members are reserved. Until the count reaches the required number, each Pod waits. When the last one arrives, all waiting Pods are approved together and their binding cycles proceed at the same moment. If the timeout runs out first, the waiting Pods are rejected, their holds are undone, and the capacity returns to the cluster.

Topology awareness

Modern servers have several processors, and memory attached to each. A Pod that uses a fast network card and a GPU runs best if all of its resources sit close together inside the machine. Plugins in the scheduler-plugins project read information about each node’s internal layout and prefer nodes where the Pod’s resources align. In the same way, the built-in topology spread plugin works at a larger scale, keeping copies of a service apart across zones and racks.

Dynamic resource allocation

Devices like GPUs used to be described only as simple counters. A newer model lets Pods request devices through claims that carry detailed properties, and the DynamicResources plugin takes part in PreFilter, Filter, Reserve, and PreBind to match claims to devices on suitable nodes. This allows richer requests than counting, such as “a GPU with at least this much memory.”

Queueing hints

Earlier you learned that Pods in the unschedulable pool are moved back when an event might help them. In recent Kubernetes versions, plugins can also give finer queueing hints, which are small functions that say whether a specific event really helps a specific Pod. This avoids needless retries. For example, a new node that has the wrong label does not need to wake a Pod that requires a different label. Fewer pointless retries means more time for real scheduling work.

The scheduling cycle is deliberately one Pod at a time, so every millisecond inside a plugin counts.

Official Kubernetes guidance for a single cluster describes clusters of up to about five thousand nodes and about one hundred and fifty thousand Pods. In such a cluster, the scheduler must examine many nodes for every Pod. Because the scheduling cycle is serial, the time one Pod spends being scheduled limits how many Pods per second the whole cluster can place. Multiply a slow plugin by thousands of nodes and thousands of Pods, and the delay becomes large quickly.

The cost of one expensive plugin

Suppose a custom Filter plugin takes one millisecond per node. With a thousand nodes and sixteen parallel workers, that adds roughly sixty milliseconds to every Pod, and during a big rollout of ten thousand Pods, the extra time adds up to many minutes of waiting. A plugin that cached its data in PreFilter and answered in microseconds would remove almost all of that. Always measure a plugin under realistic cluster size before running it in production.

If the scheduler stops, new Pods stay Pending. Running Pods keep running, which limits the damage.

The scheduler affects only Pods that are not yet placed. If it stops for a minute, existing applications keep running without a single change. What suffers is anything new: scale-ups, replacements for crashed Pods, and rollouts. For this reason, production clusters run several copies of the scheduler, but only one of them is the active leader at any time. The leader holds a lease, a small object that it must keep renewing. If the leader dies, the lease expires and a standby takes over after a few seconds, as you saw in the controller pattern.

Why a restart is safe

The scheduler keeps no precious private memory. Its queues and cache are rebuilt from the API server, which stores the true state. A restarted scheduler lists all Pods and nodes again, finds the unplaced Pods, and carries on. This is the same level-based thinking used by controllers: always derive the work from the current state, not from remembered events.

The scheduler decides where every workload runs, so it deserves careful protection.

The scheduler holds a powerful position. It reads Pods, nodes, volumes, and claims, and it writes bindings. If an attacker could control it, they might force sensitive Pods onto a machine they own, or starve important Pods of space. Several layers protect it.

Scheduling rules are not security boundaries

Taints, affinity, and node selectors keep workloads apart in normal operation, but they are not strong walls. A user who can create Pods may be able to add a toleration or a node name and cross the line. For true isolation of sensitive workloads, use admission control to enforce which tolerations and selectors are allowed in a namespace, and combine it with node-level protections. Tenants who share a cluster should never rely on scheduling hints alone.

A quiet scheduler is a healthy one, but only if you can prove it.

The scheduler publishes metrics that Prometheus can collect. The most useful ones answer four simple questions: how many Pods are waiting, how long do they wait, how long does each attempt take, and which plugin is slow.

A calm order for debugging a Pending Pod

Every design choice has a price. Knowing it helps you decide where to stay on defaults and where to customize.

The missing piece: the scheduler decides once

The scheduler makes its choice at the moment a Pod is created. It does not later move Pods around as conditions change. Over weeks, some nodes may become crowded and others empty. A separate project called the descheduler exists for this purpose. It periodically finds Pods that now violate their rules or that sit in poor places, and evicts them so that the scheduler can place them again with fresh information. The scheduler places, and the descheduler rebalances.

A few habits keep scheduling predictable. A few others cause midnight pages.

Patterns worth copying

Anti-patterns to avoid

A short checklist before you tune or extend the scheduler.

Mistakes beginners make as cluster users

• Assuming the scheduler sees real usage. It plans from requests. If requests are too low, nodes become overloaded. If too high, nodes look full while idle.
• Misspelling labels in affinity rules. A required rule with a label that exists on no node blocks the Pod forever.
• Forgetting tolerations for tainted nodes. Dedicated GPU or system pools often carry taints, and Pods without matching tolerations never land there.
• Over-using required anti-affinity. Asking for each replica to be on a different node fails once replicas outnumber nodes. Prefer topology spread with a soft setting.
• Ignoring PodDisruptionBudgets. They influence preemption and eviction, and badly set budgets can block maintenance.
• Expecting automatic rebalancing. New nodes do not pull existing Pods across. Use the descheduler or a rollout to rebalance.

The framework is not just a lab idea. It shapes how real platforms run their work.

Short answers to the questions beginners ask most often.

You began with a hotel front desk and finished with a full picture of how Kubernetes places work. The scheduler is a pipeline with fixed stages, and plugins are the specialists who fill those stages. Filtering removes nodes that cannot work, scoring ranks those that can, and the final stages hold, approve, and write the decision safely. Because even the default scheduler is made of plugins, tuning and extending it follow the same simple rules.