Start with the simplest possible picture, then add detail one layer at a time.

A Pod is the smallest unit you can run in Kubernetes. It holds one or more containers that share a network address and storage. When you create a Pod, it starts life with no machine assigned. It is simply a request that says “please run this somewhere”. The machines in the cluster are called nodes. The component that matches Pods to nodes is the kube-scheduler.

Every time the scheduler takes one Pod from its waiting line and works on it, that piece of work is called a scheduling attempt. Each attempt has two stages:

Together, the two stages are known as the scheduling context of a Pod. The official Kubernetes documentation describes the rule in one sentence: a scheduling cycle selects a node for the Pod, and a binding cycle applies that choice to the cluster. Scheduling cycles run one after another, never overlapping. Binding cycles, however, may run at the same time as each other and at the same time as the next Pod’s scheduling cycle.

The one-sentence difference

The scheduling cycle is about deciding and is kept strictly one-at-a-time so that decisions stay consistent. The binding cycle is about doing and is allowed to run in parallel because it may need to wait for slow things.

The problem: decisions are fast, side effects are slow

Choosing a node is mostly arithmetic. The scheduler asks questions such as “does this node have enough free memory?” and “does it carry a taint that this Pod does not tolerate?”. All of the information is already in the scheduler’s memory, so each question is answered in microseconds.

Making the decision real is a different story. The scheduler might have to wait for a cloud disk to be created and attached, for a network call to the API server to succeed, or for a group of related Pods to all be ready to start together. These actions cross the network and can take seconds or even minutes.

Imagine if the scheduler did both jobs in a single loop, one Pod at a time. A single slow disk creation would freeze scheduling for every other Pod in the cluster. In a busy cluster with thousands of Pods arriving per minute, that would be a disaster. The line would grow and grow while the scheduler stared at one slow disk.

The second problem: parallel decisions can collide

The obvious fix is to make the whole thing parallel. Let many workers each take a Pod and place it. But now two workers might look at the same node at the same moment, both see room for one more Pod, and both choose it. The node ends up overbooked. Fixing such collisions after the fact means undoing work and creating confusing failures.

The solution: serial decisions, parallel actions

Kubernetes keeps the decision part strictly serial, which makes it simple and consistent: each decision sees the result of the previous one. It then moves the slow part into a separate stage that runs in the background. To stop the next decision from using a node that is about to be taken, the scheduler records the choice in its own memory the moment it is made. This trick is called assuming the Pod onto the node. The next scheduling cycle sees the node as already used, even though the official record has not yet been written.

This idea has a long history. Google’s internal cluster managers, Borg and later Omega, explored how to schedule large numbers of tasks with shared state and optimistic updates. Kubernetes borrowed the spirit: make decisions quickly against a local view, then commit them, and be ready to retry if the commit fails.

What this buys you

Eight words appear again and again in scheduler discussions. Learn them now and the rest reads smoothly.

Node and Pod requests

Each node reports how much CPU and memory it can offer, called its allocatable resources. Each Pod states how much it needs in its requests. The scheduler compares requests to what is still unclaimed on every node. It cares about requests, not about what the Pod really uses at the moment. This is why a node can be “full” for scheduling while its real CPU use is low.

Scheduling queue

The waiting line of Pods that still need a node. It is not a plain list. It has three internal parts, explained in chapter 5, that decide who goes next and who must wait and retry later.

Scheduler framework and plugins

Since Kubernetes 1.19 the scheduler has been built as a framework. A framework is a skeleton with defined slots. Each slot is called an extension point. A small piece of logic that fits into a slot is called a plugin. Almost everything the scheduler does, even the built-in rules, is a plugin. This makes it possible to remove, add or reorder behaviour without rebuilding the scheduler from scratch.

Extension point

A named step in the process where plugins are called. Some extension points belong to the scheduling cycle and some to the binding cycle. The next table groups them.

Scheduler cache and snapshot

The scheduler keeps its own live copy of the cluster in memory. This is the cache. At the start of every scheduling cycle it takes a snapshot, a frozen photograph of the cache, so that all plugins in that cycle look at the same picture even if events arrive meanwhile. Reading from memory is why a scheduling cycle is so fast.

Cycle state

A small notebook shared by the plugins during one scheduling attempt. An early plugin can write something down, such as a summary it computed once, and later plugins can read it instead of repeating the work. The notebook is thrown away after the attempt.

Assume, Reserve and Unreserve

To assume a Pod means to count it on the chosen node inside the scheduler’s cache before the official record exists. Reserve is the extension point where plugins, too, claim whatever they need for that Pod. Unreserve is the matching undo step, called if anything later goes wrong. Chapters 8 and 13 explain these in depth.

Scheduling profile

A named configuration that decides which plugins run and with what settings. One scheduler process can serve several profiles. A Pod picks its profile through the field called schedulerName. If the field is not set, the default profile is used.

Before following one Pod through the cycles, meet the parts of the machine that carry it.

The kube-scheduler is a single program that runs on the control plane. It does not store the cluster’s truth. The truth lives in the API server and its database. The scheduler watches the API server for changes, keeps a local copy and sends back decisions. The picture below shows the main parts.

flowchart TB
  API["API Server"] -->|"watch Pods, Nodes, PVCs, more"| INF["Informers"]
  INF --> CACHE["Scheduler Cache"]
  INF --> QUEUE["Scheduling Queue: active, backoff, unschedulable"]
  QUEUE -->|"next Pod"| LOOP["Scheduling Loop"]
  CACHE -->|"snapshot"| LOOP
  LOOP --> SC["Scheduling Cycle: PreFilter to Permit"]
  SC -->|"assume Pod on node"| CACHE
  SC -->|"start in background"| BC["Binding Cycle: WaitOnPermit to PostBind"]
  BC -->|"Binding request"| API
  API --> KL["Kubelet on chosen node"]
  KL --> RUN["Containers start"]
  SC -->|"no fit or error"| QUEUE
  BC -->|"failure: unreserve and forget"| QUEUE
    

Reading the picture

Only one scheduler is active at a time

To avoid two schedulers making conflicting decisions, production clusters usually run several scheduler copies for safety but use leader election so that only one is active. The others wait. If the leader fails, one of them takes over within seconds. Chapter 19 covers this in more detail.

A Pod does not jump straight into a scheduling cycle. First it must get through the door and wait its turn.

The door: PreEnqueue and scheduling gates

The first extension point, PreEnqueue, runs before a Pod even enters the active line. Plugins here answer one question: “should this Pod be considered for scheduling right now?”. If any plugin says no, the Pod is parked in a holding area and is not attempted at all. The most familiar example is scheduling gates. A Pod can be created with one or more named gates in its specification. While any gate remains, the scheduler leaves the Pod alone. An external controller, for example a quota or batch admission system, removes the gates when it decides the Pod may proceed. Scheduling gates became stable in Kubernetes 1.30.

This matters for our topic because a gated Pod consumes no scheduling cycle at all. It does not pollute the failure counts or the logs, and the scheduler does not waste time retrying a Pod that is not ready to be placed.

The three waiting areas

stateDiagram-v2
  [*] --> Gated : Pod has scheduling gates
  Gated --> Active : all gates removed
  [*] --> Active : new Pod created
  Active --> InCycle : popped by the scheduler loop
  InCycle --> Bound : scheduling and binding succeed
  InCycle --> Unschedulable : no node fits
  InCycle --> Backoff : error or binding failure
  Unschedulable --> Backoff : helpful cluster event or long timeout
  Backoff --> Active : backoff time over
  Bound --> [*]
    

Why an unschedulable pool exists

Without it, a Pod that cannot fit anywhere would be tried again and again, hammering the scheduler with identical failures. The pool turns this into an event-driven wait. The Pod rests until the cluster changes in a way that might help. Plugins help the scheduler decide which events matter. For example, a Pod that failed because no node had enough memory is worth retrying when a node is added or when a Pod is deleted, but it gains nothing from an unrelated label change on a different object. Recent releases use hints supplied by plugins to avoid pointless retries. As a safety net, Pods are also moved out of the pool after a long timeout of several minutes, in case an event was missed.

Order of service: QueueSort

The QueueSort extension point decides which waiting Pod is first. Only one QueueSort plugin can be active in a scheduler, because the queue needs one agreed ordering. The built-in plugin orders by Pod priority, highest first, and for equal priority by the time the Pod entered the queue. Priority comes from a PriorityClass that you attach to the Pod. This is how critical system Pods get a node before a batch job does when both are waiting.

The first half of the decision is a process of elimination. Every node starts as a candidate, and bad ones are removed.

Step 0: take a snapshot

The cycle begins by taking a snapshot of the cache. From this moment until the cycle ends, every plugin sees the same fixed picture of nodes and their Pods. This avoids a class of confusing bugs where a plugin early in the cycle and a plugin late in the cycle see different worlds.

Step 1: PreFilter

PreFilter plugins prepare. They look at the Pod once and compute information that later steps will need for every node, writing it into the cycle state. They can also reject the Pod immediately if it can never be placed, for instance when it refers to a claim that does not exist. A PreFilter plugin can even narrow the list of nodes to consider, which can save a lot of work on a big cluster.

Step 2: Filter

Filter plugins answer a yes or no question for each node: “could this Pod run here?”. If any plugin says no, the node is removed from the candidate list. The built-in filters cover the rules most people know:

When a Pod is rejected on all nodes, the familiar message appears in the Pod’s events. It reads like “0 of N nodes are available” followed by a count for each reason, for example how many nodes lacked memory and how many had an untolerated taint. That message is the aggregated output of the Filter step.

Step 3: PostFilter, only when nothing fits

If Filter leaves zero nodes, the PostFilter extension point runs. Its job is to see whether anything can be done to create room. The default plugin here attempts preemption: finding lower priority Pods that could be removed to make space. Chapter 16 covers it fully. If no PostFilter plugin helps, the attempt ends in failure and the Pod goes to the unschedulable pool.

Not every node is examined on huge clusters

Running every filter on every node would be slow on a cluster with thousands of nodes. The scheduler therefore stops searching once it has found enough feasible nodes. The amount is controlled by a setting called percentageOfNodesToScore. If it is not set, the scheduler uses an adaptive percentage that is larger for small clusters and shrinks towards a floor of five percent for very large ones, with a minimum of one hundred feasible nodes. To keep things fair, the search starts at a different node each time and moves around the list, so the same first few nodes are not always favoured.

Once a list of workable nodes exists, the scheduler asks the better question: which one is best?

PreScore and Score

PreScore plugins do the same job for scoring that PreFilter did for filtering. They compute shared information once. Then the Score plugins give every remaining node a number. Each plugin judges from its own point of view:

Normalizing and weighing

Different plugins might produce numbers on different ranges, so each plugin may supply a normalize step that rescales its results into a common range from zero to one hundred. Then each plugin’s score is multiplied by its weight, a number that expresses how much the cluster operator cares about that rule. A few built-in plugins such as taint toleration, node affinity and topology spread carry higher weights than the rest by default. The weighted scores are added up to give every node a total.

Selecting the winner

The node with the highest total wins. If several nodes tie for first place, the scheduler chooses randomly among them. The random pick is deliberate. If it always took the first tied node in a fixed order, the same node would collect all the new Pods, creating a hot spot.

Changing the scoring strategy

Operators can tune the ranking without writing code. A cluster that wants to save money often switches resource scoring to prefer fuller nodes, so that idle nodes can be removed by the autoscaler. A cluster that wants resilience prefers spreading. A cluster running machine learning jobs might add a plugin that scores nodes by the topology of their GPUs. These choices all live in the scheduling cycle, because they are about which node, not about how to commit.

Choosing a node is not the end of the scheduling cycle. Two final steps turn a hope into a held place.

Assume: pencil the name onto the chart

As soon as a winner is selected, the scheduler tells its own cache that the Pod is on that node. It does not wait for the API server. This is the assume step. After it, every later scheduling cycle will see the node’s resources reduced by the new Pod. Without this, the next Pod could be placed on the same node using the same free space, and the node would be overbooked.

Reserve

The Reserve extension point lets plugins that keep their own state claim what they need for this Pod on this node. A storage plugin might note which volume it will use. A device plugin might note which accelerator it will give. Reserve plugins have two methods. Reserve claims. Unreserve releases the claim. The framework promises that if Reserve succeeded for a plugin and something later fails, Unreserve will be called for that plugin, in reverse order, so claims are always cleaned up. If any Reserve step itself fails, the framework calls Unreserve for all plugins and the attempt ends in failure.

Permit: the final gate

The last extension point of the scheduling cycle is Permit. Plugins here can respond in one of three ways:

The crucial detail is where the waiting happens. A Permit plugin only declares the wait during the scheduling cycle. The scheduling cycle then ends, and the actual waiting takes place at the start of the binding cycle, in the step named WaitOnPermit. That is how a Pod can wait for minutes without blocking the scheduler from placing other Pods.

The end of the scheduling cycle

When Permit returns approve or wait, the scheduling cycle is over. The scheduler immediately starts the binding cycle for this Pod in the background and returns to the queue for the next Pod. The elapsed time of the whole scheduling cycle is usually a few milliseconds, because it ran entirely from memory.

flowchart LR
  A["Pop Pod from active queue"] --> B["Snapshot of cache"]
  B --> C["PreFilter"]
  C --> D["Filter nodes"]
  D -->|"no node fits"| E["PostFilter: try preemption"]
  E -->|"cannot help"| X["Unschedulable pool"]
  D -->|"nodes remain"| F["PreScore and Score"]
  F --> G["Select highest scoring node"]
  G --> H["Assume Pod in cache"]
  H --> I["Reserve"]
  I --> J["Permit: approve, deny or wait"]
  J -->|"approve or wait"| K["Start binding cycle in background"]
  J -->|"deny"| U["Unreserve and requeue"]
    

Why these steps belong at the end of the scheduling cycle

It may seem odd that Reserve and Permit sit inside the serial cycle while the later steps are moved out. The reason is that they protect the decision. Assume and Reserve must happen before the next Pod is considered, otherwise the next decision could use the same resources. Permit must happen here because it is the last chance to say no before the scheduler commits time and effort to the binding cycle. In short, everything that must be true before the next Pod is looked at is in the scheduling cycle. Everything that can be completed after the next Pod has been looked at is in the binding cycle.

You can use this sentence as a test for any new idea. If your logic changes what the next Pod should see, it belongs in the scheduling cycle. If it only affects this Pod’s journey to becoming real, it belongs in the binding cycle.

The decision has been made. Now the cluster must be told, and the node must be made ready to receive the Pod.

The binding cycle begins as a separate background task for each Pod. While it runs, the scheduling loop is already working on the next Pod. The binding cycle has four steps, in this order.

Step 1: WaitOnPermit

If a Permit plugin asked the Pod to wait, this is where it actually waits. The Pod’s binding task pauses until every waiting plugin approves, any plugin rejects, or the time limit passes. If everything was approved immediately during the scheduling cycle, this step takes no time at all. Waiting here uses almost no resource, so thousands of Pods can wait in parallel.

Step 2: PreBind

PreBind plugins do any preparation that must be finished before the Pod can be recorded on the node. This is the extension point for slow, real-world work. The best-known user is the volume binding plugin. If the Pod needs a new disk, PreBind asks the storage system to create it and attach the claim, then waits until the volume is ready. If any PreBind plugin fails, the Pod is not bound, and the failure handling in chapter 13 runs.

Step 3: Bind

The Bind extension point actually records the decision. Plugins here are tried in order. A plugin can say “this one is mine and I did it” or “skip, not mine”, in which case the next plugin is tried. The built-in default binder creates a Binding object through the Pod’s binding subresource on the API server. That request sets the node name on the Pod, and this is the moment the decision becomes official. Other components of the cluster can now see it.

Step 4: PostBind

PostBind plugins run after a successful bind. They are informational. They cannot change the outcome. They are useful for cleaning up state kept during the attempt, and for emitting metrics or notes.

Who gets to bind: a chain of binders

More than one Bind plugin can be configured, and they form a chain. The scheduler offers the Pod to the first plugin. If that plugin answers “skip”, the Pod is offered to the next. The first plugin that takes responsibility performs the binding and the chain stops. This design allows special kinds of Pods, for example ones that need a custom resource record to be updated at the same moment as the binding, to be handled by a special binder, while ordinary Pods fall through to the default one. The older extender mechanism could also take over binding by exposing a bind endpoint.

When the commit itself runs into trouble

The binding request is an ordinary call to the API server, so it can meet ordinary problems. The Pod may have been deleted by its owner a moment earlier. The node may have been removed. The API server may be busy and answer with an error or a timeout. In each case the Bind step returns a failure, and the framework applies the same cleanup used everywhere else: Unreserve, forget the assumed Pod, requeue. Because the cleanup is the same, plugin authors do not need a special code path for “bind failed”. They only need to make sure their Unreserve is correct.

What happens after Bind

The binding request is just an update in the cluster’s database through the API server. The kubelet on the chosen node watches for Pods that name it. When it sees the new assignment, it runs its own final checks, such as confirming that the node really has room, then pulls the images, sets up storage and networking, and starts the containers. None of that is part of the scheduler. The scheduler’s responsibility ends when the Pod names its node.

Put the two cycles next to each other and the differences become easy to remember.

This is the cleverest part of the design, and the part most often misunderstood.

Two Pods, A and B, arrive one after the other. The scheduler decides A’s node, then B’s node. While A’s binding is still talking to the API server, B’s scheduling cycle begins. The next diagram shows how the two overlap.

sequenceDiagram
  autonumber
  participant Q as Queue
  participant S as Scheduling Loop
  participant C as Scheduler Cache
  participant B1 as Binding Cycle of Pod A
  participant B2 as Binding Cycle of Pod B
  participant API as API Server
  Q->>S: Hand over Pod A
  S->>C: Snapshot and choose node N1
  S->>C: Assume Pod A on N1
  S-)B1: Start binding in background
  Q->>S: Hand over Pod B
  S->>C: Snapshot already shows N1 reduced
  S->>C: Choose and assume Pod B on N2
  S-)B2: Start binding in background
  B1->>API: PreBind then Bind request for A
  B2->>API: PreBind then Bind request for B
  API-->>C: Watch confirms A is on N1
  API-->>C: Watch confirms B is on N2
    

Why assuming is safe

When the scheduler assumes Pod A on node N1, it is making a promise to itself that it will either complete the binding or take the assumption back. Because the cache is only the scheduler’s private memory, taking it back costs nothing. Meanwhile nobody else in the cluster is affected. The API server has not yet been told anything.

How the assumption becomes real

After the Bind step succeeds, the API server notifies the scheduler’s informer that the Pod now has a node. The cache sees the official record and replaces the assumption with it. For a moment, both exist in agreement. If binding fails, the scheduler explicitly forgets the assumed Pod, which removes it from the node’s tally, and the freed space is available again to the next scheduling cycle.

Optimistic concurrency

This pattern is called optimistic because the scheduler is optimistic that its assumption will turn out to be true. It acts first and corrects later if reality disagrees. The alternative, pessimistic locking, would hold up every decision until the previous one was fully committed. Optimism is faster in the common case, and the cost of being wrong is small because rollback is cheap.

Where two schedulers can still collide

The cache is private to each scheduler process. If you run a second, separate scheduler with a different name, it knows nothing about the first one’s assumptions until the official record is written. Two schedulers can then both choose the last slot on the same node. The final safety net is the kubelet, which checks again before starting the Pod and rejects it if the node truly has no room. The Pod then fails with a resource shortage message and must be replaced. This is why running multiple schedulers over the same nodes is risky, a theme we return to in chapter 17.

Follow a single Pod through every stage, with the exact conversation between components.

Roughly how long does each stage take?

The numbers below are typical orders of magnitude on a healthy cluster, not promises. They help you judge whether a delay you see is normal.

Add these up for an ordinary Pod and the scheduler’s part, the first two rows plus the bind request, is a tiny fraction of the time before the application is running. Most of the waiting a user feels is in image download and application start-up, which no scheduler setting can shorten.

The facts the Pod object shows along the way

A design is only as good as the way it behaves when things go wrong. Here the split into two cycles really pays off.

flowchart TD
  F["A step fails"] --> Q1{"Where did it fail?"}
  Q1 -->|"Filter finds no node"| A1["PostFilter tries preemption, then unschedulable pool"]
  Q1 -->|"Reserve fails"| A2["Unreserve all plugins, then requeue"]
  Q1 -->|"Permit denies or times out"| A3["Unreserve, forget assumed Pod, requeue"]
  Q1 -->|"PreBind fails"| A3
  Q1 -->|"Bind fails"| A3
  A3 --> R["Cache forgets assumed Pod, node space freed"]
  R --> BQ["Backoff queue then active queue"]
    

The golden rule of cleanup

Whenever a step fails after the Pod has been assumed and reserved, the framework does two things. It calls Unreserve on every Reserve plugin, in reverse order, so each can release what it claimed. Then it tells the cache to forget the assumed Pod, which returns the node’s space. After that the Pod is put back into the queue, where it waits through the backoff period before being tried again.

Failure scenarios and what they look like

Storage is the most common reason a Pod’s binding cycle takes noticeable time. Seeing how it works makes the whole idea concrete.

The setting

A PersistentVolumeClaim, or PVC, is a Pod’s request for storage. A PersistentVolume, or PV, is an actual piece of storage. A StorageClass describes how to create new storage on demand. When a StorageClass uses the mode named WaitForFirstConsumer, the cluster deliberately waits to create the disk until a Pod that uses it has been given a node. The reason is practical: a cloud disk usually lives in one zone, and the Pod must run in the same zone. Creating the disk first could trap the Pod in an unsuitable zone.

Where each cycle takes part

sequenceDiagram
  autonumber
  participant S as Scheduling Cycle
  participant V as VolumeBinding Plugin
  participant BC as Binding Cycle
  participant API as API Server
  participant PC as Storage Controller
  S->>V: PreFilter and Filter check claims against each node
  V-->>S: Nodes where storage can work
  S->>V: Reserve the chosen volumes for this node
  S-)BC: Start binding in background
  BC->>V: PreBind
  V->>API: Annotate claim with the selected node
  API-->>PC: Claim now has a target node
  PC->>PC: Create and attach the disk
  PC-->>API: Claim and volume bound
  V-->>BC: Volume ready
  BC->>API: Bind the Pod to the node
    

Reading the story

• Filter removes nodes where the claim could never work, such as nodes in the wrong zone or nodes that cannot attach any more disks.
• Reserve remembers which existing volumes were picked for this Pod so that another Pod in the same moment does not pick them too.
• PreBind triggers the slow part. It tells the storage system “this claim will be used on that node”, then waits until the claim is bound. By default this wait can last up to ten minutes, a limit adjustable in the plugin’s configuration.
• Bind happens only after the volume is ready, so the Pod never starts on a node with missing storage.

Some workloads only make sense when all their Pods start together. The Permit step exists to support them.

The problem

A distributed training job might need sixteen workers. If only fifteen can be placed, the job cannot make progress, yet those fifteen Pods would sit on expensive machines doing nothing. Worse, two such jobs might each grab half of the available machines and both wait forever for the rest. This deadlock is a familiar pain in machine learning and batch computing clusters. The solution is called gang scheduling: place all members of a group, or none.

How Permit solves it

Consider a Permit plugin that knows about groups. When the first worker is chosen a node, the plugin notes that its group needs sixteen members. It answers wait. The worker is assumed on its node and reserved there, and moves to the binding cycle, where it waits. The scheduling loop continues with other workers. Each time another worker is chosen a node, the plugin counts it. When the sixteenth arrives, the plugin approves all sixteen waiting Pods at once and they proceed to bind together.

Why this is only possible with two cycles

If the waiting happened inside the scheduling cycle, the loop would be stuck on the first worker and could never place the rest of the group. Because the waiting lives in the binding cycle, the loop keeps going and the group can assemble.

When no node fits an important Pod, the scheduler can ask: could someone less important make way?

When preemption triggers

Preemption belongs to the PostFilter step of the scheduling cycle, and it triggers only when Filter has ruled out every node. The default plugin examines each node and asks, “if I removed some Pods with lower priority than this one, would the Pod fit here?”. It picks the node where the least harm would be done, preferring to remove the fewest and least important Pods and to respect any disruption budgets where possible.

What actually happens

Priority classes and their limits

A PriorityClass is a cluster-wide object that maps a name to a number. A Pod that names the class receives that number as its priority. Higher numbers go first in the queue and may preempt lower numbers. Two refinements are worth knowing. First, a class can be declared non-preempting, which means its Pods are served early in the queue but never evict anyone. Batch jobs that deserve a good place in line but must not disturb running services often use this. Second, preemption tries to honour PodDisruptionBudgets, rules that limit how many Pods of a service may be unavailable at once. The scheduler treats the budget as a preference. If no other choice exists, it may still evict a Pod and break the budget, so the budget is not an absolute shield.

An important subtlety

Preemption does not place the Pod immediately. It prepares the ground and then returns the Pod to the queue. The freed room is only a hint, not a guarantee, because a different high priority Pod might arrive and take it first, or the nominated node might change in the meantime. This is why an important Pod can show a nominated node and still remain Pending for a short time. Newer releases can also run the slow part of preemption in the background so that the main loop is not delayed, again following the spirit of keeping the scheduling cycle brief.

Once you understand the two cycles, a practical question follows: where should new logic go?

Four ways to extend

A decision guide

Profiles: several behaviours in one scheduler

A single scheduler can offer several named profiles. A cost-focused batch profile might pack Pods tightly, while a latency-sensitive web profile might spread them. Each Pod picks its profile with the scheduler name. All profiles share the same cache and the same queue, so they never fight each other the way separate scheduler processes can. The only restriction is that they must agree on the QueueSort plugin, because there is one queue.

Why a second scheduler is the last resort

Another scheduler process keeps a separate cache. Neither sees the other’s assumed Pods. Unless the two are given completely separate sets of nodes, they will occasionally choose the same last slot on a node. Profiles inside one scheduler remove this danger, and are therefore preferred whenever the goal is just different behaviour for different workloads.

The two-cycle design is itself a performance feature. These are the dials around it.

Where the time goes

A healthy scheduling cycle is dominated by Filter and Score, because they run once per node examined. A healthy binding cycle is dominated by network calls to the API server and, sometimes, by PreBind waits. In upstream performance tests on very large clusters, the scheduler is generally expected to sustain throughput in the order of a hundred Pods per second, though real numbers depend on plugin choices, node counts and API server speed.

Main tuning dials

Burst behaviour

When thousands of Pods arrive at once, such as after a large Job launches or a zone recovers, the scheduling cycle places them serially while many binding cycles run in parallel. The queue drains as fast as the serial part allows. Seeing the active queue shrink steadily while the binding duration stays flat is the sign of a healthy burst. If binding duration climbs, the bottleneck is the API server or storage, not the scheduler’s decision logic.

Capacity planning for the scheduler itself

People plan capacity for their applications but forget the scheduler. Begin with two numbers: the highest rate of new Pods you expect per second, and the number of nodes. Compare the first with the throughput you measured in a test cluster. Leave generous headroom, because bursts after outages are far larger than steady traffic. If a single scheduler cannot keep up even after trimming plugins and lowering the node percentage, the usual answer is not to add a second scheduler on the same nodes. Instead, reduce the work per Pod: remove expensive cross-Pod rules, avoid very large numbers of tiny Pods, or split the estate into several clusters, each with its own control plane.

It also helps to separate two kinds of slowness. If the active queue is long but each attempt is fast, you have a throughput problem and the answer lies in the scheduling cycle. If the queue is short but Pods take long to appear on nodes, you have a binding or kubelet problem. Confusing the two leads teams to tune the wrong half.

Scale limits to keep in mind

• The serial scheduling cycle sets an upper bound on how many decisions per second a single scheduler can make.
• Cluster-wide rules between Pods increase the cost per decision roughly with the number of Pods they must be compared against.
• Very long Permit waits keep many reserved Pods in memory. Keep gang sizes and timeouts reasonable.

Leader election

Production control planes usually run at least two or three scheduler copies. They compete for a small shared record called a lease. The holder is the leader and is the only one that schedules. By default the leader renews its lease every couple of seconds and a standby takes over if renewals stop for around fifteen seconds. During a takeover, no new scheduling decisions are made, but running Pods are not affected, and Pods that were already bound stay bound. Pending Pods simply wait a little longer.

What happens to work in flight when the leader dies

This is a strong reliability property that comes from the design: because nothing is considered real until the Bind step writes it into the API server, a crash can never leave a half-made decision. The worst outcome is that a decision is made twice, not that a Pod is lost.

Reliability habits

• Run several scheduler replicas across failure zones.
• Set resource requests for the scheduler itself so it is not starved on a busy control plane node.
• Alert when there is no leader, when the active queue grows steadily or when error results rise.
• Keep the API server healthy, because both cycles depend on it, directly in binding and indirectly through watches.

Whoever controls placement controls where your workloads and, indirectly, your data run.

The permission that matters most

The step that makes a placement official is a request to create a Binding through the Pod’s binding subresource. Only the scheduler and a few trusted components should hold this right. A person or program that can create bindings can place any Pod on any node, including onto nodes meant for sensitive workloads. Treat that permission with the same care as cluster-wide write access.

Skipping the scheduler

A Pod can name its node directly in its specification. Such a Pod skips both cycles entirely. None of the filters, taints or affinity rules are checked by the scheduler. The kubelet still refuses a Pod it cannot host, but placement rules you rely on for isolation could be bypassed. Use admission policies to restrict who may set the node name directly, especially in shared clusters.

Custom schedulers and plugins

Placement as a security tool

The same machinery can help security. Taints and tolerations keep regulated workloads on dedicated nodes. Node affinity keeps data-sensitive Pods in approved regions. Topology spread keeps replicas in separate zones so one failure cannot take down a whole service. All of these rules are enforced in the Filter step of the scheduling cycle.

Metrics worth watching

A calm path from symptom to cause

flowchart TD
  A["Pod stays Pending"] --> B{"Does the Pod have a node name?"}
  B -->|"No"| C{"Is there a FailedScheduling event?"}
  C -->|"Yes"| D["Read the reasons: resources, taints, affinity, volumes"]
  C -->|"No"| E{"Is the Pod gated or using another scheduler name?"}
  E -->|"Yes"| F["Remove the gate or fix the scheduler name"]
  E -->|"No"| G["Binding cycle may be waiting: check claim events and group status"]
  G --> H["Check that the scheduler is running and holds the lease"]
  B -->|"Yes"| I["Scheduling is finished, look at the kubelet"]
  I --> J["Check image pull, volume mounts and the runtime"]
    

Reading common messages

Setting service level goals for placement

A service level goal turns vague wishes into numbers you can alert on. A useful pair for the scheduler is one goal per cycle. For the scheduling cycle, state that most attempts finish in a small number of milliseconds, measured at a high percentile. For the whole journey, state that most Pods are bound within a few seconds of being created when capacity exists, and measure this with the time from queue entry to binding. Exclude Pods that are gated or that are truly unschedulable because of their own rules, otherwise a single misconfigured workload can make the whole platform look unhealthy.

Keep the two goals separate. When the second one is breached while the first is healthy, the problem is almost certainly in the binding cycle, storage or the API server. When both are breached, look at plugin cost, queue length and node count. This simple split saves hours of guessing in a night-time incident.

Logs

At normal verbosity the scheduler logs only errors and key outcomes. Raising verbosity for a short time during an investigation reveals each plugin’s decision for a Pod. Return to normal afterwards, because verbose logging in a busy cluster produces enormous volume and slows the scheduler itself.

Patterns that work

Anti-patterns to avoid

Let us run two Pods through a tiny cluster and watch every cycle do its job.

The cluster

Pod A arrives, asking for 2 CPU

Pod B arrives a moment later, also asking for 2 CPU

The API server has not yet recorded Pod A’s binding, because that request is still on its way. If the scheduler looked only at the official record, Node 2 would still look like it has 3 CPU free and Pod B would be sent there too, overbooking the node. Instead, the scheduler uses its cache, which already includes Pod A.

• Node 1 has 1 CPU free and fails.
• Node 2 now has 1 CPU free in the cache and fails.
• Node 3 fails on the taint.

Pod B is correctly reported as unschedulable for now and parked in the unschedulable pool with a clear message. This is the assume step protecting the cluster from a double booking.

Then two things can happen

Best practices

Common mistakes

• Thinking Pending always means the scheduler is slow. Most Pending Pods are simply unschedulable or waiting in the binding cycle.
• Blaming the scheduler for slow starts after binding. Image pulls and volume mounts belong to the kubelet.
• Confusing requests with usage. A “full” node can be nearly idle.
• Using a second scheduler for a small tweak. A profile in the main scheduler does the same without cache conflicts.
• Setting extremely long Permit timeouts. Reserved capacity stays locked while the group waits.
• Ignoring the unschedulable queue growth. It is an early warning of a capacity shortage.
• Forgetting that preemption takes time. Victims need a graceful shutdown, so the important Pod is not instantly running.

The two-cycle pattern shows up across the industry, in managed platforms and in specialised projects.

No. Scheduling cycles run one after another in a single scheduler. Only binding cycles overlap. This is what keeps decisions consistent.

Not officially. The choice exists only in the scheduler’s memory until the Bind step writes it to the API server. The Pod is formally scheduled once it names its node.

Several reasons are possible. It may be gated, it may name a scheduler that is not running, or its scheduling cycle may have succeeded and the binding cycle is waiting on a volume or a group. Check the claim events and the Pod’s scheduler name.

Not directly, because binding runs in the background. It can slow them indirectly by holding capacity in reserve and by loading the API server.

Reserve claims resources for the Pod on the chosen node and promises an undo. Permit decides whether the Pod may continue, must wait or must stop. Reserve is about holding, Permit is about permission.

At the start of the binding cycle, in WaitOnPermit. The Permit plugin only declares the wait during the scheduling cycle.

That cannot happen within a single scheduler, because the second decision sees the first Pod already counted. It can happen across two separate schedulers, in which case the kubelet rejects the extra Pod.

Often not. Weights and strategies can be changed in configuration, for example to favour packing instead of spreading.

Because its purpose is to repair a failed Filter step, mainly by finding Pods that could be preempted. If any node passes Filter, there is nothing to repair.

By default it looks at resource requests, not live usage. Separate tools can feed usage into placement, but the standard decision is based on requests.

The new leader rebuilds its cache from the API server. Pods whose binding reached the API server stay bound. Others are scheduled again. Nothing is lost.

If the logic is about when a workload may start, rather than where it goes, a controller using scheduling gates is simpler and keeps the scheduler fast.