If you only remember one paragraph, make it the next one.

The kube-scheduler is a control plane program that watches for newly created pods that have no machine assigned, and picks the best machine, called a node, for each one. A pod is the smallest unit Kubernetes runs, a small wrapper around one or more containers. A container is a sealed package holding an application and everything it needs. The scheduler makes a decision and records it. It does not start the containers. Another program on the chosen node, the kubelet, does that.

What the scheduler does in three steps

How the scheduler differs from its neighbours

Beginners often blur the scheduler together with other Kubernetes parts, so it helps to draw the borders clearly. The controller manager creates and deletes pods to match the count you want, but it does not choose their location. The scheduler chooses the location, but does not create or delete anything. The kubelet starts and stops the containers on its own node, but has no say about which pods it receives. Finally, the API server stores and shares every decision, but makes none of these choices itself.

Because each part does exactly one job, you can reason about problems easily. If pods are missing, look at the controllers. If pods exist but have no node, look at the scheduler. If pods have a node but the containers never start, look at the kubelet and the image. This simple habit of asking “which part owns this step?” solves a surprising share of real debugging sessions.

To appreciate the scheduler, we first need to see its neighbours and the problem it faces.

A quick look at the cluster

A Kubernetes cluster is a group of computers working as one. The computers that make decisions form the control plane. The computers that run your applications are worker nodes. The control plane holds an API server, which is the single front door, a database called etcd, which stores all records, controller programs that keep things tidy, and our scheduler. Each worker node has a kubelet, an agent that starts containers on instructions.

flowchart TB
    U["Engineer or Deployment controller creates a Pod"] --> API["kube-apiserver"]
    API  ETCD["etcd database"]
    SCH["kube-scheduler"] -->|"1. Watch for Pods with no node"| API
    SCH -->|"2. Filter and score nodes"| SCH
    SCH -->|"3. Bind Pod to chosen node"| API
    API -->|"4. Pod now assigned"| KL1["kubelet on Node 1"]
    KL1 --> RT["Container runtime starts containers"]
    KL2["kubelet on Node 2"]
    KL3["kubelet on Node 3"]
    API --- KL2
    API --- KL3

Why is this hard?

Placing one pod on one node sounds easy. But real clusters have hundreds or thousands of nodes of different sizes, and pods arrive constantly with different needs. Each pod may want a certain amount of processor time, a certain amount of memory, a special kind of machine, or a place close to other pods, or far from them. The scheduler must respect all these wishes at once, and decide in a fraction of a second, because many pods are waiting.

What a good placement achieves

How the scheduler knows what is happening

The scheduler does not visit each node to look around. It learns about the cluster the same way other control plane parts do, by watching records in the API server. It keeps a private, always-updating copy of every node, every pod, and every relevant storage object in its memory, often called its cache. When a node joins, leaves, or changes a label, or when a pod is created or deleted, the news arrives through a watch, and the cache is updated within moments. Decisions are made against this cache, which is why they are fast.

The cache can be slightly behind reality, for example just after a node has failed. That is acceptable, because the final binding is checked by the API server, and errors are retried. If the chosen node vanished in the meantime, the binding fails, the assumed reservation is removed, and the pod returns to the queue to be decided again.

Behind each placement is a tidy sequence of steps. Learning them makes every later topic easy.

The waiting line: scheduling queues

New pods do not go straight to being placed. They join a waiting line called the scheduling queue, which actually has three parts. The active queue holds pods ready to be tried now, sorted by priority so that important pods go first. The backoff queue holds pods that just failed and must wait a short time, which grows with each failure, before trying again. The unschedulable pool holds pods that cannot fit anywhere right now and are waiting for the cluster to change, such as a new node arriving or another pod leaving.

The two cycles

For each pod taken from the active queue, the scheduler runs two cycles. The scheduling cycle chooses the node and always handles one pod at a time, in order. The binding cycle writes the decision to the API server and can run in the background for many pods at once, because talking to the API server takes time and the scheduler does not want to sit idle. To keep going without waiting, the scheduler first pretends the pod is already on the node inside its own memory. This trick is called assuming the pod, and it stops the next decision from using space that was just promised.

flowchart LR
    Q["Scheduling queue: active, backoff, unschedulable"] --> PF["PreFilter: prepare data"]
    PF --> F["Filter: remove nodes that cannot run the Pod"]
    F --> OK{"Any node left?"}
    OK -->|"No"| PO["PostFilter: try preemption"]
    PO -->|"Failed"| UQ["Back to queue and wait"]
    OK -->|"Yes"| SC["Score: rate remaining nodes 0 to 100"]
    SC --> PICK["Pick highest total score"]
    PICK --> RES["Reserve: assume the Pod is there"]
    RES --> PER["Permit: last approval"]
    PER --> BIND["Bind: write nodeName via API server"]

Extension points: the plug-in sockets

Modern Kubernetes builds the scheduler out of small plug-ins. The steps in the diagram are called extension points, like sockets into which plug-ins connect. Filtering plug-ins remove nodes. Scoring plug-ins award points. Others prepare data, reserve space, or wait for approval. This design is called the scheduling framework, and it lets the community and companies add new behaviour without rewriting the scheduler.

Filtering asks a strict yes-or-no question about every node: can this pod possibly run here?

The scheduler checks all nodes, or a sample of them in very large clusters, against a list of filter rules. A node that fails any single rule is thrown out at once. The nodes that pass every rule are called feasible nodes. If no feasible node exists, the pod cannot be placed now.

The common filters

A filtering example with real numbers

Suppose a pod requests half a processor core and 1 gigabyte of memory. Node one has 4 cores and 8 gigabytes allocatable, and its running pods already request 3.8 cores and 5 gigabytes. Only 0.2 cores remain, so node one fails the resource filter even though it has plenty of memory. Node two has 2 cores and 4 gigabytes allocatable, with 1 core and 2 gigabytes already requested. It has 1 core and 2 gigabytes left, so it passes. Node three is healthy and roomy, but carries a taint the pod does not tolerate, so it fails the taint filter. Only node two remains feasible.

Notice how many different reasons can remove a node. A node is not rejected because it is “bad”. It is rejected because it does not match one particular need of this particular pod. The same node might be a perfect choice for the next pod in line.

Large clusters: not every node is checked

Checking every node for every pod would waste time in a huge cluster. So the scheduler stops looking once it has found enough feasible nodes. The share it looks for is controlled by a setting named percentageOfNodesToScore. If you do not set it, Kubernetes chooses an adaptive number: about half the nodes in a small cluster, shrinking as the cluster grows, with a floor of 5 percent, and always at least 100 feasible nodes. In a 5,000-node cluster this works out to roughly 10 percent. The scheduler also walks through the node list in rotation, so different pods start the search at different places and the load stays fair.

After filtering, several nodes may all be acceptable. Scoring decides which one is best.

Each scoring rule looks at every feasible node and gives it a number from 0 to 100. The scheduler multiplies each rule’s number by a weight, which says how much that rule matters, adds the results, and picks the node with the highest total. If two nodes tie, one is chosen at random so that no node is treated unfairly.

The most important scoring rules

A small worked example

Suppose three nodes survived filtering. Node A is empty and has the image already downloaded. Node B is half full. Node C is nearly full but is in a zone with few copies of the app. With the default rules, Node A gets high marks for free room and image locality. Node C gets high marks for spreading but low marks for free room. Node B is average on everything. Most of the time, Node A wins, unless the topology spread rule has a heavy weight or a hard requirement, in which case Node C could win. The point is that the answer depends on weights, and an engineer can read the rules and predict the outcome.

The scheduler is a careful accountant, and it decides using the numbers you write down, not the numbers your program actually uses.

Requests and limits

Every container can declare two amounts for processor time and memory. A request is the amount the container needs to run properly, a guaranteed reservation. A limit is the ceiling it may not cross. The scheduler looks only at the requests. It adds up the requests of all pods already on a node and checks whether the new pod’s requests still fit in what remains. Limits matter later, when the node itself enforces them.

Allocatable: the space that can really be booked

A node does not offer all of its capacity to pods. Some is kept for the operating system, for the kubelet, and for a safety margin. What remains is called allocatable. The scheduler subtracts the total requests of running pods from the allocatable amount to see what is still free. This is why a node with 16 gigabytes of memory may show only about 14 gigabytes available for pods.

Quality of service classes

Kubernetes sorts pods into three classes, based on how requests and limits are written. The class decides who is removed first when a node runs short of memory.

Pods with several containers

Many pods hold more than one container, and some hold special short-lived init containers that run one after another before the main containers start, for example to prepare files. The scheduler adds up the requests of all the main containers, then compares that sum with the largest single init container, because init containers do not run at the same time as the main ones. It uses the bigger of the two numbers as the pod’s total need. Some clusters also add a fixed pod overhead, an extra allowance for the sandbox around the containers, especially when using stronger isolation runtimes. Understanding this arithmetic explains why a pod with a tiny main container can still refuse to fit on a small node.

You are not stuck with the default behaviour. Kubernetes gives you several levers to guide where pods land.

Before the levers, one small word: a label is a tag such as “disk equals fast” or “zone equals A” that you attach to nodes and pods. Most steering works by matching labels. Kubernetes also puts standard labels on nodes automatically, such as the zone and the machine type.

Required versus preferred

Many rules come in two strengths. A required rule is a hard wall: if no node satisfies it, the pod stays Pending. A preferred rule is a soft wish: it adds points during scoring, but the pod can still land elsewhere if needed. Rule names contain the long phrase “IgnoredDuringExecution”, which simply means the rule is checked only at the moment of scheduling. If node labels change later, Kubernetes does not move a pod that is already running.

Taints in more detail

A taint has an effect. NoSchedule means new pods without permission are kept out, but pods already there may stay. PreferNoSchedule means the scheduler tries to avoid the node but may still use it. NoExecute is the strongest: pods without permission are pushed off even if they are already running. Kubernetes itself adds NoExecute taints to nodes that become unreachable, which is how failed nodes are emptied automatically. Control plane nodes carry a NoSchedule taint by default, so ordinary applications stay away from the brain.

Spreading with topology constraints

A topology spread constraint names a boundary, such as a zone, and a number called maxSkew, meaning the biggest allowed difference in pod counts between the fullest and emptiest zone. With maxSkew of one and three zones, ten pods would be placed as four, three, and three. You also choose whether a broken constraint blocks the pod or is only a soft wish. This is the recommended modern way to spread copies of an application.

What happens when the cluster is full and an important pod arrives? Kubernetes has a polite but firm answer.

Priority classes

A PriorityClass is an object that gives a name to a whole number. A pod that refers to that class receives the number as its priority. Higher numbers are more important. Two effects follow. First, in the active queue, high-priority pods are tried before low-priority ones. Second, if a high-priority pod cannot fit anywhere, the scheduler may make room for it. Values above one billion are reserved for system components, and two built-in classes, one for critical cluster services and one for critical node services, use the top of that range.

Preemption: making room

When a pod fits nowhere, the filtering phase ends with an empty list. The scheduler then reaches the PostFilter step, where a plug-in named DefaultPreemption looks for a node where removing some lower-priority pods would let the new pod fit. It picks the option that harms the fewest and least important pods. The chosen victims are asked to shut down gracefully, and the new pod gets a note called the nominated node, which reserves its place while it waits for the space to appear.

What waits in the queue and how it wakes up

A pod that cannot be placed is not forgotten. It waits in the unschedulable pool. When something in the cluster changes in a way that could help, such as a new node joining, a pod being deleted, or a node label changing, the scheduler moves the affected pods back to the active queue to try again. Even without such events, pods are retried after a maximum wait of a few minutes. Failed attempts also use the backoff queue so the scheduler is not flooded by the same hopeless pod again and again.

Preemption and disruption budgets

Preemption tries to be gentle. A PodDisruptionBudget is a promise that says “at least this many copies of my app must stay running during voluntary disruptions”. The scheduler tries to respect these promises when choosing victims, but it treats them as a preference rather than an unbreakable law, because an urgent high-priority pod may still need the space. It also gives victims their full graceful shutdown time, which is thirty seconds by default, so applications can finish current requests before leaving.

Another helpful detail is the nominated node. While victims are shutting down, the waiting pod keeps a note reserving that node. Other lower-priority pods that arrive in the meantime see the reservation and stay away, so the freed space is not snatched by someone else in the gap. If a better node appears before the victims finish leaving, the scheduler is free to change its mind.

The default scheduler suits most clusters, but Kubernetes lets you reshape it and even replace it.

Profiles: one scheduler, many personalities

A single scheduler program can run several profiles at once. A profile is a named bundle of settings that says which plug-ins are switched on, which are off, and what weight each scoring plug-in carries. Every pod can pick a profile by writing its name in the pod description under a field called schedulerName. If nothing is written, the pod uses the profile named default-scheduler. In this way, one team can run batch jobs that pack tightly while another runs web servers that spread widely, all inside the same cluster.

Custom and second schedulers

If plug-in settings are not enough, you can write your own scheduler or plug-in. Because scheduling is just “watch for pods with no node, then write a binding”, a second scheduler can run beside the first one. Pods choose between them by name. Companies with special workloads, such as machine learning training or scientific computing, sometimes do this.

Why the descheduler exists

The scheduler makes its choice once, at the moment a pod is created, using the information available then. Later, conditions change: new nodes join, other pods leave, labels are edited. A perfect choice earlier may look poor now. Since the scheduler never moves running pods, the descheduler fills that gap by removing pods that no longer fit the ideal picture. The controller that owns the pod then creates a replacement, and the scheduler decides again.

Let us follow one pod from birth to its first heartbeat, and then learn to read what the scheduler says when things go wrong.

sequenceDiagram
    participant D as Deployment controller
    participant A as kube-apiserver
    participant E as etcd
    participant S as kube-scheduler
    participant K as kubelet on chosen node
    D->>A: Create Pod record with no node
    A->>E: Save Pod as Pending
    S->>A: Watch shows new Pod without a node
    S->>S: Queue it, filter nodes, score nodes
    S->>S: Reserve space in local cache
    S->>A: Create Binding to chosen node
    A->>E: Save nodeName on the Pod
    K->>A: Watch shows Pod assigned to my node
    K->>K: Pull image and start containers
    K->>A: Report status Running

The life of a pod’s placement

Reading the scheduler’s messages

When a pod stays Pending, the scheduler leaves a short sentence in the pod’s events explaining why. A typical message says that zero of some number of nodes are available, followed by a count for each reason. Learning these reasons turns confusion into a quick fix.

A calm checklist for a stuck pod

Cordon and drain: telling the scheduler a node is leaving

Nodes need maintenance such as operating system updates. Before touching a node, operators first cordon it, which marks the node as unschedulable so that the scheduler sends no new pods there. Existing pods keep running. Next they drain the node, which politely evicts its pods one by one while respecting disruption budgets. The controllers notice the missing pods, create replacements, and the scheduler places them on other nodes. When maintenance is done, the operator uncordons the node, and it becomes available again. This routine shows the scheduler doing its ordinary job in a planned event, without any special code.

A scheduler that is slow or that stops working can freeze a whole cluster’s ability to grow. Here is how it stays fast and reliable.

Speed

Kubernetes officially supports clusters of up to 5,000 nodes and 150,000 pods, and the scheduler is designed to keep up at that size. The number of pods created at one moment matters as much as the cluster size. A sudden burst, such as a batch system launching ten thousand pods together, can form a long queue, so large batch users often add specialised queueing tools.

High availability: many copies, one leader

Production clusters run several copies of the scheduler, usually one on each control plane node. But two schedulers deciding at once could bind the same pod twice or double-book a node. So the copies hold an election through a shared record called a Lease. Only the elected leader makes decisions. The others stay ready, and if the leader stops renewing its Lease, another copy takes over within a few seconds.

The scheduler holds power over where workloads run, so it must be protected and watched.

Security

The scheduler speaks to the API server using a special identity, and role-based access control limits it to exactly what it needs: reading pods, nodes, and storage information, and creating bindings. It should never have broader rights. All its connections use encrypted transport. Placement rules are also a security tool. Taints and node labels can keep sensitive workloads on dedicated machines, apart from untrusted ones, and anti-affinity can prevent a risky pod from sharing a node with a critical one.

Monitoring and metrics

The scheduler publishes a metrics page, which Prometheus, the popular monitoring tool, collects, and Grafana turns into dashboards. Its logs go to standard output. Important signals include the number of pods waiting in each queue, the time each pod takes to be placed, the count of successful and failed attempts, and the number of preemptions.

Using logs when events are not enough

Events give a short summary, but the scheduler’s own logs can show more detail when you turn up the verbosity. They record which plug-in rejected which node, how scores were added up, and when a pod moved between queues. This level of detail is noisy, so most teams turn it on only for a short investigation. A useful habit is to keep one test pod with a clear, deliberately impossible request, and use it to confirm that your alerts and dashboards react properly when the scheduler cannot place something.

flowchart LR
    SCH["kube-scheduler metrics and logs"] --> PR["Prometheus"]
    SCH --> LA["Log agent"]
    EV["Pod events from the API server"] --> LA
    PR --> GF["Grafana dashboards"]
    PR --> AM["Alertmanager"]
    AM --> OC["On-call engineer"]
    LA --> LS["Central log store"]
    LS --> INV["Investigate why a Pod was Pending"]
    GF --> INV

A simple alert can save you from a silent outage: warn when the number of Pending pods stays above zero for several minutes. Add a second alert for the time-to-placement, and you will spot both shortage and stuck scheduling early.

Teams have found a few placement shapes that work again and again, and a few tempting shortcuts that cause trouble.

Patterns that work well

Anti-patterns to avoid

These habits come from clusters that ran hot, ran empty, or ran into walls. Copying them saves months of trial and error.

Common mistakes and fixes

Scheduling is not just a Kubernetes idea. It is a problem every large computer operator has had to solve.

Short, plain answers to the questions beginners ask most.

Back to the hotel front desk for one last look.

The Kubernetes scheduler is the front desk of the cluster. It waits for pods without a home, removes every node that cannot host them, awards points to the rest, and writes the winner’s name on the pod. It follows the rules and numbers you give it, so honest resource requests, clear labels, sensible taints, and well-chosen priorities are what make it wise. Understand its two-step method and its handful of levers, and you can predict where any pod will land and fix any pod that will not.