Before writing anything custom, we need to understand what the scheduler does for you every second, and what it quietly leaves unsolved.

A quick picture of Kubernetes

A cluster is a group of computers that work together. Each computer in the group is called a node. Your application runs inside Pods, and a Pod is the smallest unit Kubernetes manages: one or more containers that always travel together, like two people sharing a taxi. When you ask for ten copies of a web server, Kubernetes creates ten Pods. But creating a Pod only writes a record saying “this should exist.” Someone still has to decide which node will actually run it.

That someone is the scheduler. It is a program that runs in the cluster’s control plane, the part of Kubernetes that makes decisions. It watches for Pods that have no node yet, picks the best node for each, and records the choice. After that, a small program on the chosen node, called the kubelet, starts the containers. The scheduler never starts anything itself. It only chooses.

Why the choice is harder than it looks

Picking a node sounds like picking a seat in a cinema, but there are many things to weigh at once. Does the node have enough free memory and processing power? Does it have the special hardware the Pod needs, such as a graphics card? Is it in the right data centre zone? Does the Pod want to sit next to its database, or far away from its twin so that one failure does not kill both? Do some nodes carry a “keep out” sign that this Pod has not been invited past?

A bad choice is expensive. Packing too many busy Pods onto one node makes them all slow. Spreading Pods too thinly leaves expensive machines nearly empty and your cloud bill high. Putting both copies of a service in one zone turns a small outage into a big one. Companies that run thousands of nodes feel these costs in real money, which is why scheduling is a topic that attracts so much engineering attention.

Where the idea came from

Kubernetes was open-sourced by Google in 2014, shaped by lessons from internal cluster managers the company had run for years. From the start, the design treated the scheduler as just another program that talks to the API server, not as a hidden part of it. That choice is what makes customization possible at all. Anyone can write a program that watches Pods and assigns them to nodes, and Kubernetes will happily accept the result.

In the early years, customization meant either running a second scheduler or using a feature called scheduler policies, where you listed rules in a file. Policies were limited, so the project introduced the scheduling framework, a plugin system with clearly defined hook points. Policies were then removed, and today the default scheduler is itself built entirely out of plugins. That is good news for you: changing its behaviour is mostly a matter of choosing and configuring plugins, and writing a new one when needed.

A short glossary now saves confusion later. Each word comes with a plain-life picture.

What “scheduled” really means

A Pod is considered scheduled once its nodeName field is filled in. Before that, the Pod sits in the Pending state with an empty node name. The scheduler’s one job is to turn that empty field into a real node name, by creating a small record called a Binding through the API server. Notice how modest this is. The scheduler does not move containers, pull images or open ports. Every other step is done by other components.

This tiny job description is why replacing the scheduler is realistic. Any program that can read Pods and nodes and write a Binding can play the role. The hard part is not the plumbing; it is making good decisions quickly at scale.

The schedulerName field

Every Pod has a field called schedulerName. If you leave it empty, Kubernetes fills in the value “default-scheduler”. Each scheduler program only handles Pods whose scheduler name matches its own name. This one simple field is how several schedulers can live in the same cluster without fighting: each one ignores Pods meant for the others.

To change something wisely, you need to know its moving parts. Here is the layout of the scheduler you are about to customize.

flowchart LR
  API["API server"] -->|watch pods and nodes| INF["Informers and local cache"]
  INF --> Q["Scheduling queue: active, backoff, unschedulable"]
  Q --> SC["Scheduling cycle: filter and score"]
  SC --> BC["Binding cycle: bind"]
  BC -->|create binding| API
  API --> KL["Kubelet on the chosen node"]
    

Read the diagram from left to right. The scheduler does not ask the API server for data every time it needs it, because that would be far too slow. Instead it keeps a local copy of all nodes and Pods in memory. The copy is kept fresh by informers, which are small helpers that open a long-lived connection to the API server and receive a message whenever anything changes. This design is what lets one scheduler make decisions in a fraction of a second even in a cluster with thousands of nodes.

The three parts of the queue

The queue is smarter than a simple line. The active queue holds Pods that are ready to be tried now, ordered by importance, so a high-priority Pod goes first. When a Pod fails to find a home, it does not hammer the scheduler by retrying instantly. It moves to the unschedulable pool, a parking area for Pods that cannot fit anywhere at the moment. A Pod waits there until something changes that could help it, such as a new node joining, or until a timer expires. Then it moves to the backoff queue, where it waits a short, growing time before it is tried again, much like a person redialling a busy phone number with longer pauses each time.

Everything is a plugin

The default scheduler has no hard-coded rules. Even “does this node have enough memory?” is a plugin, named NodeResourcesFit. The preference for spreading Pods across zones is another plugin, PodTopologySpread. Support for affinity rules, taints, volume limits and image locality are all plugins too. The scheduler core only orchestrates the order in which plugins run and combines their answers. Because of this design, you can switch plugins off, change their weights, or add your own, without touching the core.

Follow a single Pod from the moment it is created to the moment a node is chosen, and see where each plugin joins in.

flowchart TD
  A["QueueSort: order the waiting Pods"] --> B["PreFilter: prepare data once"]
  B --> C["Filter: drop nodes that cannot fit"]
  C -->|no node fits| D["PostFilter: try preemption"]
  C -->|some nodes fit| E["PreScore: prepare scoring data"]
  E --> F["Score: rate each node"]
  F --> G["NormalizeScore: scale to 0 to 100"]
  G --> H["Reserve: hold the resources"]
  H --> I["Permit: approve, deny or wait"]
  I --> J["PreBind: prepare, for example volumes"]
  J --> K["Bind: write the binding"]
  K --> L["PostBind: clean up and notify"]
    

The diagram shows the extension points, which are the named moments where plugins can join in. The first group (QueueSort through Reserve) belongs to the scheduling cycle; the last group (Permit through PostBind) belongs to the binding cycle. We walk through them below.

Two ideas that make it fast

First, optimistic assumption. The scheduler does not wait for the Binding to be confirmed before moving on to the next Pod. It assumes success and updates its own cache at once. If binding later fails, the assumption is undone and the Pod returns to the queue. This keeps the pipeline full instead of idling on network round trips.

Second, partial scoring. In a big cluster, scoring every node for every Pod would be wasteful. A setting named percentageOfNodesToScore lets the scheduler stop looking once it has found enough feasible nodes. The default is adaptive: smaller clusters check nearly everything, while very large ones check only a fraction, with a sensible minimum so choices stay good. The trade-off is clear: a slightly less perfect choice for much higher speed.

The cheapest scheduler to maintain is the one you did not write. Kubernetes already gives every Pod a rich set of controls.

Topology spread: the underrated tool

Many teams reach for a custom scheduler when what they need is a topology spread constraint. It lets a Pod say, “spread my replicas across zones so that no zone has more than one extra copy compared with any other.” There are three pieces: the maximum skew (how uneven it may get), the topology key (what to spread across, such as zone), and what to do if the rule cannot be met (refuse, or do the best possible). This single feature replaces a surprising number of custom-scheduler ideas.

Scheduling gates: a pause button

A scheduling gate is a marker on a Pod that tells the scheduler “do not look at this one yet.” The Pod stays Pending until some controller removes the gate. This is wonderful for workflows such as “wait until quota is approved” or “wait until the data has been copied to the right region.” It gives you a clean place to add decision logic before scheduling, without owning a whole scheduler.

The costs people forget

A custom scheduler is software you must run, upgrade, monitor and debug forever. Each Kubernetes release changes scheduler internals a little, so out-of-tree plugins must be rebuilt and retested. Your engineers must learn the scheduling framework. And every new scheduling rule makes placement behaviour harder to predict for the application teams who just want their Pods to start. A mature team treats a custom scheduler as a last resort and a long-term commitment.

When the built-in knobs are not enough, you have four options, ordered here from the lightest touch to the heaviest.

flowchart TD
  S["Need different placement behaviour"] --> Q1{"Can built in knobs do it?"}
  Q1 -->|yes| K["Use affinity, taints, spread, priority"]
  Q1 -->|no| Q2{"Only different weights or plugin mix?"}
  Q2 -->|yes| P["Scheduler profile via configuration"]
  Q2 -->|no| Q3{"Need new logic inside the normal cycle?"}
  Q3 -->|yes| Pl["Custom plugin in a built scheduler"]
  Q3 -->|no| Q4{"Needs outside data or another language?"}
  Q4 -->|yes| E["Extender"]
  Q4 -->|no| F["Full second scheduler"]
    

Read the diagram from the top. It always pushes you toward the lighter option first. If the problem can be solved with a Pod’s own settings, stop there. If you only need to prefer packing over spreading, a profile is enough. If you need a new kind of rule that fits into filtering or scoring, a plugin is the right tool. If the rule needs information from an outside system, an extender might do. And if you have a genuinely different scheduling model, such as a queue-based batch system, a separate scheduler is the honest answer.

The lightest and safest way to change scheduling behaviour is to leave the code alone and change the settings.

The configuration file

The scheduler reads a file at startup, passed with a command-line flag, written in a format called KubeSchedulerConfiguration. This file has a stable version (kubescheduler.config.k8s.io/v1), which means the format does not change under you from release to release. Inside it you find global settings, such as how many nodes to score, how long a Pod waits between retries, and how leader election works, followed by a list of profiles.

A profile is a named personality for the scheduler. Each profile has a schedulerName, a list of plugins to turn on or off at each extension point, and optional settings for individual plugins. One running scheduler process can hold several profiles at once. A Pod picks its personality simply by naming it in its own schedulerName field.

What you can change

Spreading versus packing: the classic switch

The most useful single setting is how the NodeResourcesFit plugin scores nodes. It offers three strategies. LeastAllocated, the default, prefers nodes with the most free room. This spreads Pods out, which is gentle on each node and good for web services. MostAllocated does the opposite: it prefers the fullest node that still fits, which packs Pods tightly. A third strategy, RequestedToCapacityRatio, lets you draw your own curve of how much each fill level is worth.

Why would anyone want packing? Because cloud nodes cost money whether they are busy or not. If Pods are packed onto fewer nodes, the other nodes become empty, and an autoscaler can remove empty nodes entirely. This is exactly the saving that batch and data-processing teams chase. A common design is two profiles in one scheduler: the default profile spreads web services for resilience, and a second profile named, say, “batch-binpack” packs jobs tightly for cost.

One rule about multiple profiles

All profiles inside one scheduler share a single waiting line. Because of that, they must all use the same QueueSort plugin with the same settings. You can vary almost everything else per profile, but not the ordering of the queue. This surprises people the first time a configuration is rejected for it.

When existing plugins cannot express your rule, you add a new plugin. This gives you full power while keeping the proven machinery of the default scheduler.

What a plugin is

A plugin is a small piece of Go code that has a name and implements one or more of the framework’s interfaces, one per extension point. A plugin that implements the Filter interface is asked, “can this Pod run on this node?” A plugin that implements the Score interface is asked, “how good is this node for this Pod, from 0 to 100?” One plugin may take part in several extension points, which is useful when the same piece of knowledge is needed at filtering time and again at scoring time.

You do not replace the scheduler; you build a slightly different copy of it. A small program imports the standard scheduler command, registers your plugin under a name, and starts the scheduler as usual. The result is a custom scheduler binary that behaves exactly like the stock one, plus your plugin. You then enable your plugin by name in a profile.

Good plugin habits

Plugin ideas that teams actually build

The price of owning a build

A plugin is compiled against a specific version of Kubernetes libraries. When the cluster moves to a new minor version, you generally have to update the dependencies, rebuild, retest and redeploy your scheduler. Plan for this work in every upgrade cycle, and keep an automated test suite that runs your plugin against a real scheduler, with both fitting and non-fitting Pods, so that the upgrade is a routine task instead of a scary one.

Sometimes a custom scheduler runs next to the default one. It could be your own plugin build, or a completely separate product.

The steps in plain words

The danger of two cooks in one kitchen

Here is the problem that surprises most beginners. Two schedulers do not talk to each other. Each keeps its own picture of the cluster. Suppose a node has room for exactly one more Pod. The default scheduler and your custom scheduler both look at the node at nearly the same moment, both see room, and both place a Pod there. The node is now over-committed.

The kubelet on that node is the safety net. When it receives the second Pod, it re-checks the resources and rejects the one that does not fit, marking it as failed with a reason such as “OutOfcpu” or “OutOfmemory”. The controller that owns the Pod, for instance a Deployment, then creates a replacement, which gets scheduled again. The system heals, but you see odd failed Pods and slower startups.

How to avoid the clash

• Partition the nodes. Label and taint a group of nodes for the custom scheduler, so the default scheduler never considers them, and the custom one only places Pods there.
• Use a profile instead. If your need is only different scoring, put a second profile in the same scheduler process. Profiles share one cache and one queue, so they cannot race each other.
• Keep Pods separate. Make sure the types of Pods each scheduler handles do not overlap, using schedulerName consistently, ideally enforced by a policy.

Version matching

Keep the scheduler’s version in step with the cluster. A scheduler several versions behind may not understand newer Pod fields, and one far ahead may use API features the server does not offer yet. The usual rule is to match the control plane’s minor version, and to upgrade the extra scheduler right after the cluster.

An extender lets the standard scheduler ask a web service for help, without any recompiling.

How an extender works

An extender is a web service you run. You tell the scheduler its address in the configuration. At chosen moments the scheduler sends it a message over HTTP containing the Pod and the candidate nodes, and the extender answers. It can take part in several steps, each called a verb: filter (remove nodes), prioritize (give scores), preempt (help choose victims) and bind (take over the final binding).

Scores from an extender use a smaller scale, 0 to 10, which the scheduler multiplies by a weight you set and scales into its normal range. This is a small detail, but forgetting it leads to extenders that are either ignored or that dominate every decision.

Useful settings

Here are the situations where teams genuinely outgrow the default scheduler, and the tools the community has built for them.

All-or-nothing: gang scheduling

A distributed training job might need eight workers. Each worker is a Pod. If only five find a node, those five sit idle holding expensive accelerators while waiting for three more, and they may wait forever if another job grabs the remaining capacity. Two such jobs can block each other completely, a situation called a deadlock. Gang scheduling solves this by treating the group as a single unit: either all members get nodes, or none start.

Queues and fair sharing

When many teams share one cluster, you want fairness: each team has a guaranteed share, can borrow spare capacity, and gives it back when needed. The default scheduler thinks in single Pods and has no concept of a team’s quota queue. Tools that add queueing decide when a job may start at all, which is a different job from deciding where its Pods land.

Hardware awareness

Modern servers are not flat. Processors have sockets, memory is attached to specific sockets, and accelerators sit on specific buses. A Pod that gets a processor core on one socket but memory from another, or two accelerators that are far apart, runs measurably slower. Topology-aware scheduling picks nodes, and sometimes specific devices, so that closely related resources are physically close together.

Real usage, not just requests

The default scheduler trusts what Pods request, not what they actually use. A node that looks full on paper might be nearly idle because everyone over-requested. Load-aware plugins score nodes using measured usage so they can safely place more work on lightly used machines.

The happy path is simple. Real clusters spend a lot of time on the unhappy one, so it deserves its own chapter.

flowchart LR
  P["Pod created, nodeName empty"] --> AQ["Active queue"]
  AQ --> SCY["Scheduling cycle"]
  SCY -->|node found| BIND["Binding"]
  BIND --> RUN["Kubelet starts the Pod"]
  SCY -->|no node fits| PRE["Preemption tried"]
  PRE -->|victims chosen| NOM["Nominated node set, wait"]
  NOM --> AQ
  PRE -->|nothing to evict| UNS["Unschedulable pool"]
  UNS -->|event or timeout| BO["Backoff queue"]
  BO --> AQ
    

Follow the diagram. A new Pod enters the active queue and gets its scheduling cycle. If a node is found, the Pod is bound and the kubelet starts it. If no node fits, the scheduler tries preemption. If it can evict lower-priority Pods, it marks a nominated node on the waiting Pod and sends it back to the queue while the victims shut down. If there is nothing to evict, the Pod goes to the unschedulable pool and waits for a helpful event or a timer, then passes through the backoff queue and tries again.

How preemption really works

Preemption is not instant. The victims are given their normal shutdown grace period to finish their work. During that time, the incoming Pod holds a “nominated node” note so that other Pods do not take the space that is being freed. When the victims are gone, the incoming Pod usually lands on that node, though if something better appears meanwhile, it may go elsewhere. Preemption also tries to respect disruption budgets on a best-effort basis, but it is not guaranteed to honour them.

Reading the failure message

When a Pod cannot be placed, the scheduler writes an event called FailedScheduling, and the message is a short report. A typical message says that none of the nodes are available, followed by a count of the reasons: so many nodes had insufficient CPU, so many carried a taint the Pod does not tolerate, so many did not match the node selector. These counts add up to the total number of nodes. They are your first and best debugging tool, because they tell you which filter is rejecting your Pod.

A scheduler that makes perfect decisions slowly is worse than one that makes good decisions quickly. Speed is part of correctness here.

Why speed matters

The scheduler handles Pods one at a time in its scheduling cycle. Imagine a shop with one cashier. If each customer takes two seconds and customers arrive faster than one every two seconds, the line grows without limit. The same happens when a big job creates thousands of Pods at once and each decision is slow: the waiting line stretches, startup time grows, and users notice. Total throughput is simply how many decisions per second the cashier can finish.

Kubernetes is officially tested at about five thousand nodes in one cluster, with a default limit of one hundred and ten Pods per node. The default scheduler is tuned to stay responsive at that size, which sets a high bar for anything you add to it.

Ways to keep it quick

The scheduler sits on the critical path for every new Pod. Treat your custom scheduler with the same care as any core system.

What happens when the scheduler stops

There is a reassuring fact here. If a scheduler goes down, Pods that are already running keep running. The kubelet does not need the scheduler to keep containers alive. What stops is the placement of new Pods, including replacements for crashed ones. They stay Pending until a scheduler returns. This is serious but not catastrophic, which is why a failing custom scheduler should never take down the cluster by itself.

Staying available

Run two or three copies of your scheduler with leader election switched on. The copies compete for a small lease object stored in the API server. The winner is the active scheduler and keeps renewing the lease every few seconds. If it dies, the others notice that the lease has expired and one of them takes over, usually within roughly a quarter of a minute. Give your scheduler its own lease name so that it never fights with the default one.

The autoscaler blind spot

Cluster autoscalers decide whether to add a node by simulating the standard scheduler: “If I add this node, would the waiting Pod fit?” Tools such as Cluster Autoscaler and Karpenter know the built-in rules. They cannot know about your private filter plugin. If your plugin rejects nodes that the simulation says are fine, the autoscaler may keep adding nodes that your scheduler then refuses to use. Always test your custom scheduler together with your autoscaler, and prefer expressing constraints through labels, taints and affinity, which autoscalers already understand.

Security

A scheduler is powerful: it can read every Pod and node, and it decides where workloads run. Treat it carefully.

• Least privilege. Give the service account only the permissions it needs. Start from the standard scheduler’s roles and remove anything your scheduler does not use.
• Control who can pick which scheduler. A Pod chooses its scheduler through schedulerName, and that field cannot be changed once the Pod exists. Without a guard, any tenant could pick a scheduler with looser rules and dodge quotas or placement policies. Use an admission policy to allow only approved scheduler names per namespace.
• Protect the build. A custom scheduler binary has great influence. Build it in a trusted pipeline, scan it, sign the image and pin its version.
• Secure the endpoints. Keep the metrics and health ports behind the cluster network, and secure any extender web service with certificates.
• Remember placement is not a security boundary. Scheduling rules keep Pods apart for performance and availability, but they do not replace proper isolation such as network policy and container sandboxing.

You cannot improve what you cannot see. The scheduler exports a rich set of signals, and your custom parts should join in.

The metrics worth watching

Logs and events

The scheduler writes logs at several levels of detail, controlled by a verbosity setting. Normal levels show start-up and errors, and higher levels explain individual decisions, which is invaluable when diagnosing a surprising placement but too noisy to leave on. For each Pod, the events visible through kubectl describe show a “Scheduled” message naming the scheduler and node, or a FailedScheduling message with the per-reason counts we discussed earlier. Collecting these events into your logging system gives you a searchable history of every placement.

A short debugging routine

Teams who have lived with custom schedulers keep repeating the same lessons. Here they are, so you can skip the expensive part.

Patterns worth copying

Traps to avoid

Best practices at a glance

Common mistakes and quick fixes

These stories show the same pattern again and again: a standard scheduler for ordinary work, and something special for the workloads that are different.

A typical journey at a growing company

A company starts with the default scheduler and web services. Costs grow, so the platform team right-sizes requests and adds topology spread for resilience. Next a data team arrives with big batch jobs, and the platform team adds a packing profile so idle nodes can be released. Machine-learning projects follow, with distributed jobs that deadlock under the default model, so the team installs a queueing tool and a gang-scheduling plugin, rolling them out by opt-in scheduler name. Only a handful of unusual rules, such as data-residency checks, end up in a small custom plugin. By then each layer has a clear job, the default scheduler still handles most Pods, and the custom parts are small, tested and easy to turn off.

The Kubernetes scheduler has one clear job: choose a node for every new Pod. It does so through a queue and a two-stage cycle of filtering and scoring, with every rule supplied by a plugin. That plugin design is what makes it customizable. You can change settings through profiles, add your own logic as plugins, ask an outside service through an extender, or run a separate scheduler entirely. The lighter the tool, the lower the lifetime cost, so reach further only when a lighter tool truly cannot express what you need.