Start with the real-world pressures that push teams toward more than one scheduling policy.

What a scheduler is

A scheduler in Kubernetes is the component that picks a node for every new Pod that does not yet have one. When you create a Pod, it first exists only as a record in the database with an empty node field. The scheduler notices that record, studies the available nodes, picks the best one, and writes the choice back. After that, the kubelet on the chosen node starts the containers. The scheduler never runs containers itself. It only makes the decision.

When the default policy falls short

The default scheduler is built to be general. It balances load, respects resource requests, honours affinity and spreads Pods across failure zones. Many teams never need anything else. Trouble starts with specialised needs, which the following cards describe.

To reason about several schedulers, first understand how a single one thinks.

Two phases per Pod

For each Pod, the scheduler runs a scheduling cycle followed by a binding cycle. The scheduling cycle is the thinking part: it filters out nodes that cannot host the Pod, then scores the remaining ones, and picks the winner. The binding cycle is the commit part: it tells the API server, “this Pod goes to that node.” The scheduling cycle handles one Pod at a time so that its decisions stay consistent. The binding cycle can run in the background while the scheduler moves on to the next Pod.

Filter, then score

Filtering removes nodes that fail hard requirements, such as not enough free CPU or memory, a taint the Pod does not tolerate, or a required label that is missing. Scoring ranks the survivors on softer preferences, such as spreading evenly or preferring nodes that already have the container image. The highest score wins, and ties are broken at random.

The scheduling queue

Unscheduled Pods wait in an internal queue. The queue has more than one part, and understanding it helps later when we talk about problems.

The scheduler’s local cache

Just like other Kubernetes controllers, the scheduler does not ask the API server about nodes and Pods at every decision. It uses informers to keep its own in-memory picture of nodes, Pods and other relevant objects, updated by watch events. When it picks a node, it immediately records that the Pod now occupies space there, before the official confirmation comes back. This early bookkeeping is called assuming the Pod, and it lets the scheduler make rapid, consistent decisions. It also hints at the heart of the multi-scheduler problem: this cache belongs to one scheduler process only, so a second scheduler has its own separate cache and does not know what the first has just assumed.

The mechanism is wonderfully simple: a single name written inside the Pod.

The scheduler name field

Every Pod specification has an optional field called schedulerName. If you leave it empty, Kubernetes fills it with the value default-scheduler. Every scheduler instance is started with a name, and it only picks up Pods whose field matches its own name. Pods that name someone else are ignored. That is the entire routing system.

What happens with an unknown name

This last point trips up many beginners. If a Pod asks for a scheduler that does not exist or is not running, nothing complains loudly. The Pod just stays in the Pending state with no node assigned, and the default scheduler also leaves it alone because the name does not match. The usual clue is that the Pod has no scheduling events at all, neither a success nor a failure message, because no scheduler ever looked at it.

Where the name is set

Because workloads like Deployments and Jobs contain a Pod template, the name is set inside that template. Changing it on a Deployment triggers a normal rolling update, and the new Pods will be handled by the new scheduler while old Pods stay where they are. Existing running Pods are never moved or re-scheduled just because their scheduler name is later changed on a controller.

Running a second scheduler program is only one of several options, and often not the best one.

Option A: scheduling profiles in one scheduler

A single scheduler process can run several profiles. A profile is a named set of rules that decides which plugins are active and how they are weighted. Each profile has its own scheduler name, so Pods pick a profile exactly as they would pick a separate scheduler. All profiles share the same process, the same queue and the same cache of the cluster.

Option B: separate scheduler programs

You can run another scheduler as a distinct program, for example as its own Deployment, with its own name, configuration and cache. It can be the standard scheduler built with different settings, a custom build with extra plugins, or an entirely different project written by someone else.

Option C: extend the default scheduler

Instead of replacing the scheduler, you can attach to it. Plugins compiled into a scheduler build add new filtering or scoring steps. Extenders are external web services that the scheduler calls during its work to get extra opinions. These approaches change how a scheduler decides without creating a competing scheduler.

flowchart TD
  Q["Need different placement behaviour?"] --> A{"Can existing plugins express it?"}
  A -- "Yes" --> P["Use a second profile in the same scheduler"]
  A -- "No, but logic fits the framework" --> X["Add a custom plugin or an extender"]
  A -- "No, needs a different queue or isolation" --> S["Run a separate scheduler with strict partitioning"]
  P --> OK["Lowest risk and lowest effort"]
  X --> OK2["Moderate effort, still one decision maker"]
  S --> OK3["Highest effort, manage race conditions"]
        

Figure 1 turns the table into a simple path. Notice that the expensive branch is the last one. Many teams jump straight to it because “a second scheduler” sounds like the obvious answer. Walking the path first usually saves months of operational pain.

The modern scheduler is a pipeline of small plugins. Knowing the stages makes profiles easy to understand.

A pipeline of stages

The scheduler is organised as a series of well-defined stages, called extension points. At each stage, any number of plugins can take part. A profile is essentially a choice of which plugins are switched on at which stages, and with what weights. The most important stages are listed below.

flowchart LR
  QS["Queue sort"] --> PF["Pre-filter"]
  PF --> F["Filter"]
  F --> PO["Post-filter and preemption"]
  F --> SC["Score"]
  SC --> RS["Reserve"]
  RS --> PM["Permit"]
  PM --> PB["Pre-bind"]
  PB --> B["Bind"]
  B --> PS["Post-bind"]
        

Why this matters for multiple schedulers

Because behaviour is built from plugins and weights, many “I need a different scheduler” requests really mean “I need different weights” or “I need one more plugin.” For example, packing Pods tightly is often a matter of changing which resource scoring strategy is used. Spreading preferences can be tuned by weights. Group scheduling uses the permit stage. Seeing the pipeline makes it much easier to ask whether a profile can do the job before building something heavier.

The safest way to run several scheduling personalities: one program, several named profiles.

How profiles are defined

The scheduler reads a configuration file, an object of the kind KubeSchedulerConfiguration, that contains a list of profiles. Each profile has a unique schedulerName. Inside it you enable or disable plugins at each stage and set weights or plugin arguments. A profile named like the default one keeps normal behaviour, and extra profiles can be added beside it with a different name.

What profiles share

All profiles share the scheduler’s cache, informers, queue and leader election. Because a single decision loop handles all Pods one at a time, the scheduler always knows which resources it has just promised to other Pods, regardless of profile. Two profiles can never hand out the same free space twice, which is why this approach avoids the most feared multi-scheduler failure.

Typical profile examples

When profiles are not enough, here is what a clean, safe separate deployment involves.

The ingredients

A second scheduler is just another workload in the cluster, usually a Deployment in the system namespace. It needs an identity, permissions, a configuration, and a unique name. None of these steps is hard, but forgetting any of them gives confusing results.

Version compatibility

A scheduler program is closely tied to the Kubernetes version because it uses the same internal frameworks and API versions. A common safe practice is to run a scheduler build of the same minor version as the control plane, and to upgrade the extra scheduler together with the cluster. A custom scheduler left behind on an old version may miss newer Pod features, for example newer scheduling directives, and quietly place Pods in ways that ignore those fields.

Managed clusters and control plane limits

On many managed Kubernetes services the control plane’s own scheduler cannot be reconfigured by users. This is actually a reason that separate scheduler programs became popular: you cannot change the built-in one, but you can run another beside it. On such platforms a separate scheduler running as a normal workload, or an extension project that provides its own scheduler name, is often the only route to custom policies. Always check your provider’s documentation about supported scheduling customisation.

Many replicas for safety, but exactly one active decision maker for each scheduler.

Why only one replica may decide

If two copies of the same scheduler decided at once, they would assign the same Pod twice or hand out the same free space to different Pods. So replicas of one scheduler take turns using leader election. They compete for a small shared record, a lease stored in the cluster. The holder renews it every few seconds. If the holder crashes and stops renewing, another replica takes over within a short time, typically within seconds up to a quarter of a minute depending on settings.

Each scheduler has its own microphone

The shared record is identified by a name and namespace. Two different schedulers, for example the default one and a custom one, must use different record names. Otherwise they fight for the same microphone and only one of them works. Replicas of the same scheduler must use the same name, so they can elect among themselves. In summary: same scheduler, same lock name; different scheduler, different lock name.

flowchart TD
  subgraph D["Default scheduler group"]
    D1["Replica 1 - leader"]
    D2["Replica 2 - standby"]
  end
  subgraph C["Custom scheduler group"]
    C1["Replica 1 - standby"]
    C2["Replica 2 - leader"]
  end
  LD[("Lease: default scheduler lock")] --- D1
  LD --- D2
  LC[("Lease: custom scheduler lock")] --- C1
  LC --- C2
        

Warm standbys

Standby replicas still run their informers and keep their caches warm. When a standby becomes leader, it starts scheduling immediately without waiting to re-read the entire cluster state. This is the same benefit seen in other controllers: a failover costs a few seconds, not a cold start of minutes.

Failure windows to expect

During a leader change, new Pods wait a few seconds before being placed. Already running Pods are unaffected, because scheduling only matters at the moment a Pod is first placed. A quiet leader failover is therefore not an outage, though a long-lasting inability to elect a leader is, because new Pods for that scheduler would pile up in Pending.

Follow two Pods, each owned by a different scheduler, from creation to running.

The shared stage

Both schedulers watch the same API server using informers. When any new Pod is created, both of them see the event. Each one checks the scheduler name in the Pod. The matching scheduler adds it to its queue, while the other ignores it. After that, each works through its own queue independently.

sequenceDiagram
  participant U as User
  participant A as API Server
  participant S1 as Default Scheduler
  participant S2 as Custom Scheduler
  participant K as Kubelet
  U->>A: Create Pod A (default name)
  U->>A: Create Pod B (custom name)
  A-->>S1: Watch event for Pod A and Pod B
  A-->>S2: Watch event for Pod A and Pod B
  Note over S1: Pod A matches, Pod B ignored
  Note over S2: Pod B matches, Pod A ignored
  S1->>S1: Filter, score, assume Pod A on node 1
  S2->>S2: Filter, score, assume Pod B on node 2
  S1->>A: Bind Pod A to node 1
  S2->>A: Bind Pod B to node 2
  A-->>K: Pod A and Pod B assigned to their nodes
  K->>K: Start containers
        

Reading the flow step by step

The user creates Pod A with the default name and Pod B with the custom name. The API server stores both and the watch events reach both schedulers. Each scheduler runs its own filter and score steps for the Pod it owns and records, in its private cache, that the chosen node now has less free space. Each then sends a binding request to the API server. The API server accepts the bindings, and the kubelet on each chosen node, which is watching for Pods assigned to it, starts the containers.

Where the schedulers learn about each other

Look closely at the middle of the diagram. When the default scheduler assumes Pod A on node 1, the custom scheduler does not hear about it at once. It only learns after the binding is stored and the resulting watch event arrives. That delay is typically tiny, from milliseconds to perhaps a few hundred milliseconds under load, but it is not zero. During that window, the custom scheduler might still believe node 1 has the space that Pod A has just claimed. The next chapter shows what happens when both of them want the same space.

The race condition that makes experienced operators nervous about separate schedulers.

The race in plain words

Suppose a node has room for exactly one more Pod. At almost the same moment, the default scheduler decides to place Pod A there, and the custom scheduler decides to place Pod B there. Both read the same free space from their caches, both think it is available, and both send a binding. The API server does not check resource arithmetic when storing a binding. It only checks that the Pod is not already bound. So both bindings are accepted, and the node is now promised more than it can hold.

Who catches the mistake

The safety net is the kubelet on the node. When it receives a newly assigned Pod, it performs its own admission check against the real resources of the node. If the node cannot actually accommodate the Pod, the kubelet rejects it and marks it as failed, with a reason such as insufficient CPU or memory. A controller above the Pod, for example a Deployment’s ReplicaSet, then creates a replacement, which goes through scheduling again. So the system does heal, but it pays with a failed Pod, a short delay and some noisy events.

sequenceDiagram
  participant S1 as Scheduler 1
  participant S2 as Scheduler 2
  participant A as API Server
  participant K as Kubelet on node X
  Note over S1,S2: Both caches show room for one Pod on node X
  S1->>A: Bind Pod A to node X
  S2->>A: Bind Pod B to node X
  A-->>K: Pod A assigned
  A-->>K: Pod B assigned
  K->>K: Admit Pod A, space used
  K->>K: Check Pod B, no space left
  K-->>A: Mark Pod B failed, out of resources
  Note over A: Controller creates a replacement Pod
        

Why it is worse than it sounds

A single rejected Pod seems minor. The danger grows in busy clusters. If batch jobs submit thousands of Pods at once and two schedulers compete for the same nodes, conflicts become common. Failed Pods accumulate, replacement Pods trigger more scheduling load, and operators see a flood of failures that look like node problems but are really coordination problems. Some Pod types, for example those from a Job with a low retry limit, may even cause the whole Job to be marked failed because too many of its Pods were rejected.

The cleanest way to avoid the race is to make sure two schedulers never compete for the same nodes.

The big idea

If each scheduler only ever considers its own set of nodes, they can never double-book the same space. This is partitioning. It turns a risky shared-resource problem into two independent, safe problems. There are several tools that help, and they can be combined.

flowchart LR
  subgraph PoolA["Node pool A: general workloads"]
    N1["Node 1"]
    N2["Node 2"]
  end
  subgraph PoolB["Node pool B: batch and GPU workloads"]
    N3["Node 3"]
    N4["Node 4"]
  end
  S1["Default scheduler"] --> PoolA
  S2["Custom scheduler"] --> PoolB
        

Layered defence works best

Relying on only one tool is fragile. For example, node labels alone depend on every workload author remembering the right selector. Adding a taint on the batch pool protects it from stray default-scheduler Pods. Adding an admission rule that forces the right scheduler name for the batch namespace removes human error. With all three, mistakes are caught at the earliest and cheapest place.

Priority is a cluster-wide idea, but each scheduler acts on it independently. That has consequences.

Priority classes

A priority class gives a Pod a number that represents how important it is. When the scheduler cannot place a high-priority Pod anywhere, it may decide to evict lower-priority Pods to make room. This act is called preemption. The evicted Pods are given their normal grace period to shut down, and then the high-priority Pod takes the freed space.

What changes with several schedulers

Each scheduler evaluates preemption on its own cache. A custom scheduler may decide to evict Pods that were placed by the default scheduler, and the reverse is also possible. If the two schedulers are not coordinated, they can end up in a cycle where one evicts Pods that the other had just placed, and the evicted Pods get scheduled again somewhere else, possibly triggering more evictions. In a partitioned setup, where the nodes are separate, this is largely avoided because each scheduler can only evict Pods on nodes it manages.

Practical advice

Before writing your own scheduler, look at what the community and the framework already provide.

Scheduler extenders

An extender is an external web service that the scheduler calls at certain points, to filter nodes, to add scores or to take part in binding and preemption. This lets teams add custom logic in any language without rebuilding the scheduler. The price is speed, since each call goes over the network, and failure handling, since an unreachable extender can slow or block scheduling unless it is marked as optional. Extenders are generally seen as a legacy-friendly option, with in-process plugins preferred for new work.

Out-of-tree plugins

The scheduling framework lets you compile extra plugins into your own scheduler build, without changing the main Kubernetes code. These run in-process and are fast. The community project often called the scheduler-plugins collection offers ready-made plugins for things like group scheduling and capacity-aware scheduling, and is a good first place to look before inventing anything.

Alternative schedulers for specialised worlds

Where teams actually reach for multi-scheduler setups, and what each one gains.

Scheduling speed matters most when large bursts of Pods arrive together.

What makes scheduling slow

Scheduling cost grows with the number of Pods waiting, the number of nodes, and the number and cost of the plugins used. Expensive plugins, such as those that examine relationships between Pods, are costly because the work multiplies by the number of nodes and existing Pods. Extenders add network delays to each decision. Large queues of unschedulable Pods that keep being retried can also burn CPU.

Tuning options

Profiles versus separate programs for throughput

A single scheduler process makes decisions one Pod at a time, so heavy load on one profile can delay Pods of another profile sharing the queue. A separate scheduler program has its own decision loop and can work in parallel with the first, so total throughput can be higher. This is one of the few genuine performance arguments in favour of a second program. If you have a high-volume batch workload next to latency-sensitive services, isolating them in separate programs can protect service placement latency.

A scheduler decides where code runs. That makes it a sensitive component to protect and to constrain.

Permissions for each scheduler

A scheduler needs to read many object kinds and to create bindings and events. It does not need to read Secrets or to modify Deployments. Binding it to the standard scheduler roles, and nothing more, keeps its access small. A custom scheduler built with extra features may need extra permissions, which should be added deliberately and reviewed. A scheduler with broad rights is an attractive target, because anyone who controls it also controls where every Pod lands.

Who may choose a scheduler

Any user who can create Pods can write any scheduler name into them. In a shared cluster that is risky. A tenant could pick a permissive custom scheduler to escape placement rules enforced by the default one, or send Pods to nodes they should not use, or choose a non-existent name and leave Pods pending forever. Cluster rules enforced at admission time can restrict which scheduler names are allowed in which namespaces, closing this gap.

Trust and supply chain

Audit and detection

Audit logs record who created Pods with unusual scheduler names, who changed scheduler configurations, and who bound Pods to nodes directly. Reviewing these entries regularly helps spot misconfiguration and abuse. Alert when a Pod appears with a scheduler name that does not belong to any running scheduler, as that usually means either a typo or a probing attempt.

You will need to see each scheduler’s health, and to find out quickly why a Pod is not placed.

Metrics worth watching

The scheduler exposes metrics in the standard monitoring format, and many are labelled by profile, which is very helpful when several profiles or schedulers coexist. The most useful signals are listed below in plain language.

Reading Pod events

The easiest tool is the event list on a Pod. A successful scheduling event names the component that did it, so you can tell at once whether the default or the custom scheduler placed the Pod. A “failed scheduling” event lists, in counts, why each node was rejected, such as insufficient resources, taints or affinity. A Pod with no scheduling events at all is the warning sign for a wrong or unowned scheduler name, or a scheduler without a leader.

A troubleshooting path for a Pending Pod

Logging with purpose

Raise scheduler log detail only temporarily, since high verbosity is very noisy. Log the Pod identity, the chosen node and the profile. Keep records of configuration changes with who made them and when. Correlate scheduler logs with node-level rejections when investigating suspected double-booking.

Habits that keep multi-scheduler clusters calm.

Patterns that work

Do and do not

Failures that appear again and again in real clusters.

An honest balance sheet for the multi-scheduler approach.

Choosing between the options

Common questions from people meeting multi-scheduler setups for the first time.

Yes. Kubernetes is designed to allow it. Each Pod names the scheduler it wants, and each scheduler only handles Pods carrying its own name. The default scheduler keeps working for every Pod that does not ask for another one.

It stays Pending indefinitely. No scheduler claims it, including the default one, and there are usually no scheduling events, which makes the problem look silent. Fixing the name or deploying the missing scheduler resolves it.

From a Pod’s point of view they look the same, because the Pod just names a scheduler. Internally a profile lives inside one scheduler program, sharing its queue and cache, while a separate scheduler is its own program with its own cache. That difference decides whether the double-booking race exists.

They can, but not without risk. If both hold stale views of free space they may promise the same capacity twice, and the node will reject the later Pod. Safe sharing is possible for light loads, yet partitioning nodes is the dependable solution.

No. Running Pods stay where they are. The change causes a rolling update, and only the newly created Pods are handled by the new scheduler.

Because only one replica should decide at a time. Two active copies of the same scheduler would behave like two competing schedulers. Standby replicas keep warm caches and take over quickly if the leader stops.

Yes, you set the scheduler name to the profile’s name in the Pod template. No other change is needed. Workloads that never set a name continue to use the default profile.

Rarely. Most needs are met by plugin weights, extra plugins or an existing project. A fully custom scheduler brings a heavy long-term maintenance burden and is justified only when nothing else fits.

Look at the Pod’s events. The successful scheduling event names the component responsible. Comparing it with the scheduler name in the Pod confirms that routing worked as intended.

Yes. A Pod with a node name already set is not scheduled by anyone, and scheduler filters are skipped, though the node’s own admission check still applies. Limit who can create such Pods if your isolation depends on scheduling rules.

Not usually. Many queueing controllers decide when a job may start and then let the ordinary scheduler place its Pods. They avoid the double-booking risk, but they do not provide custom node-selection rules. Always read a tool’s description to see whether it replaces the scheduler or sits in front of it.

Check that the scheduler name in the Pod exactly matches a running scheduler, then that the scheduler has an active leader and enough permissions. Those three checks solve most cases quickly.

The Kubernetes scheduler picks a node for every new Pod by filtering unsuitable nodes and scoring the rest, working from a private in-memory picture of the cluster. Pods choose their scheduler with a single name field, and each scheduler only handles Pods carrying its own name. When one policy is not enough you can add profiles to the same scheduler, extend it with plugins or extenders, or run separate scheduler programs. The first two keep one decision maker and one cache, so they are the safest. The last gives full independence but introduces the central risk of this topic: two caches, two decision loops and the chance that both promise the same free space to different Pods.

Return to the airport from the beginning. Several gate controllers are fine if each owns separate gates, if each has its own radio channel, and if a supervisor makes sure that no passenger can walk through an unmarked door. In Kubernetes terms, that means partitioned node pools, unique lease names, admission rules and good monitoring. With those in place, multiple schedulers let each kind of workload get the placement it deserves, without turning your cluster into a place where Pods fight over the last seat.