Most real companies keep their application images private. Kubernetes must therefore prove who it is before it can download them.

Where images live

An image is a packaged copy of a program together with everything it needs to run. Before Kubernetes can start a container, it must fetch that image from a place that stores images. That place is called a container registry. Some registries are public, like a public noticeboard: anyone can read. Many are private, like a company filing cabinet: only approved people or programs may open it.

Your company’s own applications almost always sit in a private registry. The images may contain business logic, internal tools, or code that competitors would love to see. Public access would be a serious risk, so the registry asks every visitor for credentials, meaning a username and a password or token, before it hands over an image.

The problem Kubernetes must solve

In a cluster, nobody sits at a keyboard typing a password each time a container starts. Pods are created automatically, on any machine, at any hour, sometimes hundreds at once. Kubernetes needs a way to hold the credentials safely and hand them to the right machine at the right moment, without putting them inside the image or the application code.

The answer is a Kubernetes object called a Secret, of a special type meant for registry logins. A pod description points to it, and Kubernetes passes it to the machine that will pull the image.

What happens without it

If a pod asks for an image from a private registry and no valid credentials are available, the pull fails. The pod stays stuck, and its status shows messages such as ErrImagePull, and then ImagePullBackOff. Almost every Kubernetes user meets these messages sooner or later. Understanding pull secrets turns them from a mystery into a five-minute fix.

Four characters appear in every image pull: the registry, the pod description, the secret, and the kubelet. Once you know their roles, everything else is detail.

The four characters

flowchart LR
    POD["Pod description: image name and secret name"] --> KL["Kubelet on the machine"]
    SEC["Image pull secret in the cluster"] --> KL
    KL -->|"presents login"| REG["Private registry"]
    REG -->|"sends image layers"| KL
    KL --> RUN["Container starts"]
        

Read the figure from left to right. The pod description says what to run and which secret to use. The kubelet reads both, then goes to the private registry and presents the login. If the registry accepts it, the image is sent back in pieces, and the container can start.

Important: the machine does the pulling

Many people imagine the pod itself downloads its image. In truth, the pod does not exist yet. The kubelet, which runs on the machine, pulls the image before the pod’s containers start, using the underlying container runtime, which is the software that actually runs containers. This is why the secret must be given to the kubelet, and why the credentials are not visible to the application inside the pod.

A few small ideas will make the rest of the tutorial simple. Each is explained in everyday words.

What an image name means

An image name has parts. First comes the registry address, such as the web name of the registry. Then a path, which often includes the team or project. Then the image name itself, and finally a tag or a digest. A tag is a friendly label, such as a version number. A digest is a long fingerprint that points to one exact image and never changes. If no registry address is written, Kubernetes assumes a default public registry.

Pods, nodes, and the kubelet

A pod is the smallest thing Kubernetes runs. A node is a computer, real or virtual, that runs pods. The kubelet is the agent on each node that receives instructions from the control part of the cluster and makes them real, including pulling images.

Namespaces

A namespace is a named area inside the cluster that groups related things. Many objects, including Secrets, belong to one namespace and can only be used by pods in that same namespace. This has a big consequence for pull secrets, which we will meet again: a secret in one namespace cannot be used by a pod in another.

Secrets, and what base64 really is

A Secret is a Kubernetes object designed to hold small pieces of sensitive information, such as passwords. Its values are stored in a form called base64. Base64 is only a way of writing data using letters and numbers. It is not encryption. Anyone can turn it back into the original text in a second. This is why access to Secrets must be carefully controlled, which we cover in the security chapter.

Service accounts

A service account is an identity that pods use inside the cluster. Every namespace has a default one, and each pod runs as some service account. A service account can also carry a list of image pull secrets, and any pod using that account will automatically get them. This is a very handy shortcut.

The image pull policy

Each container also has an image pull policy, which says when Kubernetes should pull. With one choice, the image is pulled every time a container starts. With another, it is pulled only if the machine does not already have it. With a third, it is never pulled, and the image must already be on the machine. This policy matters, because a pull secret is only needed when a pull is actually attempted.

An image pull secret is an ordinary Secret with a special type and a special shape. Knowing what is inside makes mistakes easy to spot.

The type and the key

Secrets have types, which tell Kubernetes what kind of information they hold. The type for registry logins is named kubernetes.io/dockerconfigjson. A Secret of this type must contain one entry, with the key named .dockerconfigjson. The value of that entry is a small piece of structured text in a format originally invented for Docker’s own login file.

What the text contains

Inside that text is a list of registries. For each registry, there is the address, and the credentials for it. The credentials appear as a username, a password or token, and often a combined field called auth, which is the username and password joined together and written in base64. The registry address in the secret must match the registry in the image name, otherwise Kubernetes will not use the secret for that image.

Ways to create one

There are several ways to make the secret. The most common uses the command line tool: you ask it to create a docker-registry Secret, and give it a name, the registry address, a username, and a password. The tool builds the correct structure for you. Another way is to point the tool at an existing Docker login file on your computer and let it copy the contents. A third is to write the full description as a file and apply it, which is popular in automated pipelines. A fourth is to let a controller or operator create and refresh the secret for you.

Use a robot account, not a person

Registries usually let you create robot accounts, sometimes called service accounts or deploy tokens, meant for machines. Choose one with read-only permission for just the repositories the cluster needs. If a personal account is used, the pulls will fail the day that person leaves the company or changes their password.

A secret does nothing until pods are told to use it. There are two main ways, and choosing well saves a lot of repetition.

Way 1: name it in the pod

Every pod description has an optional field called imagePullSecrets. It is a list of names of Secrets in the same namespace. When the pod is scheduled, the kubelet uses those secrets to pull the images for that pod. In a Deployment, the field sits inside the pod template, so that every pod the Deployment creates carries it.

This approach is explicit and clear. Anyone reading the description sees which secret is needed. But it must be repeated in every workload that pulls private images, which is tedious and easy to forget.

Way 2: attach it to the service account

A service account can also carry a list of image pull secrets. When a pod is created using that service account, Kubernetes automatically copies the list into the pod. So you attach the secret once to the account, and every pod that uses the account gets it. Most teams prefer this for a whole namespace, by attaching the secret to the default service account, or to a dedicated account used by their applications.

flowchart TD
    S["Image pull secret in namespace"] --> SA["Service account lists the secret"]
    S --> P1["Pod names the secret directly"]
    SA --> P2["Pod uses the service account"]
    P1 --> K["Kubelet gets the secret list"]
    P2 -->|"secrets copied into pod automatically"| K
    K --> PULL["Image pulled from private registry"]
        

The same-namespace rule

The secret must be in the same namespace as the pod. If a team has ten namespaces, each one needs its own copy of the secret. This is the most common reason a secret seems correct but is not used. It also means you need a plan for copying and refreshing the secret across namespaces, which we discuss later.

Multiple secrets

A pod may list more than one pull secret. This is useful when its containers come from different registries, for example one from your own registry and one from a partner’s. The kubelet chooses the credentials that match the registry of each image, and will try the available ones until one works.

A pull secret is the most common tool, but it is not the only one. Sometimes a different approach is simpler or safer.

Credentials configured on the node

The container runtime on each node can be given registry credentials directly, in a configuration file on the machine. Every pod on that node can then pull from that registry without any secret in the cluster. It is simple, but it has drawbacks. Every pod on the node shares the same access, the credentials must be placed on every node, and rotating them means touching every machine. It is used mostly in small or special setups.

Cloud credential providers

Managed Kubernetes services from big cloud providers often have their own private registries. They can give the kubelet short-lived credentials automatically, based on the identity of the machine or the workload. Kubernetes supports this through a plugin system for the kubelet, called credential providers. The kubelet asks the plugin for a fresh login whenever it needs to pull an image from a matching registry. No long-lived password is stored in the cluster at all.

Registry mirrors and pull-through caches

Some organisations run a registry near their cluster that fetches and keeps copies of images from elsewhere. Pods pull from this nearby registry. This can reduce delay, avoid limits set by public registries, and centralise credentials, because only the mirror needs a login to the upstream registry. The mirror itself may still need a pull secret for pods, or may be open only inside the private network.

Public registry limits

Some public registries limit how many images an anonymous visitor may pull in a period of time. In a busy cluster, many nodes share a few outside addresses, and the limit is reached quickly. A pull secret with a proper account, or a mirror, often raises the limit. So a pull secret is sometimes used even for public images, not for privacy, but to avoid being throttled.

Several parts of the cluster, and some outside it, take part in every image pull. Knowing each one tells you where to look when something goes wrong.

The parts, from top to bottom

Where the secret travels

The Secret is stored in the cluster’s store, usually in its base64 form, and sometimes encrypted at rest if the administrator has enabled that. When the kubelet needs it, it asks the API server for it. The kubelet is only allowed to read secrets for pods that are scheduled on its own node. This is a safety measure, so that one compromised node cannot read every secret in the cluster.

The kubelet then hands the login to the container runtime for the pull. It does not put it into the pod, and it does not write it to the pod’s environment.

The network between node and registry

The pull is a network trip from the node to the registry. Firewalls, proxies, and network policies can block it, even when the secret is perfect. Some organisations force all traffic through a proxy. Others use private links to their cloud registry. The nodes, not the pods, need to be able to reach the registry.

Let us follow a single image pull from the moment a pod is scheduled until the container starts.

The sequence

First, the pod is created and the scheduler assigns it to a node. The kubelet on that node notices the new pod and reads its description. For each container, it checks the image pull policy. If the image must be pulled, the kubelet gathers the credentials it may use: any secrets listed in the pod, and, if configured, credentials from node files or credential providers.

The kubelet then asks the container runtime to pull the image, passing the login. The runtime contacts the registry. The registry checks the login, and, if it is valid and allowed to read that image, it sends the image’s manifest, which is a small list of the pieces that make up the image, followed by the pieces themselves, called layers. When all layers arrive, the runtime unpacks them, and the kubelet starts the container.

sequenceDiagram
    participant K as Kubelet
    participant A as API server
    participant R as Container runtime
    participant G as Registry
    K->>K: New pod on this node, check pull policy
    K->>A: Fetch listed pull secrets
    A-->>K: Secret contents
    K->>R: Pull image with credentials
    R->>G: Request image with login
    G-->>R: Login accepted, send manifest
    R->>G: Request each layer
    G-->>R: Layers
    R-->>K: Image ready
    K->>R: Start container
        

Follow the figure from top to bottom. The kubelet first learns which secrets to use, and asks the API server for them. It passes the login to the runtime. The runtime and the registry then exchange the login, the list of layers, and the layers themselves. Only when the image is ready does the container start.

How the kubelet chooses a credential

When several credentials are available, the kubelet matches the registry address of the image against the registry addresses stored in the secrets. Matching may include the path, so a secret written for one part of a registry can be chosen over a more general one. If more than one matches, the kubelet may try them in order, and will stop at the first one that works. A failed attempt with one credential does not stop the others from being tried.

What happens after a failure

If the pull fails, the kubelet reports an error such as ErrImagePull. It then waits before trying again, and the waiting time grows each round, up to a limit of a few minutes. While it waits, the pod status shows ImagePullBackOff. This is called back-off, and it prevents a broken pull from hammering the registry. Once the cause is fixed, the next attempt succeeds, though you may wait a little for the timer.

Images already on the node

If an image is already on the node, the policy may allow a pod to use it without a new pull. This means a pod without any secret could start using an image that another pod pulled earlier with proper credentials. This is a known subtlety. For sensitive images, consider using the always-pull policy, so that the registry is asked every time, or use newer Kubernetes features that check credentials even for cached images, where they are available in your version.

Almost every team has faced this error. The good news is that the causes are few, and a fixed routine finds them quickly.

Start with the message

Describe the stuck pod and read its events. The events tell you the real reason. Messages mention things like authentication required, access denied, image not found, connection timed out, or unauthorised. Each points to a different layer of the problem, so reading the exact words saves much time.

flowchart TD
    S["Pod shows ImagePullBackOff"] --> E["Read the pod events for the exact message"]
    E --> Q1{"Message says not found?"}
    Q1 -->|"Yes"| F1["Check image name, tag, and registry spelling"]
    Q1 -->|"No"| Q2{"Message says unauthorised or denied?"}
    Q2 -->|"Yes"| F2["Check secret exists, is in the same namespace, and is referenced"]
    F2 --> F3["Check credentials are valid and have read permission"]
    Q2 -->|"No"| Q3{"Message says timeout or no route?"}
    Q3 -->|"Yes"| F4["Check network, proxy, firewall, and DNS from the node"]
    Q3 -->|"No"| F5["Check rate limits and registry status"]
        

The figure shows a routine you can repeat. First read the message. If the image is not found, the problem is the name or tag. If access is denied, the problem is the secret or the credentials. If the connection times out, the problem is the network. If none of these fit, look at limits and the health of the registry.

The usual causes, in order of how often they occur

A useful check: look from the node

If you can reach the node, try pulling the image by hand from there using the runtime’s tools and the same login. If it works by hand, the problem is with how Kubernetes passes the secret. If it fails by hand, the problem is with the credentials, network, or registry. This splits the search in half.

A pull credential has a life of its own: it is issued, stored, distributed, used, rotated, and finally revoked. Managing the whole life matters more than the first setup.

Expiry is the silent troublemaker

Some registries, especially those run by cloud providers, hand out login tokens that expire after hours or days. A pull secret built from such a token works perfectly at first and then, quietly, stops working. The cluster looks fine until a new pod needs a pull, perhaps at night when a node is replaced, and then it fails. Long-lived static tokens avoid this but are riskier. The best answer is automatic refresh, described in the next chapter.

Copying a secret by hand into thirty namespaces, and renewing it every twelve hours, is not a plan. Automation turns pull secrets from a chore into a background service.

Three jobs to automate

Common approaches

One approach is an external secrets operator. It reads credentials from a central store, such as a cloud secret manager or a vault, and creates and updates Kubernetes Secrets from them on a schedule. Change the value in the central store, and every copy is updated. Another approach is a small controller or scheduled job that logs in to the registry, gets a fresh token, and writes it into the Secrets. A third is to use a tool that replicates one Secret across selected namespaces.

Teams that use GitOps, where the cluster follows files kept in a repository, must be careful. The Secret must never be stored in plain form in the repository. Sealed or encrypted secrets, or references to an external store, keep the repository safe.

flowchart LR
    ST["Central secret store or registry token service"] --> OP["Operator or scheduled job"]
    OP --> N1["Secret in namespace A"]
    OP --> N2["Secret in namespace B"]
    OP --> N3["Secret in namespace C"]
    N1 --> SA1["Service account A links the Secret"]
    N2 --> SA2["Service account B links the Secret"]
    N3 --> SA3["Service account C links the Secret"]
        

In the figure, a single trusted source holds the real credential. An operator or job copies it into every namespace and refreshes it when needed. Each namespace’s service account links the Secret, so new workloads get it automatically. Changing the credential means changing only the source.

Rotation without downtime

To rotate safely, create the new credential first, and update the Secrets while the old credential still works. Watch that pulls succeed with the new one. Only then revoke the old credential. Because pods only need the Secret at the moment of a pull, there is no need to restart running pods, unless they were created with a stale copy of a service account’s list.

A pull secret is a key to your company’s software. Handle it with the care you would give any key to a valuable warehouse.

What could go wrong

If an attacker obtains a pull credential, they may download your private images, study them for weaknesses, and find any secrets accidentally baked inside. If the credential has write permission, they could even push a poisoned image that your cluster would then run. The strength of your defences should match how powerful the credential is.

Controls to put in place

Who can use a secret?

Anyone who can create a pod in a namespace can reference any pull secret in that namespace, and thereby use its power to pull images. So a pull secret in a shared namespace gives its access to everyone who can create pods there. If teams should not share access, give them separate namespaces and separate credentials.

Scope credentials by team

A single company-wide credential is convenient but dangerous. If it leaks, every image is exposed. Instead, use one credential per team or per environment, limited to the repositories that team uses. A leak then affects little, and the source is easier to trace.

Pull secrets are the standard, simple answer for private registries. They also have costs that grow with the size of your cluster.

The trade-offs in plain words

How it compares with the alternatives

Patterns are ways of arranging things that many teams have found to work. Anti-patterns are shortcuts that cause trouble later.

Helpful patterns

Anti-patterns to avoid

A short list of habits that keep image pulls reliable and safe, and the mistakes that most often cause outages.

Best practices

Common mistakes

A gentle start plan

In week one, create a read-only robot account for one team, make a registry Secret in one namespace, attach it to the default service account, and deploy one application to prove it works. In week two, write the automation so that every new namespace gets the Secret. In week three, add refresh and rotation, and encrypt Secrets at rest. In week four, review permissions and set up alerts for pull failures. Then repeat for other teams.

You do not want to learn about an expired credential from an angry customer. Watch for the signs early, and keep a simple routine for fixing problems.

What to watch

Where the signals appear

Alerts worth having

Alert when any pod has been stuck pulling an image for more than a few minutes. Alert when a credential will expire within a few days, or a refresh job fails. Alert when a new namespace lacks the pull Secret. Also alert on unusual registry activity, such as pulls from unknown addresses, which might mean a stolen credential.

Troubleshooting the usual suspects

Emergency path

If a credential leaks, revoke it at the registry first, since that stops the harm at once. Then create a new credential, update the Secrets, and restart any pods that are stuck. Afterwards, check the registry logs to see what the old credential was used for, and record the lessons.

Three short stories show the ideas in action. The details are typical rather than taken from any one company.

Example 1: The missing namespace secret

Example 2: The token that expired at night

Example 3: The leaked administrator token

What the stories share

In each case, the fix was less about clever tricks and more about a steady process: automate the boring parts, limit what each credential can do, and watch for early signs. Teams that built that process spent far less time fighting fires.

Short answers to questions that beginners often ask.

It is a Kubernetes Secret of the registry-login type. It holds the address of a registry and the credentials needed to pull images from it, and the kubelet uses it when downloading images for pods.

The pull failed and Kubernetes is waiting before it tries again. Common causes are a wrong image name or tag, missing or wrong credentials, a secret in the wrong namespace, network blocks, or rate limits. Read the pod events for the exact reason.

No. The pull secret must be in the same namespace as the pod. You need a copy in each namespace, ideally created and refreshed automatically.

For most teams, the service account is easier, since one link covers all pods that use the account. Naming the secret in the pod is more explicit. Either works, and they can be combined.

Not for access. But anonymous pulls from some public registries are limited, so using a login or a mirror can prevent failures in busy clusters.

Not by default. It is stored in base64, which is only an encoding. Turn on encryption at rest for the cluster store, and limit who can read Secrets.

Many cloud registries work with credential providers that give the kubelet short-lived logins automatically. That avoids storing passwords in the cluster. Check the documentation of your provider for the supported setup.

As often as your process can handle safely. For static tokens, a regular calendar such as every few months is common, plus immediately after any suspected leak. For short-lived tokens, refresh well before they expire.

One node may already have the image cached, so it does not need to pull, while the other must pull and lacks working credentials or a network path. Test with the always-pull policy or on a fresh node.

Running pods do not need the secret again unless they are recreated. The secret is used only when a pull happens. But if you added a secret to a service account, existing pods will not get it, so recreate them if they need to pull.

Yes. List all of them. The kubelet picks the credentials that match each image’s registry.

Create a read-only robot account, make a registry Secret in one namespace, attach it to a service account, and deploy one application. Then automate creation for new namespaces and set up refresh and alerts.

A last look at the whole picture, in plain words.

Private container images sit behind a login, like books in a members-only library. Kubernetes needs a membership card to borrow them, and that card is the image pull secret. It is a Secret of a special type that holds a registry address and credentials. The kubelet on each node reads it and uses it, together with the container runtime, to pull images before a pod starts.

The secret must live in the same namespace as the pod, and must be attached either directly in the pod or, more conveniently, through the service account. Other options exist, such as node-level credentials, cloud credential providers, and mirrors. Identity-based, short-lived logins are safest, because nothing long-lived needs storing.

When pulls fail, read the pod events, and follow the simple path from message to cause. Over the long term, treat credentials as things with a life: issue narrow read-only accounts, distribute and refresh them automatically, protect them with access rules and encryption, and revoke them quickly when needed. Do this, and image pulls become a quiet part of the background.