Containers are sealed boxes by design. That is wonderful for running software and awkward for fixing it.

A container is a running copy of a packaged application. It carries its own files, its own settings, and its own view of the world. From the outside, all you see is a name in a list. The container does not have a door for visitors. It was built to run a program, not to welcome people.

This sealed design is a strength in normal life, because it keeps applications neatly separated. But when something goes wrong, sealed boxes are frustrating. You may need to ask questions that only the inside can answer. Which files are here? What does this program think its settings are? Can it reach the database? Is the disk full? Guessing from outside is slow. Asking from inside is fast.

Questions that only the inside can answer

kubectl exec runs a command of your choice inside a container that is already running, and shows you the result.

Let us take the name apart. kubectl is the command-line tool people use to talk to a Kubernetes cluster. It is pronounced in different ways, such as “cube control” or “cube cuttle”. exec is short for execute, which means to run. So kubectl exec means “Kubernetes, please run this command for me inside that container.”

You tell it three things. First, which pod, meaning which running group of containers. Second, optionally, which container inside that pod. Third, what command to run. Kubernetes then carries your request to the right machine, starts the command inside the right container, and sends the output back to your screen.

What it is, why it exists, where it is used

What it is: a way to start a new process inside an existing container and connect its input and output to your terminal.

Why it exists: containers are isolated, and engineers still need to inspect and fix live systems. Exec gives a controlled way in, without needing to install remote login software inside every container.

Where it is used: debugging in development and test clusters, investigating incidents in production, running one-off maintenance jobs, checking connections and settings, and exploring how an application behaves.

Six small ideas are enough to understand the rest of this guide.

Kubernetes is a system that runs containers across many machines and keeps them healthy. You describe what you want, and Kubernetes makes it true and keeps it true. Here are the parts you will meet in this guide.

How a container is kept apart

A container feels like a small separate computer, but it is really an ordinary group of programs on a shared machine. Linux fences the group in using features called namespaces and control groups. Namespaces control what the group can see: its own files, its own network, its own list of processes. Control groups control what the group can use: how much processor time and memory.

This detail matters a great deal for kubectl exec. When exec starts a new program, it puts that program inside the same fences as the container. That is why the new program sees the container’s files and network, and why it uses the container’s share of memory.

A single command passes through four different programs on its way to a container.

The diagram below shows the journey. Your request begins on your own computer. It travels to the cluster’s front desk, then to the machine that hosts the container, and finally into the container itself. Each stop has a job.

flowchart LR
    User["Engineer with kubectl"] -->|secure request| API["API Server"]
    subgraph CP["Control Plane"]
        API --> Auth["Identity and permission checks"]
        Auth --> Audit["Audit log"]
    end
    API -->|streaming connection| Kubelet
    subgraph Node["Worker Node"]
        Kubelet["kubelet"] --> Runtime["Container runtime"]
        Runtime --> Proc["New process inside the container"]
    end
    

Meet the components

Exec does not restart your application and does not replace it. It adds a second program next to it.

Every container has a main process, the program the container was built to run. It is the reason the container exists. When you use kubectl exec, Kubernetes does not touch that main process. Instead, it starts a new process, inside the same fences, alongside the main one. When your command finishes, the new process ends, and the container carries on as if nothing had happened.

flowchart TD
    Cont["Running container"] --> FS["Same files and folders"]
    Cont --> Net["Same network address and ports"]
    Cont --> Env["Same mounted volumes and secrets"]
    Cont --> Lim["Same processor and memory limits"]
    New["New exec process"] --> FS
    New --> Net
    New --> Env
    New --> Lim
    New -.->|runs beside, never replaces| Main["Main application process"]
    

What the new process shares

• The filesystem. It sees the same files, folders, and mounted volumes as the main application.
• The network. It uses the same address and can connect to the same places, which is why it is good for testing connections.
• The environment. The settings the container was started with are visible to it, including any passwords delivered as settings.
• The resource limits. Its processor and memory use are counted against the container’s own limits.

What it does not share

The new process is not the main process, so it does not receive the main process’s input, and its output does not appear in the container’s logs. If you run a command that prints something, the text goes to your screen, not to the log system. This surprises many beginners who later look for that output in the logs and cannot find it.

Exec can run a single command and stop, or it can open a live session that stays open while you explore.

Style one: a single command

In its simplest form, you ask for one command, such as listing a folder or printing the current date. Kubernetes runs it, prints the result, and returns you to your own terminal. This is ideal for quick questions and for automation scripts. It is also the safest form, since it does exactly one thing and ends.

Style two: an interactive session

In the second form, you ask for a shell, which is a program that accepts typed commands one after another. You also ask for two extra options: one to keep your keyboard connected, called interactive mode, and one to make the session behave like a real terminal, called terminal or tty mode. With both switched on, your screen feels like you have logged in to the container. You can type, read results, move around folders, and use editing shortcuts.

Before you can step inside, you must say which door you mean.

Namespace and pod

Pods live in named sections of the cluster called namespaces, and two pods in different namespaces may even share a name. If you do not specify a namespace, kubectl uses your current default. Running exec against the wrong namespace is a common cause of the error “pod not found”. Always confirm where you are before you knock on a door.

Pods with more than one container

Many pods hold a main application and one or more helper containers, called sidecars. A sidecar might collect logs or handle secure connections. If you do not say which container you want, kubectl picks a default and prints a short message telling you which one it chose. Pods can also carry a note naming their preferred default container, which helps when the first container is not the one people usually need.

Using a Deployment instead of a pod name

Pod names contain random characters and change each time a pod is replaced. Kubectl lets you name the higher-level object, such as a Deployment, instead. It then picks one pod belonging to it. This is convenient, but remember that it picks one pod, often the first in its list. If your Deployment has ten copies, you are looking into only one of them, and that one may not be the copy that shows the problem.

Behind a simple command is a carefully built chain of streaming connections.

Step one: an ordinary request that becomes a stream

Most requests to Kubernetes are short: ask a question, get an answer, done. Exec is different. It needs a connection that stays open so that data can flow in both directions for as long as the command runs. Kubectl starts with an ordinary secure request, then asks the API server to upgrade the connection into a long-lived, two-way stream. Older versions used a protocol called SPDY for this. Newer versions use WebSockets, a standard technology also used by chat applications and live dashboards.

Step two: checking identity and permission

Before any stream is opened, the API server checks who you are and whether you are allowed to do this. Exec has its own permission, separate from reading or listing pods. A person who can list pods cannot necessarily exec into them. Administrators grant this permission carefully.

Step three: the API server becomes a relay

The API server then opens a second connection to the kubelet on the node that hosts your pod, and joins your connection to it. From now on it acts as a relay, passing bytes both ways. You never talk to the node directly, which is very useful. Your machine does not need any network path to the node at all, only to the API server.

Step four: the kubelet talks to the runtime

The kubelet asks the container runtime, using a standard interface called the Container Runtime Interface, to start the requested command inside the target container. The runtime creates the process, places it inside the container’s fences, and connects its input, output, and error streams to the connection from the kubelet.

sequenceDiagram
    participant E as Engineer
    participant K as kubectl
    participant A as API Server
    participant L as kubelet
    participant R as Container runtime
    participant P as Process in container
    E->>K: Ask to run a command in a pod
    K->>A: Request exec on that pod
    A->>A: Check identity and permissions
    A->>L: Open a streaming connection to the node
    L->>R: Ask the runtime to start the command
    R->>P: Start a new process in the container
    P-->>R: Output
    R-->>L: Output stream
    L-->>A: Output stream
    A-->>K: Output stream
    K-->>E: Show the output
    P->>R: Process exits
    R-->>K: Exit code returned along the chain
    

Let us follow one interactive session from the first keystroke to the last.

What happens when the connection breaks

If your network drops, or your laptop sleeps, the stream closes. When a terminal was in use, the shell running inside the container receives a hang-up signal and usually stops, along with anything you started from it. If you were running a long job through exec, it may be killed halfway. This is why exec is a poor tool for long tasks. Use a proper Kubernetes Job for anything that must finish reliably.

Exec is one of the most powerful permissions in a cluster. Treat it like a master key.

When someone can run commands inside your containers, they can read your application’s files, see its secrets, talk to its database, and change its behavior. In a production cluster, this is a serious power. Many security incidents in Kubernetes begin with someone who obtained the exec permission when they should not have had it.

How permission is controlled

Kubernetes uses role-based access control. Permissions are grouped under roles, and roles are given to people or programs. Exec is a separate item in that system, tied to pods. A role must explicitly include the permission to create exec sessions on pods, together with ordinary read access to the pod itself. Some newer versions also ask for read permission on the exec item. If a person lacks it, they see a “forbidden” error.

What audit logs show and do not show

An audit log records that a person opened an exec session, on which pod, at what time, and often the starting command. It does not record what a person types afterward in an interactive shell. So audit logs tell you that someone went in, but not always what they did. For very sensitive systems, teams add session recording tools or avoid interactive shells entirely.

Most of the time, exec is used for a short list of small, safe questions.

Exec is powerful, but it has firm limits. Knowing them saves hours of confusion.

Nothing you change is permanent

Files you create or edit inside a container live in a temporary layer. When the container restarts, that layer is thrown away and the container starts again from its original image. Anything not written to a proper persistent volume is lost. This is a feature, not a flaw: it keeps containers predictable. But it means that a fix made by exec is only a bandage. The real fix belongs in the image, the configuration, or the deployment description.

Minimal images have nothing to run

Security-conscious teams often build images that contain only the application and nothing else, with no shell and no debugging tools. Such images are smaller and harder to attack. But if you try to open a shell in one, you get an error saying the program cannot be found. Exec cannot run what is not in the image. The solution is a debug container, explained in the chapter on alternatives.

Kubernetes has several tools for looking at running pods. Each is best at a different job.

Beginners often reach for exec first because it feels the most powerful. But many jobs are easier, safer, or more reliable with a different tool. The table below sets them side by side.

flowchart TD
    Start["Something is wrong with a pod"] --> Q1{"Need to read what the app already wrote"}
    Q1 -->|Yes| LOGS["Use logs"]
    Q1 -->|No| Q2{"Need to reach the app from your laptop"}
    Q2 -->|Yes| PF["Use port-forward"]
    Q2 -->|No| Q3{"Does the container have a shell and tools"}
    Q3 -->|Yes| EX["Use exec"]
    Q3 -->|No| DBG["Use a debug container"]
    

Exec versus attach

These two are easily confused. Attach connects you to the main process that is already running, so you see its output and can type into its input. If it is a web server, there is usually nothing to type. Exec starts something new. If you attach to a program and press the wrong key combination, you may stop the main process itself. With exec, stopping your own session leaves the main process alone.

A hidden fact about copying files

The file-copy command is not magic. Behind the scenes, it uses exec to run an archive program inside the container and streams the result to you. So it fails if the image has no such program, even though it looks like a separate feature.

If the container has no shell, bring your own tools instead of giving up.

Modern best practice is to keep production images tiny, with only the application and its direct needs. This is good for security and speed, but it leaves you empty-handed when you need to investigate. Kubernetes solves this with ephemeral containers: temporary helper containers that you add to an existing, running pod. They share the pod’s network and can be told to share the process view of another container, so they can see what that container is doing.

Three ways to debug

Limits and cautions

Ephemeral containers cannot be removed from a pod once added, and they cannot be changed. They disappear only when the pod does. They also need permission, which is separate from exec and just as sensitive. Node-level debugging in particular gives very wide access, so it must be tightly controlled.

One exec session is cheap. Thousands of them are not.

The cost of a single session

A single session adds very little load. The new process is small, the stream carries only text, and the relay through the API server needs a modest amount of memory. For people using exec now and then, performance is a non-issue.

Where it can hurt

Exec depends on a chain of healthy parts. If any link is broken, it fails.

For exec to work, your network path to the API server must be open, the API server must be healthy, the node must be reachable from the API server, the kubelet must be running, the runtime must respond, and the container must exist and be running. That is six links. Knowing them turns a mysterious failure into a checklist.

The moving target

Pods are replaced frequently. A pod you are exploring may be deleted, moved, or restarted while you are inside. Your session then ends abruptly. During an incident, a crash-looping container might live for only a few seconds at a time, making exec nearly impossible. This is another reason to use logs, events, and debug copies first.

You cannot secure a doorway you cannot see. Watch who uses exec, and when.

Alerts worth setting up

• Any exec session opened in a production namespace by someone outside the on-call group.
• Exec against sensitive workloads, such as databases holding personal data.
• Exec sessions started from unusual locations or at unusual hours.
• Sudden spikes in exec requests, which may signal scripts gone wrong or an intruder.
• Shells starting inside containers that normally never run one.

Exec is a sharp tool. It saves time, and it can cut the hand that holds it.

The main trade-offs

Convenience versus control. The easier it is to step into containers, the more careful the rest of your system must be about who can. Teams choose a point between “anyone in development” and “nobody in production without approval”.

Rich images versus safe images. Images with shells and tools are easy to debug but easier to attack. Minimal images are safer but need debug containers. Most mature teams choose minimal images plus a debug process.

Speed of fix versus quality of fix. Exec can restore service in minutes, but a fix made by hand inside a container is invisible to your deployment process and vanishes on restart. The quick fix must always be followed by a real one.

Some habits around exec make teams safer. Others make them fragile.

Patterns that work

Anti-patterns to avoid

A small set of habits keeps exec safe and useful.

Best practices

Common mistakes

The same practices appear in teams of every size.

Exec errors look scary, but each one points to a specific broken link in the chain.

Understanding how permission is written down makes both security and troubleshooting much easier.

Kubernetes decides what a person may do using a system called role-based access control, usually shortened to RBAC. It has three simple parts. A role is a list of allowed actions. A subject is a person, a group, or a program. A binding connects a subject to a role. Together they answer the question, “Is this identity allowed to do this action on this thing?”

Exec is its own door

Kubernetes treats objects and their special actions as separate items. A pod is one item. The exec feature attached to a pod is another item, called a sub-resource. A role that allows reading pods does not automatically allow exec, and a role that allows exec does not automatically allow deleting pods. This separation lets administrators grant fine slices of power.

Scope: one namespace or the whole cluster?

A role can apply to a single namespace or to the entire cluster. This choice is one of the most consequential in Kubernetes security. An exec permission granted for one namespace lets a person enter pods only in that section. The same permission granted at cluster scope lets them enter every pod in every namespace, including the system pods that manage the cluster itself. Always start narrow.

Let us investigate a pretend problem from start to finish, without typing a single command, so you can see the thinking behind the tool.

Suppose a small web application shows an error saying it cannot reach its database. You suspect that the application has the wrong database address. Here is how a careful engineer approaches it.

What this story teaches

Exec was used exactly once, for a short, read-only question that logs could not answer. The fix was made in the proper place, so it lasts and reaches every copy. Most good uses of exec look like this: a quick look, a clear answer, and a durable fix elsewhere.

You do not need these on day one, but knowing they exist will save you confusion in real clusters.

Exec inside health checks

The last card deserves a moment. A pod can be configured with a probe, a small test that Kubernetes repeats regularly to judge whether a container is alive or ready for traffic. One kind of probe runs a command inside the container and treats a zero exit code as success. This shows the same idea from a different angle: exec is not only a human tool, but a basic building block that Kubernetes uses on its own behalf.

Where exec fits in the bigger story

As you grow, you will notice that Kubernetes access tools form a ladder. Reading descriptions and events is the safest step. Reading logs comes next. Tunneling to an application port is after that. Running a command inside a container is higher. Adding a debug container or opening the node itself sits at the top. Each rung gives more power and needs more care. Good engineers climb only as high as the problem requires.

Quick answers to the questions beginners ask most often.