What Is kubectl logs in Kubernetes?
How to listen to what your applications are saying from the inside, even though they live in sealed boxes on computers you may never touch, explained step by step from zero.
Imagine a chef working behind a closed kitchen door. You are in the dining room. The food is late, and you have no idea why. Is the oven broken? Did the vegetables run out? Is the chef simply stuck on a hard recipe? You cannot see through the door. But there is a small window in the wall, and through it you can read the chef’s order notes as they are written: “table 4 waiting”, “oven not heating”, “out of onions”.
Applications in Kubernetes live behind a closed door just like that chef. They run inside sealed boxes, spread over many computers. When something goes wrong, you cannot walk in and look around. The window is a command called kubectl logs, and it lets you read the notes your application writes while it works.
In this guide you will learn what logs are, what kubectl logs does, how it travels through the cluster to fetch those notes, how to use its options, what to do with crashed containers, where the command runs out of power, and how professionals build a safer, longer-lasting memory for their logs. Every new word will be explained the first time it appears.
1The Problem: You Cannot See Inside the Box
Before we meet the tool, let us feel the pain it removes. A tool only makes sense once you see the trouble it solves.
A quick look at Kubernetes
A container is a small sealed box holding a program and everything the program needs. Kubernetes is the free, open-source system that starts, stops, moves and repairs thousands of these boxes across many computers. A group of computers managed together is a cluster, and each computer in it is a node. Containers run inside small wrappers called Pods, the smallest thing Kubernetes manages.
Why problems are hard to find
On your own laptop, if a program misbehaves, you can see its messages in the window where you started it. In Kubernetes, nobody starts the program by hand. The cluster starts it somewhere, on a computer you may not even know the name of. Consider what can go wrong.
- The Pod is running, but a page in your website shows an error to users.
- The Pod keeps restarting, and you do not know why.
- The Pod started fine yesterday, but today it is slow and you want to see what it is doing.
- A new version was released, and you want to confirm it started without complaints.
In each case, the first useful clue is almost always the same: what did the application itself say?
Think of the flight recorder in an airplane. Pilots and engineers cannot see inside a jet engine while it is flying, but the recorder keeps a running diary of what happened, so that people can read it afterwards. Application logs are the diary of a program, and kubectl logs is the way to open the diary and read it.
When something is wrong in Kubernetes, logs are usually the first place to look. They tell you what the program itself believed was happening.
2Building Blocks You Need First
Eight small ideas carry this whole topic. Learn them once and the rest will feel natural.
Log
A line of text an application writes to describe something that happened, such as “user signed in” or “could not reach the database”. Many lines together form the log.
Standard output and standard error
Two built-in “pipes” that every program can print to. Standard output (stdout) is for normal messages. Standard error (stderr) is for problems. Kubernetes listens to both.
kubectl
The command-line tool used to talk to a Kubernetes cluster. Its name is often read as “cube control” or “cube cuttle”. It is how you ask the cluster questions and give it orders.
API Server
The front desk of the cluster. Every request, including your request for logs, goes through it first. It checks who you are and whether you are allowed.
Kubelet
A helper program on every node. It starts the Pods assigned to its node, and it is also the one that can hand out that node’s container logs.
Container runtime
The program on the node that actually runs the containers, such as containerd. It also catches the messages a container prints and writes them into files.
Namespace
A named room inside a cluster that keeps one team’s things apart from another’s. When you ask for logs, you usually need to say which namespace the Pod is in.
Permission (RBAC)
The rulebook saying who may do what in the cluster. Reading logs is a permission of its own, so not everyone who can list Pods can read their logs.
Standard output and standard error are like two boxes on a chef’s counter: one labeled “normal orders” and one labeled “problems”. The chef drops notes into whichever box fits. The kitchen manager (the container runtime) collects notes from both boxes and files them in one folder, in the order they were written.
3What Exactly Is a Log in Kubernetes?
Kubernetes has a very simple idea of a log. Understanding it explains almost everything else in this guide.
The simple rule: print it and Kubernetes will keep it
Kubernetes expects your application to write its messages to standard output and standard error. That is all. It does not ask you to open log files, connect to a special service or install a library. The container runtime on the node catches everything the container prints and writes it to a file on that node, adding a small stamp to each line with the time and which of the two pipes it came from.
This idea is popular well beyond Kubernetes. A famous set of guidelines for building modern applications, known as the Twelve-Factor App, says a program should treat its logs as a stream of events and simply print them, leaving the collecting and storing to the environment around it.
What a good log line contains
- A time. When did it happen?
- A level. Is it just information, a warning, or a real error?
- A message. What happened, in plain words?
- Context. Which user, which request, which order number. Something you can search for later.
| Level | Meaning | Example in a shop |
|---|---|---|
| Debug | Very detailed notes for developers | Cart total calculated with three items |
| Info | Normal events worth noting | Order 1042 placed successfully |
| Warning | Something odd but not fatal | Payment provider replied slowly |
| Error | Something failed | Could not save order 1043 to the database |
| Fatal | The program cannot continue | Cannot connect to the database, shutting down |
If your application writes its messages only into a file inside the container, and not to the standard pipes, kubectl logs will show nothing. The command can only read what Kubernetes captured from those two pipes.
4What kubectl logs Really Is
Let us define it in plain words, then list what it promises and what it does not.
kubectl logs is a read-only command that asks the cluster to print the messages that a container has written to standard output and standard error. You name the Pod (and the container inside it, if there is more than one), and the messages appear on your screen. It can print what already happened, and it can also keep the connection open so that new lines appear as they are written, like watching a live ticker.
Think of a security camera monitor in a shop’s back office. You can rewind and watch what was recorded earlier, or you can watch the live view as things happen. You cannot change what the camera recorded, and you can only see cameras you have been given access to. kubectl logs is that monitor for the messages of your containers.
What you get
- A quick view of what a container has said, from anywhere you can reach the cluster.
- No login to the node. You never have to connect to the machine itself.
- Live following of new lines, useful during a deployment or while reproducing a bug.
- Filters by time and count, so you can look at just the last few minutes or the last hundred lines.
- Permission control, since the cluster checks whether you are allowed to read the logs.
What it does not do
- It does not search across many Pods at once in a smart way. It is a viewer, not a search engine.
- It does not keep logs forever. Old lines are rotated away, and logs disappear when their Pod is deleted.
- It does not show messages the app wrote to a private file instead of the standard pipes.
- It does not explain what went wrong. It shows what the program said. You still have to read and think.
- It does not show events from Kubernetes itself, such as failed image downloads. Those are seen with a different command.
5Architecture and Components
Here is the whole picture in one drawing. Follow the arrows from you on the left to the log file on the right.
flowchart LR
USER["You run kubectl logs"] --> KC["kubectl"]
KC --> API["API Server"]
API -->|checks permission| RBAC["Access Rules"]
API --> KUB["Kubelet on the Pod's Node"]
KUB --> FILE["Log File on the Node"]
APP["Your App Container"] -->|prints to stdout and stderr| RT["Container Runtime"]
RT -->|writes each line| FILE
Reading the diagram
There are two separate stories here. The writing story is at the bottom: your application prints a line, the container runtime catches it and appends it to a file on the node. This happens all day, whether or not anyone is looking. The reading story is at the top: when you run the command, your request travels through the API server to the kubelet on the right node, which opens the file and sends the lines back the same way.
Meet each component
kubectl
Turns your command into a web request to the cluster. It also prints the results nicely on your screen.
API Server
Checks who you are and whether you may read the logs of this Pod, then forwards your request to the right node.
Kubelet
The node’s local manager. It knows where the Pod’s log files are and streams their contents when asked.
Container runtime
Catches what the container prints and stores it in a file, one line at a time, with a time stamp and a note saying which pipe it came from.
Log file
A plain text file kept under a folder for that Pod on the node. Kubernetes also keeps friendly shortcuts to these files so that logging tools can find them easily.
The container
The source of all the messages. It only needs to print. It does not know or care who reads them later.
6How It Works Internally: The Journey of a Log Request
What actually happens between pressing Enter and seeing lines on your screen? Let us follow the trip slowly.
sequenceDiagram
participant U as You
participant K as kubectl
participant A as API Server
participant L as Kubelet
participant F as Log File on Node
U->>K: Ask for the logs of a Pod
K->>A: Request the Pod log endpoint
A->>A: Check who you are and what you may do
A->>L: Forward the request to the right node
L->>F: Read the container log file
F-->>L: Log lines
L-->>A: Stream the lines back
A-->>K: Stream the lines back
K-->>U: Print the lines on the screen
kubectl builds a request
It reads your connection settings to learn which cluster to contact and which identity to use. It then prepares a request aimed at a special “log” address that belongs to the Pod you named.
The API server checks your identity
It confirms who you are. This may be a certificate, a token or a sign-in from your company’s login system.
The API server checks your permission
The rulebook is consulted. Reading logs counts as its own permission, called “get” on the Pod’s “log” part. If you lack it, you receive a refusal message and nothing else happens.
The API server finds the right node
It looks up which node the Pod runs on, and opens a secure connection to the kubelet on that node.
The kubelet reads the file
It finds the log file for that container and reads it, applying any limits you asked for, such as “last 50 lines” or “last 10 minutes”.
Lines travel back and appear
The lines flow back through the API server to kubectl, which prints them. If you asked to follow, the connection stays open, and new lines keep arriving as the container writes them.
Reading logs is like asking a hotel receptionist to fetch the guest diary from a particular room. The receptionist first checks your ID, then checks that you are allowed to read that room’s diary, then phones the floor manager (the kubelet) who walks to the room, picks up the diary and reads it out over the phone while the receptionist repeats it to you.
Because the request must reach the node, kubectl logs fails if the node is down or unreachable, even if the Pod’s record still exists in the cluster. In that case, the logs may be readable only from a central log store, if you have one.
7A Tour of the Most Useful Options
The command becomes much more powerful when you learn a handful of options. Here is a guided tour, in plain words.
| Option | What it does | When to use it |
|---|---|---|
-f or --follow | Keeps the connection open and prints new lines as they arrive | Watching a deployment or reproducing a bug live |
--tail | Shows only the last given number of lines | When the log is huge and you only care about recent lines |
--since | Shows lines from the last few seconds, minutes or hours | Looking at “the last 15 minutes” around an incident |
--since-time | Shows lines after an exact date and time | Starting from a known moment, such as a release time |
--timestamps | Adds a time to the front of every line | When the app’s own lines have no time on them |
-c or --container | Chooses one container in a multi-container Pod | Pods with sidecars or helper containers |
--all-containers | Prints logs of every container in the Pod | When you are not sure which container has the clue |
-p or --previous | Shows logs of the earlier run of a restarted container | Crashing containers, which is the classic use |
-l or --selector | Reads logs from all Pods that carry a given label | Looking across several replicas of one app |
--prefix | Adds the Pod and container name to each line | Telling apart lines from different Pods |
--limit-bytes | Stops after a given amount of data | Protecting your screen or connection from a flood |
-n or --namespace | Names the namespace of the Pod | Almost always, unless you use the default room |
Naming something bigger than a Pod
You do not have to look up a Pod’s long, machine-made name. You can point the command at a controller, such as a Deployment or a Job, and kubectl will pick one of its Pods for you. This is very convenient when you only want a quick look, but remember that you see only one Pod’s story. If your app has ten replicas, the message you need might be in one of the other nine.
Reading several Pods at once
When you use a label to select many Pods, kubectl shows only a short tail from each one by default, and it limits how many Pods it will follow at the same time. You can raise these limits, but at some point you are better off with a proper log search tool, which we cover later.
Think of the options as the buttons on a TV remote for recorded programs. “Follow” is live view. “Tail” is skipping to the end. “Since” is jumping to a time. “Previous” is playing the earlier recording that was saved before the channel restarted. You will use two or three of these buttons ninety percent of the time.
A good everyday combination is to ask for the last few minutes with time stamps and to follow the stream. That gives you fresh context first and then live updates, without drowning you in old lines.
8Pods with More Than One Container
Many real Pods hold more than one container. Knowing how to pick the right one saves a lot of confusion.
App container
The container doing the real job, such as running your web server.
Sidecar container
A helper running beside the main one, for example to handle secure connections, collect metrics or ship logs elsewhere.
Init container
A short-lived container that runs before the main one to prepare things, such as waiting for a database. If a Pod is stuck at startup, its logs are often the answer.
Ephemeral container
A temporary container added to a running Pod to investigate a problem. It has its own logs, too.
What happens if you do not choose
If a Pod has several containers and you do not name one, kubectl either picks a default container and tells you which it chose, or asks you to choose from a list. That short message at the top of the output is easy to miss, and beginners sometimes read the wrong container’s log for an hour. Always glance at which container the output belongs to.
Startup problems live in init containers
If a Pod shows a status like “Init” followed by numbers, it is still working through its init containers, and the main container has not started at all. Asking for the main container’s logs will show nothing useful, because it has not started yet. You must ask for the logs of the init container that is stuck.
A restaurant kitchen has several staff members working at once: a chef, an assistant and a dishwasher. Each writes their own notes. If you want to know why the soup is late, reading the dishwasher’s notes will not help. You need to choose whose notes to read.
The status column of the Pod list tells you which stage the Pod is at. Learn to glance at it before you decide whose logs to read.
9Reading Logs of Crashed and Restarting Containers
The moment you most need logs is often the moment the container has just died. Kubernetes has a special way to help.
The restart loop
When a container crashes, Kubernetes usually restarts it. If it crashes again and again, Kubernetes waits a little longer between each attempt. The Pod’s status then shows CrashLoopBackOff, which is simply the cluster’s way of saying “I keep trying, but it keeps failing, so I am slowing down”.
The current container may be only a few seconds old and may have said nothing yet. The clue is in the container that just died. That is what the “previous” option is for. It shows the log of the last finished run of the container, which is where the final words before the crash were written.
What each status tells you about logs
| Pod status | Meaning | Will logs help? |
|---|---|---|
| Pending | No node has been chosen yet | No. The container has not started. Look at the Pod’s events instead. |
| ContainerCreating | The node is preparing the container | No. Look at events for storage or image problems. |
| ImagePullBackOff | The container image could not be downloaded | No. The program never ran. Events name the cause. |
| Init and a number | An init container is still running or failing | Yes, but ask for the init container’s logs. |
| CrashLoopBackOff | The container keeps starting and dying | Yes. Use the previous-run option to see the dying words. |
| Running but unhealthy | The container is alive but misbehaving | Yes. Follow the logs while you reproduce the problem. |
| Completed | A one-time job finished | Yes. The logs show what the job did. |
Imagine a factory machine that keeps switching itself off. When you walk up, it has just restarted and its display is blank. The useful information is on the printout it produced right before it switched off. The “previous” option is asking for that last printout instead of the blank screen.
Kubernetes keeps the log of only the most recent finished run. After several more crashes, earlier runs are gone. If a problem happened hours ago, you may only see the newest crash, or nothing at all if the Pod was replaced. This is why many teams also send logs to a central place.
10Data Flow and Lifecycle: How Long Do Logs Live?
A log line has a life story. It is born when the app prints it, and it can die in several ways. Knowing them explains why lines sometimes vanish.
flowchart LR
APP["App prints a line"] --> RT["Container runtime writes it to a file"]
RT --> ROT["Kubelet rotates old files"]
RT --> AG["Log agent on the node reads the file"]
AG --> STORE["Central log store"]
STORE --> UI["Search and dashboards"]
ROT -. old lines removed .-> GONE["Lines gone from the node"]
Birth
The application prints a line to standard output or standard error. The container runtime catches it immediately.
Written to a file
The runtime appends the line, with a time stamp, to a file on the node. From this moment, kubectl logs can read it.
Rotation
The file cannot grow forever, or it would fill the node’s disk. When it reaches a size limit, which is about ten megabytes by default, the kubelet starts a new file and keeps only a small number of the older ones. The oldest lines are dropped.
Optional copy elsewhere
If the cluster has a log collector, it reads the file as it grows and forwards each line to a central store. This copy survives even when the original disappears.
Death on the node
When a Pod is deleted, or the node is removed, its log files are removed with it. Kubernetes also cleans up files of old, finished containers to save space.
Think of the recording box of a shop’s security camera. It has a small hard disk that holds the last few days. When the disk is full, the oldest video is recorded over. If the box is thrown away, all the video goes with it. A company that wants to keep footage for a year sends copies to a separate archive every night. Log rotation is the recording over. The log collector is the nightly archive copy.
After a rotation, kubectl logs shows only the newest file. Lines that were in older files are not returned, even though they existed a short while ago. A busy app that prints a lot can push its own earlier lines out within minutes.
Logs also cost disk space
Container logs sit on the node’s disk and count toward the Pod’s temporary storage. A program that prints too much can fill a disk and cause the node to become unhealthy. Kubernetes may then evict Pods to protect itself. So there is a real cost to noisy logs, beyond just being hard to read.
11Where kubectl logs Runs Out of Power
The command is excellent for quick looks. It is not a complete logging system, and it helps to know exactly where it stops.
It is great at
- Checking why one Pod is unhappy right now.
- Watching a deployment start up.
- Reading the last run of a crashing container.
- Quick checks during development and testing.
- Needing no extra software at all.
It struggles with
- Searching for one order number across fifty Pods.
- Looking at what happened last week.
- Pods or nodes that no longer exist.
- Combining logs from several services into one story.
- Making charts or alerts from log content.
Which tool for which question?
| Question | Best tool | Why |
|---|---|---|
| What did this app say a minute ago? | kubectl logs | Fast, simple, no setup |
| Why could the Pod not even start? | Pod description and events | Startup failures are reported by Kubernetes, not the app |
| What is inside this container right now? | Running a shell inside it, or a debug container | You can look at files and settings directly |
| How many errors happened this week? | Central log search | Long storage and counting |
| How slow is my service over time? | Metrics | Numbers are built for trends |
| Where did one request spend its time? | Tracing | Follows a request across services |
| I want colored logs from many Pods at once | A community tool such as stern | Tails many Pods with color and one command |
A pocket flashlight is perfect for looking under the sofa. It is the wrong tool to inspect a whole warehouse. In the same way, kubectl logs is your flashlight. When the question becomes “everything, everywhere, for the last month”, you need the warehouse floodlights: a central logging system.
Kubernetes events, which are short notes about what the cluster itself did, are also kept only for a short time, about an hour by default. Look at them soon after a problem, or capture them into your logging system.
12Building a Longer Memory: Central Logging Patterns
Kubernetes does not ship with a permanent log store. Instead, it gives you clear places to plug one in.
flowchart TB
subgraph A["Pattern 1: Node agent"]
A1["App Pod"] --> A2["Log file on node"] --> A3["Agent on every node"] --> A4["Central store"]
end
subgraph B["Pattern 2: Sidecar"]
B1["App writes to a file"] --> B2["Sidecar prints the file to stdout"] --> B3["Node agent"]
end
subgraph C["Pattern 3: Direct push"]
C1["App sends logs itself"] --> C2["Central store"]
end
Node agent
A small collector program runs on every node, usually started by a DaemonSet, which is a Kubernetes object that runs one copy per node. It reads all the container log files on that node and forwards them. Your apps need no changes.
Sidecar
If an application insists on writing to a file, add a small helper container in the same Pod that reads the file and prints it to its own standard output. Now Kubernetes can capture it, and kubectl logs works again for that helper.
Direct push
The app sends its logs straight to the store using a library. It skips the node, but it ties your code to the logging system, and if the store is down the app must cope. kubectl logs shows nothing.
Central store
A searchable home for logs, such as Grafana Loki, Elasticsearch or OpenSearch, or a cloud service. It keeps lines for weeks or months and lets teams search, chart and alert.
Popular collectors
Two well-known open-source collectors are Fluent Bit and Fluentd, both part of the Cloud Native Computing Foundation family. Fluent Bit is lighter and is often chosen to run on every node. Cloud providers also offer their own: Google Kubernetes Engine can send container logs to Google Cloud Logging automatically, and Amazon and Microsoft provide similar options through CloudWatch and Azure Monitor.
A node agent is like a mail carrier who walks every street in a neighborhood every few minutes, picks up letters from each mailbox and brings them to the central post office. The houses (your apps) never need to leave home. A sidecar is a neighbor who reads out a resident’s handwritten diary and drops the pages into the mailbox. Direct push is when the resident walks to the post office themselves.
The best setups keep both worlds: the node agent copies logs to the central store, and the files on the node stay in place for a short time so that kubectl logs still works for quick checks.
13Writing Logs Worth Reading
The command can only show what your app wrote. A little care when writing messages pays off a hundred times when you read them at night.
Use a consistent shape
Many teams write each line as a small structured record, often in a format called JSON, with fields for time, level, message and context. Machines can then search and count them easily.
Add a request identifier
Give each incoming request a short unique id and include it in every line about that request. Then you can follow one customer’s journey across many lines and many services.
Choose levels honestly
If everything is marked as an error, then nothing stands out. Reserve the error level for real failures.
Do not print too much
Logging every tiny step floods disks and hides important lines. Keep detailed debug lines switched off by default and switch them on only when investigating.
Say what and why
“Failed” is a poor message. “Could not save order 1043 because the database timed out after 5 seconds” is a good one.
Never log secrets
Passwords, access tokens, card numbers and personal details must never be printed. Logs are copied and stored in many places, and lots of people can read them.
A good log is like a good ship captain’s diary. It has a date, a short clear entry and enough detail that a stranger could understand what happened. A bad log is like a diary that says “stuff happened” on every page, or one that writes down every breath the captain takes. Neither one helps in an emergency.
14Advantages, Disadvantages and Trade-offs
Every tool has strengths and costs. Knowing both helps you use it wisely.
Advantages
- Built into kubectl, so it works on nearly every cluster.
- No login to the node is needed.
- Quick to learn: a handful of options cover most cases.
- Live following makes debugging interactive.
- Access is controlled by the same permission system as everything else.
- Works with old and new apps, as long as they print to the standard pipes.
Disadvantages
- Short memory: rotation and Pod deletion remove lines.
- One Pod at a time is the natural view.
- No search, no charts and no alerts.
- Depends on the node being reachable.
- Does not capture messages written to private files.
- Heavy use can put load on the API server and kubelet.
The trade-offs behind the choices
Simplicity versus power. Printing to the standard pipes is so simple that anyone can do it, but the price is that you must add your own tools for search and long storage.
Detail versus cost. Very detailed logs help debugging but fill disks, slow searches and increase storage bills. Terse logs are cheap but may hide the very clue you need. A common compromise is to log at a modest level normally and raise it only when needed.
Node files versus central storage. Node files are immediate and free but temporary. Central stores are durable and searchable but take money and care to run.
Access versus privacy. The more freely people can read logs, the faster they solve problems, but the greater the risk that sensitive data is seen by the wrong person.
15Security: Who May Read the Diary?
Logs often contain surprising amounts of private detail. Guard access to them as you would guard the shop’s records.
Grant log access on purpose
Reading logs is a separate permission on the Pod’s log part. Give it only to people and tools that need it, and limit it to specific namespaces rather than the whole cluster.
Keep secrets out of logs
Configure applications and libraries not to print passwords, tokens, keys, card numbers or personal details. Mask them if they must appear at all.
Protect the path to the node
The API server talks to each kubelet over a secure connection. Configure the API server to verify the kubelet’s certificate so that nobody can pose as a node.
Protect the central store
Encrypt stored logs, restrict who can search them, and set retention periods so that old data is deleted when it is no longer needed.
Audit who reads what
The cluster’s audit records can show who asked for logs of which Pod and when. Reviewing them helps you spot misuse.
Beware of log injection
If user input is written into logs without care, a bad actor can insert fake lines that mislead investigators. Clean or encode user input before logging it.
Granting a broad “read everything in the cluster” role to a helper tool or a whole team can quietly give access to every Pod’s logs, including those of payment or identity services. Check who holds such roles.
A hospital keeps patient notes in a records room with a locked door, a sign-in sheet and rules on who may read which files. Logs deserve the same seriousness because they can quietly contain names, emails and other private facts.
16Monitoring the Logging System Itself
Logs help you watch your apps. But somebody must also watch the logging system, or it may fail silently.
Volume
Track how many lines and how many bytes each app produces. A sudden jump may mean a bug printing in a loop, and it will soon fill disks and raise costs.
Disk pressure on nodes
Watch the free space on each node. Logs are a common reason for sudden disk trouble and for Pods being evicted.
Collector health
Make sure each node’s log agent is running, keeping up, and not dropping lines. A broken collector means the central store quietly stops receiving data.
Error rates from logs
Turn counts of error lines into charts and alarms so that you learn about problems before customers report them.
Restarts
Watch how many times each container restarts. A high count is a signal to go and read the previous run’s logs.
Alarms worth setting
- A container restarting repeatedly within a short window.
- A log agent that is down on any node.
- A jump in log volume of several times the normal amount.
- Node disk usage approaching a dangerous level.
- A steady rise in the number of error-level lines for a service.
- No logs arriving at all from a service that should be busy. Silence can be the loudest warning.
Prometheus with Grafana is a popular pair for cluster numbers such as restarts and disk usage. Pairing them with a log store lets you jump from a spike on a chart straight to the log lines from that minute.
17Design Patterns and Anti-patterns
Patterns are solutions that many teams found reliable. Anti-patterns are traps many teams fell into.
Patterns that work
Pattern: Log to standard output, collect at the node
Apps only print. A node agent gathers everything. This keeps application code simple and moves all logging decisions to the platform.
Pattern: Structured logs with request identifiers
Every line is a structured record that includes a request id. Searching for one id reveals a whole customer journey across services.
Pattern: Sidecar for legacy file-based apps
A small helper container prints the app’s log file to its own standard output, making old software work with the standard tools without rewriting it.
Pattern: Short local memory, long central memory
Let nodes keep only recent files for quick command-line checks, and rely on the central store for anything older.
Pattern: Log levels controlled by settings
Apps read their log level from a setting, so operators can raise detail during an incident without rebuilding the app.
Pattern: Labels that make logs easy to find
Give Pods consistent labels such as app name and version. Then a single label selection in the command, or in the log store, finds every Pod of a service.
Traps to avoid
What it looks like
Writing logs only into a file inside the container.
Why it hurts
The command shows nothing, agents cannot see the file, and the file vanishes when the container is replaced.
Better approach
Print to standard output and standard error, or add a sidecar that does it.
What it looks like
Relying on kubectl logs as the only log system for production.
Why it hurts
After a crash, an eviction or a deployment, the evidence is gone. You cannot investigate anything that happened before the last few minutes.
Better approach
Send logs to a central store from day one, and keep the command for quick checks.
What it looks like
Printing secrets, full request bodies or personal details “just for debugging”.
Why it hurts
Sensitive data gets copied into files, backups and search tools where many people can read it. Leaks and compliance problems follow.
Better approach
Log identifiers and outcomes rather than raw private data, and mask anything sensitive.
What it looks like
Running with debug-level logging permanently in production.
Why it hurts
Disks fill, searches slow down, bills rise, and real errors are lost in the noise.
Better approach
Keep a moderate level by default and raise it temporarily only while investigating.
18Best Practices and Common Mistakes
Small habits make the difference between a calm investigation and a frantic one.
Habits that pay off
- Check the Pod status first. It tells you whether logs even exist yet, and which container to ask about.
- Always name the namespace. Many “no such Pod” confusions are simply looking in the wrong room.
- Use the previous-run option for restarts. It shows the dying words of a crashed container.
- Narrow by time. Ask for the last few minutes around the problem rather than reading the whole log.
- Add time stamps. They let you match log lines to charts and events.
- Send logs to a central store. Do it before you need it, not after the incident.
- Set rotation limits on purpose. Make sure files are large enough to hold useful history but small enough to protect the disk.
- Read events as well as logs. Many startup problems appear only in events.
- Review log content regularly. Look for secrets, personal data and useless noise.
Mistakes beginners make
Reading the wrong container
In a Pod with helpers, the default container may not be the one with the clue. Always check which one you are reading.
Asking a Pending Pod for logs
A Pod that has not started has no logs. Look at its events instead to learn why it is stuck.
Forgetting the previous option
Looking at the fresh, silent container of a crash loop and concluding “there is nothing in the logs”.
Trusting one replica
Naming a Deployment shows only one Pod. The error may be in another replica.
Assuming logs last forever
Rotation and Pod replacement erase history quickly. If it matters, copy it out early.
Permission errors misread as bugs
A “forbidden” message means you lack the log permission, not that the app is broken. Ask an administrator for access.
19Real-World and Industry Examples
Here is how the ideas in this guide show up in everyday practice.
The Twelve-Factor App
The Twelve-Factor guidelines, first written by developers at Heroku, describe logs as a stream of events that the app should simply print. Kubernetes follows this idea closely, which is why printing to standard output is the recommended way to log in a cluster.
Managed clouds collect logs for you
Google Kubernetes Engine can send container logs to Google Cloud Logging without extra work in your apps. Amazon EKS users often use Fluent Bit with CloudWatch, and Azure Kubernetes Service offers logging through Azure Monitor. In each case, the container only prints, and the platform does the collecting.
Open-source logging stacks
Many teams run their own stack. Fluent Bit or Fluentd runs on every node, and the lines are stored in Grafana Loki, Elasticsearch or OpenSearch, then searched through dashboards. Because these tools are open source, they are used by companies of every size.
The community tail tool
The open-source tool called stern was created to make it easier to follow logs from many Pods at once, with a color for each Pod. It shows how developers extend the basic command when they outgrow it, and it sits on top of the same log endpoint that kubectl logs uses.
A typical online shop
Picture a store on a managed cluster. Every service prints structured lines that include an order id. A node agent forwards the lines to a central store, where support staff can search for one customer’s order. During a release, the deploying engineer follows the new Pods with the command to confirm a clean start. When a payment service crashes, the on-call engineer reads the previous run to find the cause in seconds, then uses the central store to see how many customers were affected.
Batch jobs
A nightly job that builds reports runs in a Pod that finishes and stops. Its logs are the only record of what it did. Teams keep the finished Pods around briefly and also ship the lines to storage so the history survives.
20Frequently Asked Questions
Quick answers to the questions beginners ask most often.
21Summary and Key Takeaways
You began with a chef behind a closed door. Now you know exactly where the little window is and how to use it well.
Key Takeaways
- What it is:
kubectl logsis a read-only command that prints what a container wrote to standard output and standard error. - How it works: The container runtime writes each line to a file on the node, and your request goes through the API server to the kubelet, which reads the file and streams it back.
- Most useful options: Follow for live lines, tail and since to narrow the range, container to pick one in a multi-container Pod, and previous to read a crashed run.
- Know the status: Pending and image-pull problems have no logs and are explained by events, while crash loops and running Pods do have logs.
- Its memory is short: Files rotate, and logs vanish when Pods or nodes are removed, so send logs to a central store for anything important.
- Write logs well: Print to standard output, use structured lines with levels and request ids, keep noise low, and never print secrets.
- Run it safely: Grant log permission narrowly, protect the path to the nodes, secure and limit the central store, and monitor log volume, disk space and the health of the collectors.