Before opening the Corefile, we need to understand the problem DNS solves, and why the ordinary internet version is not enough inside a cluster.

What DNS is, in plain words

Computers talk to each other using numbers called IP addresses, such as 10.96.14.7. People prefer names, such as “orders” or “example.com”. The Domain Name System, or DNS, is the worldwide phone book that turns names into numbers. When your application wants to talk to “example.com”, a small piece of code inside it, called the resolver, sends a question to a DNS server: “What is the address for this name?” The server replies with an answer and a number saying how long the answer may be remembered, called the TTL, short for time to live.

Why Pods cannot just use IP addresses

In Kubernetes, your application runs in Pods, the smallest units of work, each with its own IP address. But Pods are temporary. They are created, destroyed and replaced all the time, during updates, failures, and scaling. Every new Pod gets a different IP address. If the “checkout” service saved the IP of the “payments” Pod yesterday, that address might belong to nothing, or to a different application, today.

Kubernetes solves part of this with a Service, a stable front door with a fixed virtual IP, called the ClusterIP, which spreads traffic across the Pods behind it. But a fixed IP is still a number to remember, and it changes if the Service is recreated or moved to another environment. What you really want is a stable name. Cluster DNS gives every Service a name that always works, in every environment, from every Pod.

Where CoreDNS came from

The first Kubernetes DNS add-on was called kube-dns. It was built from several programs working together, including a general-purpose DNS cache called dnsmasq. It worked, but it was hard to extend and had memory and performance quirks. The community then adopted CoreDNS, a DNS server written in Go that is built entirely from small plugins chained together. CoreDNS became the default in Kubernetes around version 1.13 and is now the standard on nearly every distribution. For compatibility, the Service in front of it is still named “kube-dns”, which surprises many newcomers.

A short glossary makes every later chapter easier to read.

The anatomy of a cluster name

Every Service in the cluster has a full name built from four parts, read from the most specific to the most general: the Service name, the namespace it lives in, the word svc, and the cluster domain, which is cluster.local by default. So a Service called “orders” in the “shop” namespace has the full name orders.shop.svc.cluster.local. A name that is spelled out completely like this is called a fully qualified domain name, or FQDN.

You rarely type the full form. Inside the “shop” namespace you can just say “orders”. From another namespace you can say orders.shop. The shorter forms work because of the search domains that Kubernetes places inside every Pod, which we examine in Chapter 7.

Cluster DNS is not one magic box. It is a small set of ordinary Kubernetes objects working together.

flowchart LR
  APP["Application in a Pod"] --> RES["Resolver library reads resolv.conf"]
  RES -->|query to kube-dns ClusterIP| SVC["kube-dns Service"]
  SVC -->|load balanced by kube-proxy| CD["CoreDNS Pod replicas"]
  CD --> KP["kubernetes plugin: in memory view of Services and EndpointSlices"]
  CD --> FW["forward plugin: upstream resolvers"]
  API["API server"] -->|list and watch| KP
  FW --> UP["Node or VPC resolver and the wider internet"]
    

Read the diagram from left to right. An application asks its resolver a question. The resolver reads a small file, /etc/resolv.conf, to learn where the nameserver is. That address is the virtual IP of the kube-dns Service. Behind the Service run several CoreDNS Pods. Inside each, the kubernetes plugin holds a live picture of the cluster, built by watching the API server, and the forward plugin knows how to pass other questions to outside resolvers.

Why the Service is called kube-dns

Older programs and tools look for a Service named “kube-dns” in the system namespace, so the name was kept when CoreDNS replaced the old implementation. If you run kubectl get svc -n kube-system and see kube-dns, do not worry, it is front-ending CoreDNS. The Pods carry a label with the same old name for the same reason.

Two layers of high availability

CoreDNS Pods are normal Pods and can be rescheduled, and the Service in front of them spreads queries between the healthy ones. So DNS survives the loss of a Pod or a node. But it is only as strong as the replica count and placement. A cluster with a single CoreDNS Pod has a single point of failure, and a cluster with two replicas that both land on one node is nearly as fragile. We return to this in the reliability chapter.

Let us follow a single lookup, for the name “orders”, from the moment the application asks to the moment it gets an address.

How CoreDNS knows the answer without asking anyone

The key to speed is that CoreDNS does not query the API server for every lookup. The kubernetes plugin keeps a live in-memory copy of all Services, endpoint slices and namespaces, kept fresh by the same list-and-watch pattern used by other Kubernetes components. When you create a Service, the API server announces it, the plugin updates its memory within a moment, and the next query for that name succeeds. A lookup is then just a quick search in memory, which takes microseconds.

Cluster names versus outside names

CoreDNS sorts questions by zone, which means the part of the name tree it is responsible for. Names ending in cluster.local belong to the kubernetes plugin. Anything else, such as example.com or a cloud database endpoint, is passed to the forward plugin, which asks outside resolvers, usually the ones the node itself uses.

Different kinds of Kubernetes objects produce different DNS answers. Knowing the pattern lets you predict any name.

Normal Services: one stable address

For a typical Service, the answer is the ClusterIP, a virtual address that never changes during the Service’s life. The client connects to that address, and kube-proxy on the node spreads the connection across real Pods. The DNS answer does not change when Pods come and go, which is why DNS caching for ordinary Services is safe and cheap.

Headless Services: the directory, not the switchboard

A headless Service is one created without a ClusterIP. For it, DNS returns the addresses of the actual Pods, one A record per ready Pod. The client sees the list and can choose, or connect to all. This is how databases and message brokers let their members find each other. Because the list changes whenever Pods change, TTLs matter much more here, and caching clients can hold stale answers.

By default, only Pods that are ready appear in these answers. Some systems, especially during start-up of a clustered database, need peers to find each other before any of them are ready. For those, the Service can be told to publish not-ready addresses as well.

StatefulSets and stable identities

A StatefulSet gives each Pod a permanent name with a number, like db-0, db-1, db-2. Paired with a headless Service, each Pod also gets its own DNS name, such as db-0.db.shop.svc.cluster.local. Even if db-0 is rebuilt on another node with a new IP, the name stays the same and the DNS record is updated. This is why databases and clustered systems depend on StatefulSets.

SRV records: names and ports together

When a Service port has a name, CoreDNS also creates an SRV record. An SRV answer carries both the host name and the port number, so a client does not need to know the port in advance. This is useful for service discovery in tools that look up ports dynamically.

ExternalName: an alias to the outside world

A Service of type ExternalName has no Pods and no virtual IP. Instead, DNS answers with a CNAME alias pointing to an outside name, for example a managed database hostname. Your applications use the stable in-cluster name, and you can change the outside target in one place. Note that it is purely a DNS trick. No traffic is proxied, and if the target needs special certificate names, they are checked against the outside name.

Pod records

CoreDNS can also answer for individual Pods using a name made from the Pod’s IP with dashes instead of dots. This is controlled by a setting in the kubernetes plugin. It is rarely needed by applications and is more commonly turned on for compatibility with older tools.

Everything CoreDNS does is a plugin, and the Corefile is the instruction sheet that turns plugins on and tunes them.

flowchart TD
  Q["Query arrives at CoreDNS"] --> E["errors, prometheus, health: housekeeping"]
  E --> CA{"cache has a fresh answer?"}
  CA -->|yes| A1["Reply from cache"]
  CA -->|no| K{"kubernetes plugin owns this zone?"}
  K -->|yes| A2["Answer from in memory Services and Pods"]
  K -->|no| FW["forward plugin asks upstream resolvers"]
  A2 --> ST["cache stores the answer"]
  FW --> ST
  ST --> A3["Reply to client"]
    

Read the diagram from the top. A query enters and passes through housekeeping plugins that record errors and metrics. The cache is checked first; a fresh answer is returned at once. Otherwise, if the name belongs to the cluster zone, the kubernetes plugin answers from memory. If not, the forward plugin asks an outside resolver. Either way, the new answer is stored in the cache before the reply goes back.

How the file is organized

A Corefile is made of server blocks. Each block names a zone and a port, such as “everything, on port 53”, and lists the plugins that apply to it. The standard Kubernetes Corefile has one server block for all names. You can add more blocks to send particular domains, like your company’s internal domain, to special servers. Each plugin inside a block can carry its own settings.

The plugins in a default setup

Customizing: the common recipes

Half of all DNS problems in Kubernetes are not caused by CoreDNS at all. They come from what the Pod is told to do before the query is even sent.

What is inside a Pod’s resolver file

When the kubelet creates a Pod, it writes a small text file into it called /etc/resolv.conf. A typical file contains three kinds of lines. A nameserver line holds the address of the cluster DNS Service. A search line lists suffixes to try on short names, usually three: the Pod’s own namespace followed by svc.cluster.local, then svc.cluster.local, then cluster.local. Your cloud’s own search domains are often added after those. An options line contains ndots:5.

dnsPolicy: choosing how a Pod gets this file

dnsConfig: small adjustments

With the dnsConfig field you can add extra nameservers, extra search domains, or options like a lower ndots, either on top of the default policy or as a complete replacement under None. Be aware of limits: the common C library only honours three nameservers, and Kubernetes warns when the search list exceeds six entries or 256 characters in total.

The ndots trap, step by step

The ndots option says: “If a name has fewer than this many dots, treat it as a short name and try the search suffixes first.” With the default of 5, almost every ordinary name counts as short. The name example.com has one dot. api.payments.example.com has three. Both are below five. So the resolver walks the whole search list before trying the name as written.

flowchart TD
  N["App looks up example.com"] --> D{"Fewer dots than ndots 5?"}
  D -->|yes| S1["Try example.com.default.svc.cluster.local: NXDOMAIN"]
  S1 --> S2["Try example.com.svc.cluster.local: NXDOMAIN"]
  S2 --> S3["Try example.com.cluster.local: NXDOMAIN"]
  S3 --> S4["Try node search domains: NXDOMAIN"]
  S4 --> S5["Finally try example.com as written: success"]
  D -->|no| S5
    

Follow the diagram. The application only wanted one answer. But the resolver first asks for example.com with the Pod’s namespace suffix, gets “does not exist”, tries the next suffix, gets “does not exist” again, and so on, finally succeeding on the original name. And because most resolvers ask for both IPv4 and IPv6 addresses, every one of those attempts is doubled. A single external lookup can therefore turn into roughly ten packets, of which about eight are wasted.

Why the default is five

The value exists so that short, natural names like orders.shop.svc (two dots) get completed by the search list. Cluster names can have up to four dots in their usual forms, so five ensures that any partially written cluster name is completed before an outside lookup is tried. It is a good default for cluster-internal traffic and a costly one for traffic that goes outside.

Ways to tame it

This plugin is the heart of cluster DNS. Understanding its settings explains most of the behaviour you will see.

How it stays up to date

When CoreDNS starts, the kubernetes plugin asks the API server for a full list of Services, endpoint slices and namespaces, then opens a watch to hear about changes. It builds in-memory tables indexed by name. This is the same list-and-watch pattern used across Kubernetes. If the watch drops, it reconnects and refreshes; if the API server is unreachable for a while, CoreDNS keeps answering from the last copy it has, so brief control-plane trouble does not break existing lookups.

Notice the cost of this design: each CoreDNS replica holds a copy of the whole Service and endpoint data of the cluster, so memory use grows with the number of Services and Pods, and each replica holds its own watch on the API server.

Endpoint slices, not whole Endpoints

For headless Services and for SRV records, CoreDNS needs the list of Pod addresses behind a Service. It reads these from endpoint slices, which are small groups of endpoint addresses that Kubernetes keeps for each Service. Slices keep the data compact and make updates cheaper than the older single-record form, which matters for Services with thousands of Pods.

The settings that matter

Freshness: how quickly DNS follows reality

There is a short delay between a change in the cluster and the matching DNS change. The API server must announce the change, the plugin must update its memory, and any copy already cached in CoreDNS or in clients must expire. CoreDNS publishes a metric for the first part, often called the DNS programming latency, measuring how long it takes from the moment a change happens to the moment DNS reflects it. In a healthy cluster the figure is small, typically well under a second, but it rises when the API server or the nodes are overloaded.

For every name outside the cluster, CoreDNS becomes a middle-man. How well it plays that role decides most of your real-world latency.

The forward plugin

The forward plugin sends questions it cannot answer to one or more upstream resolvers. By default these are the servers listed in the CoreDNS Pod’s own resolver file, which comes from the node. On a cloud network that is usually the cloud provider’s built-in resolver for the private network. You can instead name specific servers, send some domains to your own corporate servers, and choose how to spread load among several upstreams: randomly, in turn, or always trying the first before falling back.

The plugin also checks the health of upstreams. If one stops responding, it is skipped for a while. And a setting caps the number of queries in flight at once, so a slow upstream cannot cause CoreDNS to pile up an unbounded backlog of waiting queries.

Loops: when DNS asks itself

A forwarding loop is a classic trap. Suppose the node’s resolver file points at a local helper on the node, a common arrangement on some Linux distributions. CoreDNS copies that setting, forwards a query to it, and the helper hands the query back to the cluster DNS because it was configured that way. The query circles forever. The loop plugin sends itself a test query at start-up, notices the echo, and shuts CoreDNS down with a clear message rather than burning CPU. The usual fix is to give the kubelet a different resolver file that lists the real upstream servers.

The cache plugin

The cache plugin remembers answers, both positive (“here is the address”) and negative (“that name does not exist”). It keeps each answer for the time it was told by the original source, capped by a maximum you set, often thirty seconds. Lower caps keep data fresher; higher caps reduce load.

Caches outside CoreDNS

CoreDNS is not the only place answers are remembered. Applications and their runtimes have their own habits. Many do not cache at all and ask DNS for every new connection. Some, such as certain Java configurations, cache answers for a long time or even forever, ignoring the TTL, which can leave an application talking to an old address after a change. Connection pools and proxies often resolve a name once and keep using it. When “DNS changes are not picked up,” the culprit is usually one of these client-side caches, not CoreDNS.

When one cluster-wide DNS service is not enough, the next step is to put a small cache on every node.

flowchart LR
  P["Pod"] -->|UDP to link local address| NL["NodeLocal DNS cache on the same node"]
  NL -->|hit| P
  NL -->|miss, TCP| CD["CoreDNS Pods"]
  NL -->|external names, miss| UP["Upstream resolver"]
  CD --> UP
    

How it works

NodeLocal DNSCache is a small DNS cache that runs as a DaemonSet, which means one copy on every node. Each copy listens on a special local address that exists only on its own node. Pods are pointed at that address instead of the cluster-wide Service. Questions that the node-local cache has answered recently never leave the machine. Only cache misses travel on to CoreDNS, and when they do, the cache uses TCP instead of UDP.

The problems it solves

DNS is a snapshot of a moving target. Seeing how the snapshot is refreshed explains many odd moments during deployments.

A rolling update, as DNS sees it

This shows why ordinary Services are more forgiving than headless ones: the DNS answer never changes, and the changing part is handled by routing rules on each node. With headless Services, the client uses real Pod addresses, so every cache between CoreDNS and the client matters.

When CoreDNS itself is updated

CoreDNS Pods also come and go. During a rolling update, a Pod being removed should finish in-flight queries and stop receiving new ones. The health plugin supports a short lameduck period: for several seconds after shutdown begins, the Pod keeps answering while the rest of the cluster learns it is leaving. Without it, clients may send queries to a Pod that no longer exists, and because UDP routes are tracked on the node for a time, some packets may be lost until stale entries expire.

Where the delays come from

Startup order and readiness

A new Pod’s first lookup happens as soon as its code starts. If CoreDNS is not ready, or the Pod has not yet learned about a Service, that first lookup can fail. Healthy applications retry with a short delay instead of giving up, and init steps in deployments sometimes wait for dependencies by name. A well-behaved cluster does not assume DNS is perfect at the first millisecond.

CoreDNS itself is fast. The challenge is the volume and shape of the traffic sent to it, and the limits of what lies beyond it.

Where the load comes from

Understanding the AWS packet limit

Inside an Amazon VPC, DNS queries to the Amazon-provided resolver are limited per network interface. A node that runs a CoreDNS Pod and forwards many external lookups can reach this limit without being busy at all, because the ndots multiplication creates so many small packets. When the limit is hit, packets are dropped, and applications see random timeouts or five-second delays. The symptoms look like a flaky CoreDNS, but the problem is the interface limit. The usual remedies are spreading CoreDNS Pods over more nodes, caching on the node with NodeLocal DNSCache, lowering ndots, and using trailing dots.

Sizing CoreDNS

DNS is a dependency of almost everything. When it fails, the symptoms appear everywhere except in DNS itself.

What happens when CoreDNS is down

The first thing to understand is the pattern of failure. Connections that are already open keep working, because they use IP addresses. Lookups answered from a cache keep working until the cache entries expire. But any new lookup that misses a cache fails. So an outage starts slowly, with a trickle of errors, then grows as caches expire, and reaches full strength after a minute or so. By then, applications are failing to start, retry storms are growing, and engineers are looking at the wrong systems.

Resolver timeouts make it worse

The default resolver in most Linux systems waits about five seconds for each attempt and tries twice. A single failed lookup can therefore stall an application for ten seconds or more before returning an error, and when each name in a long search list fails in turn, the delays add up. Slow, silent hangs are the typical user experience of a DNS problem, not clean errors. Lowering the timeout and attempts in the Pod’s options can turn a long hang into a quick failure that retry logic can handle.

DNS is both a map of your cluster and a channel that almost every firewall lets through. Treat it with respect.

Trust in the answers

By default, CoreDNS does not validate the cryptographic signatures that some domains use to prove their answers are genuine. It trusts what its upstream resolvers say. Within the cluster, this trust is extended to the network: a malicious Pod with the right network position could try to forge replies. Keeping Pods apart with network policy, encrypting service-to-service traffic with certificates that verify names, and not treating DNS alone as proof of identity are the true defences.

Logging with care

Query logs are a powerful forensic tool, because they show which Pod looked up which name. They also reveal sensitive information, such as the names of customers or internal systems, and they can be huge. Turn logging on for a short time to investigate, or send it to a protected store with a clear retention period.

CoreDNS is easy to observe if you know which numbers to read. Most DNS mysteries can be solved in a few minutes with the right ones.

Metrics worth graphing

Symptoms and likely causes

A troubleshooting routine

Operators of large clusters keep repeating the same advice. Here it is, in one place.

Patterns worth copying

Traps to avoid

Best practices at a glance

Common mistakes and quick fixes

The same few patterns show up in every large environment. Here are the places you are most likely to meet them.

A typical journey at a growing company

A team starts with the default CoreDNS and two replicas. As the cluster grows from fifty to five hundred nodes, intermittent five-second delays appear. After weeks of searching, they discover that applications are sending several wasted queries per lookup through the search list, and that the cloud resolver is dropping packets on busy nodes. They add trailing dots to external hostnames, set ndots to a lower value for chatty workloads, deploy a node-local cache, and let CoreDNS scale with the node count. Metrics for NXDOMAIN rate and upstream latency go on a dashboard, with alerts for errors. The delays disappear, and DNS becomes quiet and boring, which is exactly what a good phone book should be.

Cluster DNS turns the shifting world of Pods into stable names that applications can rely on. CoreDNS does this with a chain of small plugins: the kubernetes plugin answers cluster names from a live in-memory picture of Services and endpoint slices, the cache remembers answers, and the forward plugin asks upstream resolvers about everything else. Most of the cost and most of the confusion come from how Pods ask questions, especially the search list and ndots setting, and from layers of caching and external limits. Understanding those lets you keep DNS fast, reliable and quiet.