Amazon EKS, Under The Control Plane
A deep, engineer-level walkthrough of the control-plane/data-plane boundary, IP allocation internals, IRSA, and the scaling and security decisions that separate a toy cluster from a production one.
If you already know that EKS is “managed Kubernetes on AWS” and you’ve deployed a pod or two, this article skips past that. We’re going into the boundary AWS actually manages versus what you still own, why pod IP addresses can silently become your scaling bottleneck, how IRSA lets a pod assume an IAM role without ever holding a static credential, and how the best-run clusters keep node scaling, security boundaries, and upgrade cadence from becoming a full-time firefighting job.
1Advanced Cluster Concepts
Skipping “what is a pod” — this is the layer where EKS decisions actually diverge from generic upstream Kubernetes guidance.
EKS is upstream Kubernetes with AWS operating the control plane and integrating it deeply with IAM, VPC networking, and load balancing. The advanced decisions live in exactly where that integration creates trade-offs you don’t face on a self-hosted cluster.
Managed Node Groups vs. Self-Managed vs. Fargate
Managed node groups automate node lifecycle and draining; self-managed nodes give full AMI and bootstrap control; Fargate removes node management entirely at the cost of per-pod billing granularity and some networking flexibility.
Karpenter vs. Cluster Autoscaler
Cluster Autoscaler scales predefined node groups up and down; Karpenter provisions right-sized nodes directly against EC2 based on actual unschedulable pod requirements, often achieving faster and more cost-efficient scaling decisions.
IRSA and Pod Identity
IAM Roles for Service Accounts, and its newer successor EKS Pod Identity, let a pod assume a scoped IAM role via short-lived, automatically rotated credentials rather than static access keys baked into an image or environment variable.
VPC CNI and ENI-Based Pod Networking
Each pod gets a real VPC IP address by default, allocated from Elastic Network Interfaces attached to the node — a design that gives native VPC routing but ties pod density directly to instance ENI and IP limits.
EKS Add-ons
AWS-managed lifecycle for core components (VPC CNI, CoreDNS, kube-proxy, EBS/EFS CSI drivers) removes manual version-tracking for these specific pieces, though application-layer add-ons still require your own upgrade discipline.
EKS Auto Mode
A higher-abstraction operating mode where AWS manages node provisioning, scaling, security patching, and core add-ons together, trading some fine-grained control for significantly less operational overhead.
Think of renting a serviced office floor versus owning the building. The building’s structural engineering, fire safety, and elevators (the control plane) are handled entirely by the landlord — you never see that layer. But you still choose your own furniture layout, decide which rooms get keycard access, and manage how many desks fit in each room (nodes, networking, and IAM). EKS Auto Mode is the landlord additionally handling the furniture and desk arrangement for you, for a management fee in flexibility.
What Interviewer May Ask
2Internal Working
What actually lives on AWS’s side of the line, and what still lives on yours.
AWS runs the Kubernetes control plane — the API server, etcd, the scheduler, and the controller manager — across multiple Availability Zones inside an AWS-managed VPC that you never directly access. Your nodes, running in your own VPC, communicate with that control plane over a managed network path. This split is the single most important mental model for reasoning about what you can and cannot control.
flowchart TB
subgraph AWSVPC["AWS-Managed Control Plane VPC"]
API[API Server - Multi-AZ]
ETCD[etcd Cluster - Multi-AZ Quorum]
SCHED[Scheduler]
CM[Controller Manager]
API --- ETCD
API --- SCHED
API --- CM
end
subgraph CustomerVPC["Your VPC"]
N1[Node in AZ-1]
N2[Node in AZ-2]
N3[Node in AZ-3]
FG[Fargate Pods]
end
N1 -->|kubelet| API
N2 -->|kubelet| API
N3 -->|kubelet| API
FG -->|kubelet equivalent| API
IP Address Allocation Internals
The VPC CNI plugin pre-allocates a pool of secondary IP addresses to each node’s Elastic Network Interfaces ahead of actual pod scheduling, so that pod startup doesn’t wait on an IP allocation API call. The maximum number of pods a node can run is therefore a function of the instance type’s maximum ENIs multiplied by IPs per ENI, not the instance’s CPU or memory capacity — a fact that regularly surprises engineers who size nodes only by compute and then hit a pod-count ceiling with capacity to spare.
IRSA Under The Hood
IRSA works by exposing an OIDC identity provider backed by the cluster, associating a Kubernetes service account with an IAM role trust policy scoped to that OIDC provider and service account combination. When a pod using that service account starts, a projected token is mounted into it; the AWS SDK exchanges that token for short-lived IAM credentials via STS automatically — no static AWS access key ever exists inside the pod.
3Data Flow & Lifecycle
From a pod manifest hitting the API server to a running container, and everything that has to happen along the way.
Submission
A pod spec is submitted to the API server, authenticated via IAM-backed cluster authentication (aws-auth mapping or EKS access entries) and authorized by Kubernetes RBAC.
Scheduling Decision
The scheduler evaluates node capacity, including available pod-IP slots per node (not just CPU/memory), taints, tolerations, and affinity rules to pick a target node — or, on Fargate, triggers provisioning of dedicated compute for that pod.
Unschedulable Trigger
If no node has room, Karpenter or Cluster Autoscaler observes the unschedulable pod and provisions new capacity — Karpenter typically reacts directly to the pod’s actual resource requirements rather than a pre-defined instance type list.
Network Attachment
The VPC CNI assigns the pod a secondary IP from the node’s pre-warmed pool, and if network policies are enforced (via the CNI’s own policy engine or a third-party like Calico), those rules attach at this stage too.
Container Start & Identity Injection
The container image is pulled and started; if the pod’s service account is mapped via IRSA or Pod Identity, the projected credential token is mounted at this point, ready for the application’s AWS SDK calls.
Termination & Draining
On scale-down or node replacement, managed node groups and Karpenter both respect pod disruption budgets and graceful termination periods before removing capacity, though misconfigured budgets can still cause brief service disruption.
4Advantages, Disadvantages & Trade-offs
Advantages
- Multi-AZ, self-healing control plane removes an entire category of operational burden that self-hosted Kubernetes carries
- Native VPC pod networking gives first-class security group and routing integration without an overlay translation layer
- IRSA and Pod Identity eliminate static credentials inside containers as a default working pattern, not an advanced afterthought
- Choice across managed nodes, self-managed nodes, and Fargate lets teams trade control for operational simplicity per workload
Disadvantages & Trade-offs
- Pod density is capped by instance ENI/IP limits, not compute capacity, which can force oversized nodes purely to fit more pods
- Kubernetes version upgrades still require careful application-level testing — AWS manages the control plane’s availability, not your workload’s compatibility
- Fargate simplifies operations but restricts certain networking and DaemonSet-based tooling patterns common in node-based clusters
- The breadth of choice (three compute models, two autoscalers, two identity mechanisms) is itself a decision-fatigue cost for teams new to the platform
Anti-Pattern
Sizing nodes purely by CPU and memory headroom while ignoring maximum pods per node, then wondering why nodes sit half-utilized on compute but fully packed on pod count.
Why It Fails
Because the default VPC CNI ties pod capacity to ENI and secondary-IP limits per instance type, a node can hit its pod ceiling with plenty of spare CPU and memory left unused, wasting money on oversized instances that never get to use their full compute.
Better Approach
Size instance types against both compute needs and maximum-pods-per-node together, or adopt prefix delegation / custom networking to increase IP density per node when workloads are many small pods rather than few large ones.
5Performance & Scalability
The control plane itself scales transparently — AWS scales API server and etcd capacity behind the scenes as cluster size and API call volume grow. The scalability decisions actually in your hands are almost entirely about the data plane: how fast new nodes come online, how efficiently pods pack onto them, and how the cluster behaves under bursty scheduling load.
Where Scale Actually Bites
Bursty workloads — batch jobs, sudden traffic spikes — expose the gap between “a node exists” and “a node is ready to schedule pods,” since new EC2 instances still need to boot, join the cluster, and warm their IP pool before they’re useful. Karpenter’s direct EC2 provisioning generally closes this gap faster than Cluster Autoscaler’s node-group-based scaling, but neither is instantaneous, so latency-sensitive bursty workloads often pair autoscaling with a small buffer of pre-warmed, over-provisioned capacity.
sequenceDiagram
participant Pod as Unschedulable Pod
participant Sched as Scheduler
participant Karp as Karpenter
participant EC2 as EC2
Pod->>Sched: Pending, no capacity
Sched->>Karp: Unschedulable event observed
Karp->>EC2: Launch right-sized instance
EC2-->>Karp: Instance running
Karp->>Sched: Node joins cluster
Sched->>Pod: Bind pod to new node
6High Availability & Reliability
AWS runs the control plane’s API server and etcd replicas across multiple Availability Zones with automatic failover, so control-plane HA is largely solved by default. Data-plane HA — making sure your workloads survive a node or AZ failure — remains entirely your design responsibility.
Running a healthy multi-AZ control plane doesn’t help if every replica of a critical Deployment happens to land on nodes in a single Availability Zone. Pod topology spread constraints and anti-affinity rules are what actually convert AWS’s control-plane HA into workload-level HA.
Node group and Karpenter provisioner configuration should explicitly span multiple AZs and subnets, and workload manifests should use topology spread constraints so the scheduler actively distributes replicas rather than allowing them to cluster by coincidence in whichever AZ happened to have capacity first.
Upgrade-Related Availability
Kubernetes minor-version upgrades on EKS are in-place for the control plane, but node upgrades typically require replacing nodes running the old kubelet version. Planning a rolling node replacement with proper pod disruption budgets is what keeps an upgrade from becoming a reliability incident.
7Security
EKS security splits cleanly along the same control-plane/data-plane line as everything else: AWS secures the control plane’s infrastructure, and you secure everything about how workloads authenticate, communicate, and are isolated from each other.
IRSA / Pod Identity Over Node IAM Roles
Granting broad IAM permissions to the node’s instance role means every pod on that node inherits those permissions by default — scoping permissions per pod via IRSA or Pod Identity is the actual least-privilege boundary.
Network Policies
By default, any pod can reach any other pod on the same cluster network — Kubernetes NetworkPolicy resources, enforced via the CNI’s policy engine or a dedicated policy controller, are required to actually restrict east-west traffic.
Envelope Encryption for Secrets
Kubernetes Secrets stored in etcd can be additionally encrypted using a customer-managed KMS key via envelope encryption, protecting secret data even in the (AWS-managed) etcd store itself.
Pod Security Standards
Enforcing the Restricted or Baseline Pod Security Standard profile at the namespace level prevents privilege-escalation patterns like privileged containers or host-path mounts from being schedulable at all.
8Monitoring, Logging & Metrics
EKS exposes control-plane logging as an opt-in feature per log type, and none of them are enabled by default — a detail that regularly leaves teams without audit trails they assumed already existed.
| Control Plane Log Type | What It Captures | Why It Matters |
|---|---|---|
| API server | All requests to the Kubernetes API | The primary audit trail for who did what against the cluster |
| Audit | Detailed, structured record of API server activity | Forensic-grade detail beyond the general API log |
| Authenticator | IAM-to-Kubernetes-RBAC authentication events | Confirms which IAM identity mapped to which Kubernetes user or group |
| Controller manager | Internal controller loop activity | Useful for diagnosing cluster-level control loop issues |
| Scheduler | Scheduling decisions and failures | Explains why a pod landed where it did, or why it didn’t schedule at all |
Enable at minimum the API server, audit, and authenticator log types from day one, routed to CloudWatch Logs or a central log-archive account, alongside Container Insights for node and pod-level resource metrics — reconstructing this history after an incident is far harder than logging it from the start.
9Deployment & Cloud
Production EKS clusters are provisioned and evolved through code, not console clicks, and increasingly through GitOps for the workloads running on top of them.
Cluster Provisioning as Code
Terraform, CloudFormation, or eksctl-generated configuration defines the cluster, VPC, subnets, and initial node groups as reviewable, repeatable infrastructure.
Add-on and Autoscaler Bootstrap
Core add-ons (VPC CNI, CoreDNS, kube-proxy, CSI drivers) and Karpenter or Cluster Autoscaler are installed as part of the same provisioning pipeline, not as a manual post-cluster-creation step.
GitOps for Workloads
Tools like Argo CD or Flux continuously reconcile cluster state against a Git repository, so what’s actually running is always traceable to a reviewed commit rather than an untracked kubectl apply.
Upgrade Pipeline
Kubernetes version upgrades follow a tested sequence — control plane upgrade, then add-on version alignment, then node replacement — often validated first against a staging cluster with the same manifests.
Choosing Between EKS Auto Mode and Manual Configuration
Teams without dedicated platform engineering capacity increasingly choose EKS Auto Mode specifically to offload node provisioning, patching, and core add-on management, accepting less granular control in exchange for meaningfully lower ongoing operational load — a trade-off best made deliberately, not by default inertia in either direction.
10Design Patterns & Anti-patterns
Per-Workload IRSA Scoping
Every service account that needs AWS access gets its own narrowly-scoped IAM role via IRSA or Pod Identity, rather than sharing a broad role across unrelated workloads.
Topology-Spread-First Deployments
Every critical Deployment manifest includes topology spread constraints across AZs by default, converting the control plane’s multi-AZ resilience into actual workload resilience.
Karpenter for Heterogeneous Workloads
Karpenter’s direct, requirement-driven provisioning fits clusters running a wide mix of pod sizes better than a fixed set of pre-defined node group instance types.
Broad Node IAM Roles
Attaching a permissive IAM role to the node’s instance profile “to make things work” grants that access to every pod scheduled on the node, defeating per-workload least privilege entirely.
No Network Policies
Leaving the cluster’s default any-pod-to-any-pod networking in place means a single compromised pod has unrestricted lateral movement across every namespace on the cluster.
Ignoring Pod Density Limits
Choosing instance types by compute specs alone and discovering the pod-count ceiling only in production, after nodes are already oversized and underutilized.
11Best Practices & Common Mistakes
Best Practices
- Scope IAM access per workload via IRSA or Pod Identity rather than through the node’s own instance role
- Enable API server, audit, and authenticator control-plane logging from day one
- Use topology spread constraints on every critical Deployment to convert control-plane HA into workload HA
- Enforce Pod Security Standards at the namespace level rather than relying on developer discipline alone
- Size nodes against both compute needs and maximum-pods-per-node together
Common Mistakes
- Assuming AWS managing the control plane means the cluster is fully secured, including workload-level access
- Leaving control-plane logging disabled and discovering the gap only during an incident investigation
- Skipping network policies because “it’s inside the VPC anyway”
- Treating a Kubernetes version upgrade as complete once the control plane is upgraded, without replacing outdated nodes
- Not testing pod disruption budgets before a rolling node replacement, causing avoidable downtime
12Real-World & Industry Examples
Airbnb — Large-Scale Container Platform Migration
As Airbnb scaled its service-oriented architecture, it adopted Kubernetes-based container orchestration to standardize deployment across hundreds of services, relying on managed control-plane resilience to keep platform-team overhead from growing linearly with service count.
Snap Inc. — Multi-Tenant Cluster Efficiency
Organizations running many teams on shared EKS clusters commonly lean on Karpenter-style right-sized provisioning specifically to avoid the cost of maintaining separate, permanently-provisioned node groups per team, consolidating spare capacity across tenants instead.
Financial Services — IRSA as a Compliance Control
Regulated organizations running workloads on EKS frequently point to IRSA and Pod Identity specifically in audit documentation as the mechanism eliminating static long-lived credentials from container images and environment variables, directly addressing a common compliance finding from earlier container platforms.
13Frequently Asked Questions
14Summary and Key Takeaways
What to Carry Forward
- EKS splits cleanly into an AWS-managed, multi-AZ control plane and a customer-managed data plane — knowing exactly where that line sits explains most of the platform’s behavior and limits.
- Pod density per node is bounded by ENI and secondary-IP limits from the VPC CNI, not by CPU or memory — size nodes with pod count in mind, not compute alone.
- IRSA and EKS Pod Identity let pods assume scoped IAM roles via short-lived credentials — granting access through the node’s own instance role instead defeats per-workload least privilege.
- Karpenter provisions capacity directly against actual unschedulable pod requirements, generally reacting faster and more precisely than pre-sized node-group autoscaling.
- Control-plane HA from AWS doesn’t automatically become workload HA — topology spread constraints and anti-affinity rules are what make that resilience real for your applications.
- None of the five control-plane log types are enabled by default — turn on at least API server, audit, and authenticator logging before you need them, not after an incident.
- Default cluster networking allows any pod to reach any other pod — NetworkPolicy resources are required to actually establish isolation boundaries between workloads.