Amazon EKS – Beyond the Basics
A working engineer's guide to how the control plane, VPC CNI, and IAM actually cooperate — node group choices, pod identity, IP exhaustion, and the upgrade discipline that keeps a cluster boring.
If you already know that EKS is “managed Kubernetes on AWS,” this article picks up one level higher. We’ll spend our time on what AWS actually manages versus what remains your responsibility, how the VPC CNI plugin’s IP-per-pod model creates a real capacity constraint most teams don’t plan for early enough, how workloads get scoped AWS permissions without static credentials, and the node group and autoscaling choices that determine whether your cluster gracefully absorbs a traffic spike or falls behind it. We’ll treat managed node groups and the VPC CNI as the default configuration throughout, calling out Fargate and alternative CNI plugins where the distinction genuinely matters.
A fast recap of the shared responsibility line, framed for someone already running a cluster.
AWS manages the Kubernetes control plane — the API server, etcd, scheduler, and controller manager — running it across multiple Availability Zones with automatic scaling and patching of that layer. What remains squarely your responsibility is everything on the data plane side: node provisioning and patching (unless using Fargate), workload scheduling decisions, cluster and application-level upgrades, networking configuration choices, and IAM/RBAC design. This division is the source of most “wait, why do I still have to do this” moments for teams new to EKS coming from a fully self-managed Kubernetes background.
Control plane availability & patching
API server scaling, etcd durability and backups, control plane version patches within a supported window, and Multi-AZ control plane resilience.
Nodes, workloads, and cluster configuration
Node group sizing and AMI updates (for managed/self-managed nodes), workload scheduling and resource requests, RBAC and IAM mapping, add-on versions, and Kubernetes version upgrade timing.
The vocabulary intermediate EKS work depends on — compute options and networking primitives.
Managed Node Groups — EC2 instances provisioned and lifecycle-managed by EKS (rolling updates, graceful draining on termination) through an Auto Scaling group AWS creates on your behalf, while you still choose the instance types, AMI, and scaling configuration.
Self-Managed Node Groups — EC2 instances you provision and manage entirely yourself (your own Auto Scaling group, your own AMI pipeline), joined to the cluster manually — more control, more operational burden.
Fargate — a serverless compute option for pods; no EC2 instances to manage at all, with each pod running in its own isolated compute environment, at the cost of some feature restrictions (no DaemonSets, no privileged containers) and generally higher per-vCPU cost than EC2 for steady-state workloads.
VPC CNI Plugin — the default networking plugin, which assigns each pod a real, routable IP address directly from your VPC’s subnet ranges, rather than an overlay network — this is what lets pods communicate with other VPC resources (RDS, ElastiCache) as if they were regular VPC-native workloads.
How the control plane, data plane, and add-ons fit together inside your VPC.
flowchart TB
subgraph ControlPlane["AWS-Managed Control Plane (Multi-AZ)"]
API["API Server"]
ETCD["etcd"]
Sched["Scheduler & Controller Manager"]
end
subgraph VPC["Your VPC"]
subgraph AZa["Availability Zone A"]
NodeA["Worker Node"]
PodA1["Pod — VPC IP"]
PodA2["Pod — VPC IP"]
NodeA --- PodA1
NodeA --- PodA2
end
subgraph AZb["Availability Zone B"]
NodeB["Worker Node"]
PodB1["Pod — VPC IP"]
NodeB --- PodB1
end
end
API --> NodeA
API --> NodeB
NodeA -->|kubelet reports status| API
NodeB -->|kubelet reports status| API
Cluster networking also depends on subnets being correctly tagged for load balancer discovery and node group placement, and on security groups controlling traffic between the control plane and nodes — a common early-cluster networking issue traces back to nodes failing to join because a security group rule blocking control-plane-to-node communication was too restrictive.
The constraint that eventually surfaces in every growing EKS cluster on the default CNI.
Because the VPC CNI assigns each pod a genuine IP address from an attached Elastic Network Interface’s (ENI) allocated range, the number of pods a node can run is bounded by how many ENIs and secondary IPs that instance type supports — a smaller instance type can run far fewer pods than its CPU and memory would otherwise suggest, purely because of this IP ceiling.
Think of each worker node as an apartment building and each pod as a tenant who needs their own street address for mail to reach them directly. A building’s number of available street addresses is fixed by its physical mailbox panel, regardless of how many empty rooms it has — you can have plenty of living space (CPU and memory) left, but no room for a new tenant if the mailbox panel (IP allocation) is already full.
This has a second-order effect at the VPC level: a cluster running many small pods across many subnets can exhaust available subnet IP space faster than teams expect, especially in subnets originally sized for traditional EC2-based capacity planning rather than dense pod scheduling. Mitigations include using larger CIDR ranges dedicated to pod networking, enabling prefix delegation (assigning /28 IP prefixes to ENIs instead of individual IPs, dramatically increasing pod density per node), or switching specific workloads to Fargate, which manages its own networking outside this per-node ENI constraint.
What happens between submitting a pod spec and it actually running somewhere.
sequenceDiagram
participant User as kubectl / CI Pipeline
participant API as EKS API Server
participant Sched as Scheduler
participant Node as Worker Node
participant CNI as VPC CNI Plugin
User->>API: Apply Deployment manifest
API->>Sched: New unscheduled pod detected
Sched->>Sched: Evaluate node resources, taints, affinity rules
Sched->>Node: Bind pod to selected node
Node->>CNI: Request pod network setup
CNI-->>Node: Assign VPC IP to pod
Node-->>API: Report pod Running
Resource requests and limits set on pods directly influence the scheduler’s placement decision — a pod without a CPU/memory request is treated as effectively weightless for bin-packing purposes, which can lead to nodes becoming genuinely overcommitted even though the scheduler’s own accounting shows capacity available.
How pods get scoped AWS permissions without embedding static credentials.
IAM Roles for Service Accounts (IRSA) associates a Kubernetes service account with an IAM role via an OIDC identity provider trust relationship, letting pods using that service account assume the role and receive temporary AWS credentials automatically — no access keys stored in a Secret, no credentials baked into a container image.
EKS Pod Identity is a newer, simpler mechanism accomplishing a similar goal — associating an IAM role with a service account through a dedicated EKS Pod Identity agent rather than requiring you to manage the OIDC provider trust policy configuration yourself, reducing setup complexity for the common case.
Scoping IAM roles per service account, rather than granting broad permissions to the node’s own instance role, is what keeps a compromised pod from inheriting every AWS permission every workload on that node might need — the node instance role should carry only what’s needed for node-level operation (like pulling container images and reporting to the control plane), not application-level AWS access.
Two permission systems that have to agree before a human or workload can do anything.
Kubernetes RBAC (Roles, ClusterRoles, and their Bindings) controls what an authenticated identity can do inside the cluster. But authentication into the cluster in the first place is an IAM concern — historically bridged by the aws-auth ConfigMap mapping IAM principals to Kubernetes usernames and groups, and more recently by the native EKS access entries API, which manages that same mapping as a first-class EKS resource rather than an editable ConfigMap prone to accidental corruption.
Both layers have to line up: an IAM user or role needs a mapping that grants it a Kubernetes identity, and that Kubernetes identity in turn needs an RBAC binding granting it actual permissions inside the cluster — missing either half results in either “who are you” (authentication) or “you’re recognized but not allowed to do that” (authorization) errors that are easy to conflate when troubleshooting.
Two different philosophies for adding and removing capacity.
Cluster Autoscaler
- Scales existing, pre-defined Auto Scaling groups up or down based on pending/unschedulable pods
- Simpler mental model, but capacity choices are constrained to whatever instance types your node groups were already configured with
Karpenter
- Provisions right-sized nodes directly in response to unschedulable pods, choosing from a broad range of instance types dynamically rather than a pre-fixed node group
- Generally faster to provision new capacity and better at bin-packing efficiently, at the cost of a different (and to some teams, less familiar) operational model than a traditional Auto Scaling group
Neither tool scales the control plane itself — that’s handled entirely by AWS. Both address the data plane side of the shared responsibility split introduced in Chapter 1.
How pods get persistent storage that outlives the pod itself.
The Amazon EBS CSI Driver provisions EBS volumes as Kubernetes PersistentVolumes for workloads needing block storage — the natural fit for a single-writer database-like workload, but bound to a single Availability Zone, meaning a pod using an EBS-backed volume can only be rescheduled within that same AZ.
The Amazon EFS CSI Driver provisions access to EFS file systems, which support concurrent multi-AZ, multi-pod read/write access — the right choice for workloads genuinely needing shared file storage across many pods simultaneously, at the cost of higher per-GB cost and generally higher latency than EBS for a single workload’s exclusive use.
Why Kubernetes upgrades on EKS require more planning than a simple version bump.
Kubernetes supports a limited version skew between the control plane and nodes (nodes can typically run up to a small number of minor versions behind the control plane, but not ahead of it), which means upgrading the control plane first and then rolling nodes forward afterward is the standard, required order — upgrading nodes ahead of the control plane isn’t a supported configuration.
Deferring Kubernetes version upgrades for too long risks landing on a version that has already reached end of standard support, forcing an unplanned, urgent multi-version jump under time pressure instead of the smaller, routine, one-minor-version-at-a-time upgrades that are far safer to test and roll back.
Add-ons (VPC CNI, CoreDNS, kube-proxy, the EBS/EFS CSI drivers) have their own version compatibility matrices against the control plane version, and EKS-managed add-ons can be upgraded independently of a full cluster version bump — but letting add-on versions drift far behind the control plane version is itself a common source of subtle, hard-to-diagnose compatibility issues.
Layered controls spanning IAM, Kubernetes-native policy, and network isolation.
- IRSA / Pod Identity — scoped, temporary AWS credentials per workload, covered in Chapter 6, replacing the older and riskier pattern of granting broad permissions to the node instance role.
- RBAC — least-privilege in-cluster permissions, ideally granted per namespace or workload rather than cluster-wide by default.
- Network Policies — Kubernetes NetworkPolicy resources (enforced by a compatible CNI or an add-on like Calico) restrict which pods can communicate with which others, since the default VPC CNI configuration otherwise allows any pod to reach any other pod on the cluster network.
- Secrets Encryption — EKS supports envelope encryption of Kubernetes Secrets using a customer-managed KMS key, adding a layer of protection beyond the base etcd encryption AWS manages.
- Private Endpoint Access — restricting the cluster’s API server endpoint to private VPC access only (rather than a public endpoint) removes an entire class of external exposure for the control plane API itself.
The signals that separate a healthy cluster from one quietly degrading.
| Signal | What It Tells You | Watch For |
|---|---|---|
| Control Plane Logs (API server, audit, authenticator) | Requests made against the API server, including authentication and authorization decisions | Not enabled by default — a common gap discovered only during an incident investigation when the logs turn out not to have been on |
| Container Insights (CloudWatch) | Aggregated CPU, memory, and network metrics at the cluster, node, and pod level | Sustained memory pressure on nodes ahead of visible pod evictions is an early warning worth alerting on directly |
| Pending / Unschedulable Pods | Whether the scheduler currently has pods it cannot place | A rising trend here, faster than autoscaling responds, points to capacity growth outpacing your scaling configuration |
| Available IPs per Subnet | Remaining pod-networking headroom under the VPC CNI model | Approaching exhaustion here causes new pods to fail scheduling with networking errors, not capacity errors — a distinct failure mode from CPU/memory pressure |
How EKS is provisioned and combined with the rest of AWS’s ecosystem.
Clusters, node groups, and add-ons are commonly provisioned through Infrastructure as Code (Terraform, CDK, or eksctl-generated configuration) since manually clicking through cluster setup makes reproducing an identical environment for disaster recovery or a second Region unnecessarily difficult.
AWS Load Balancer Controller
Watches Kubernetes Ingress and Service resources and automatically provisions and configures ALBs or NLBs to match, bridging Kubernetes-native manifests to real AWS load balancer resources.
Continuous Deployment Tooling
Tools like Argo CD or Flux are commonly layered on top of EKS to reconcile cluster state against a Git repository, treating the desired cluster state as version-controlled and auditable rather than applied ad hoc.
What a resilient multi-tenant cluster design looks like — and the mistake that turns a busy Friday into an incident.
Namespace-per-team with RBAC and resource quotas — combining namespace isolation, scoped RBAC bindings, and ResourceQuota objects lets multiple teams share a cluster safely, each unable to starve the others of compute or accidentally touch another team’s resources.
Separate node groups by workload profile — isolating memory-intensive workloads, GPU workloads, or workloads needing specific taints/tolerations onto dedicated node groups avoids forcing the scheduler into awkward bin-packing decisions across a single, undifferentiated fleet.
Pattern
Granting the node’s own IAM instance role broad AWS permissions “so every pod on it can access what it needs,” instead of using IRSA or Pod Identity per workload.
Why It Fails
Every pod scheduled on that node inherits whatever the instance role grants, regardless of whether that specific workload actually needs it — a single compromised or misconfigured pod effectively gains the combined AWS permissions intended for every workload sharing that node.
What To Do Instead
Keep the node instance role scoped to what nodes themselves need (image pulls, control-plane communication, CNI operation) and grant application-specific AWS access through IRSA or Pod Identity, scoped to the individual service account that actually needs it.
Advantages
- AWS-managed, Multi-AZ control plane removes a large class of operational burden compared to self-hosted Kubernetes
- VPC CNI gives pods first-class VPC networking, simplifying integration with other AWS services
- IRSA/Pod Identity enables fine-grained, credential-free AWS access per workload
- Flexible compute choice (managed nodes, self-managed, Fargate) fits different operational and cost profiles
Disadvantages & Trade-offs
- The VPC CNI’s IP-per-pod model creates a real, sometimes underestimated pod-density ceiling per node and per subnet
- Data plane management (nodes, upgrades, scaling) remains substantially your responsibility despite “managed” in the name
- Version skew rules force a specific, sometimes inconvenient upgrade order between control plane and nodes
- Fargate’s per-pod isolation model excludes certain workload types (DaemonSets, privileged containers) outright
Plan IP address space before you scale, not after
Enable prefix delegation or design subnets with pod density in mind early — retrofitting IP capacity into a live, growing cluster is far more disruptive than planning for it up front.
Scope IAM per workload with IRSA or Pod Identity
Avoid broad node instance role permissions; grant AWS access at the service-account level so a compromised pod’s blast radius stays limited to what it actually needs.
Upgrade one minor version at a time, on a schedule
Routine, small upgrades are far safer to test and roll back than an urgent multi-version jump forced by an approaching end-of-support deadline.
Set realistic resource requests and limits on every pod
Unrequested resources make the scheduler’s bin-packing decisions inaccurate, leading to nodes that look fine on paper but are genuinely overcommitted in practice.
Enable control plane logging before you need it in an incident
API server and audit logs are off by default — discovering that during a security investigation is far worse than enabling them proactively.
Microservices platforms with mixed compute needs
Organizations commonly mix managed node groups for steady-state services with Fargate for bursty, low-frequency batch jobs, avoiding the cost of keeping EC2 capacity provisioned for workloads that only run occasionally.
Multi-tenant SaaS platforms
Namespace isolation combined with RBAC and network policies lets a SaaS provider run multiple customer workloads on shared cluster infrastructure while maintaining strict logical separation between tenants.
Machine learning training and inference
GPU-backed node groups, isolated via taints and tolerations from general-purpose workloads, are a common pattern for teams running training jobs and inference services on the same cluster as their standard application workloads.
Carry This Forward
- AWS manages the control plane’s availability and patching; nodes, workloads, upgrades, and cluster configuration remain your responsibility.
- The VPC CNI’s IP-per-pod model creates a real pod-density ceiling per node and subnet — plan for it with prefix delegation or Fargate before it becomes a live incident.
- IRSA and Pod Identity let workloads get scoped, temporary AWS credentials without broad node instance role permissions or static keys.
- Kubernetes RBAC and IAM-to-Kubernetes identity mapping (aws-auth or access entries) are two separate systems that both must be correctly configured for access to work.
- Version skew rules require upgrading the control plane before nodes, and deferring upgrades risks a forced, urgent multi-version jump later.
- Cluster Autoscaler and Karpenter represent two different scaling philosophies — pre-defined node groups versus dynamic, right-sized provisioning.
- Scope node instance roles narrowly and grant workload-specific AWS access per service account, not per node.