Amazon VPC, Under the Hood

Amazon VPC, Under the Hood

An expert-level walkthrough of how a Virtual Private Cloud is actually implemented as a software-defined network beneath the hypervisor — for engineers who already know what a subnet, route table, and security group are, and want to understand the packet-level mechanics, the stateful-vs-stateless distinction, and the multi-VPC connectivity patterns that separate a working architecture from a fragile one.

Amazon VPC is so foundational that most engineers stop asking how it actually works the moment they’ve drawn their first subnet diagram. That’s a mistake, because nearly every advanced AWS networking incident — an asymmetric routing failure, a security group that “should” have blocked traffic but didn’t, a Transit Gateway design that quietly becomes a full-mesh nightmare — traces back to a misunderstanding of what a VPC actually is underneath its console abstraction: a software-defined network implemented at the hypervisor level, with routing, security, and DNS resolution all happening in software before a single packet reaches a real network cable. This guide skips the “what is a subnet” tour and goes straight into the advanced mechanics: how packets actually get mapped and routed, why security groups and NACLs behave so differently despite looking similar, and how multi-VPC connectivity patterns scale — or catastrophically don’t — as an organization grows.

1Internal Working: A Software-Defined Network, Not a Physical One

A VPC has no dedicated physical switches or routers assigned to it — it is a logically isolated network implemented through a mapping service running below the hypervisor on every host, and that single fact explains nearly every advanced VPC behavior.

Every EC2 instance’s network interface is, underneath the abstraction, an Elastic Network Interface (ENI) whose packets are intercepted at the hypervisor layer before they ever touch a physical NIC. AWS’s mapping service — the software layer that implements VPC — decides, per packet, based on the VPC’s route tables, security groups, and NACLs, whether that packet is delivered, dropped, or forwarded elsewhere, entirely in software, on shared physical infrastructure that many customers’ VPCs traverse simultaneously without ever seeing each other’s traffic.

Analogy

Think of the physical AWS data center network as a vast shared highway system, and a VPC as a private, encrypted radio channel used by your vehicles alone. Every other driver on the same physical roads is broadcasting on a completely different channel — the roads are shared, but the mapping service ensures your vehicles only ever “hear” traffic meant for them, with total isolation enforced in software rather than by separate physical lanes.

This software-defined nature is precisely why VPC features that sound like they should require physical hardware — a new subnet, a new route, a security group covering millions of instances — can be created or modified in seconds: there is no physical provisioning step, only an update to the mapping service’s routing and policy state, propagated across the relevant hosts.

!
Advanced Gotcha

Because enforcement happens at the hypervisor level before a packet reaches a physical wire, traditional network packet-capture intuition from on-premises networking doesn’t fully transfer — you cannot “sniff” traffic between two instances at a physical switch the way you could on-prem, because there is no shared physical segment for that traffic to traverse in the first place. VPC Traffic Mirroring exists specifically to give you visibility into this software-defined path.

2Data Flow: The Life of a Packet Through a VPC

A single packet leaving an EC2 instance passes through a strict, ordered sequence of enforcement points before it reaches its destination, and knowing that exact order is the fastest path to root-causing a connectivity problem.

flowchart TD
    A["Packet leaves instance via ENI"] --> B["Security Group evaluated (stateful, instance-level)"]
    B --> C["Subnet outbound NACL evaluated (stateless, subnet-level)"]
    C --> D["Route Table lookup: most specific match wins"]
    D --> E{"Destination"}
    E -- Same VPC --> F["Local route: delivered directly via mapping service"]
    E -- Internet --> G["Internet Gateway / NAT Gateway"]
    E -- Peered VPC / On-Prem --> H["Peering connection / Transit Gateway / VGW"]
    F --> I["Destination subnet inbound NACL evaluated"]
    G --> I
    H --> I
    I --> J["Destination Security Group evaluated"]
    J --> K["Packet delivered to destination ENI"]
    
Fig. 1 — The full enforcement order a packet passes through, from source ENI to destination ENI

The order matters enormously for troubleshooting: a security group is evaluated first on egress and last on ingress, is stateful (a response to an allowed outbound request is automatically permitted back in, with no separate inbound rule needed), and operates at the instance level. A NACL, by contrast, is evaluated at the subnet boundary, is stateless (return traffic must be explicitly allowed by a separate rule, since the NACL has no memory of the original request), and evaluates numbered rules in order, stopping at the first match — a fundamentally different mental model from a security group’s “allow list, no ordering” approach.

Route table lookup follows longest-prefix-match logic identical to traditional networking: the most specific matching route wins, which is why a deliberately narrow route to a specific peered VPC’s CIDR takes precedence over a broader default route to a NAT Gateway, even though both technically match the destination.

Layer

Security Group

Stateful, instance-level, allow-only rules evaluated on every packet in both directions.

Layer

NACL

Stateless, subnet-level, numbered allow/deny rules requiring explicit return-traffic rules.

Layer

Route Table

Longest-prefix-match routing determines the next hop for every packet leaving a subnet.

Layer

Mapping Service

The hypervisor-level software that actually enforces isolation and delivers packets, invisible in the console.

3Subnetting & Routing: Public, Private, and the Gateways That Define Them

“Public” and “private” subnet are not AWS-defined properties of a subnet at all — they are purely a consequence of what that subnet’s route table happens to point at, and understanding that removes a great deal of confusion for advanced network design.

A subnet is “public” purely because its route table contains a route to an Internet Gateway (IGW) for 0.0.0.0/0 — nothing else distinguishes it. A subnet is “private” simply because its default route instead points to a NAT Gateway (for outbound-only internet access) or has no internet-bound route at all. Because this distinction lives entirely in the route table, an engineer can change a subnet’s public/private status instantly by editing a single route, with no other configuration implying or enforcing that designation — which is exactly the kind of change that can go unnoticed in a large route table and quietly expose resources that were assumed private.

NAT Gateways deserve particular advanced attention: they are AWS-managed, highly available within a single AZ, but not inherently multi-AZ — a NAT Gateway deployed in one AZ becomes a single point of failure for every private subnet routing through it if that AZ experiences an outage, which is why production multi-AZ architectures deploy one NAT Gateway per AZ, each serving only the private subnets in its own AZ, rather than a single shared NAT Gateway serving subnets across multiple AZs.

ComponentEnablesDirectionHA Consideration
Internet GatewayPublic subnet internet accessBidirectionalRegionally resilient by design
NAT GatewayPrivate subnet outbound-only accessOutbound onlyAZ-scoped; deploy one per AZ
Virtual Private GatewaySite-to-Site VPN to on-premisesBidirectionalRedundant tunnels per connection
Egress-Only Internet GatewayIPv6 outbound-only accessOutbound onlyRegionally resilient by design
i
Advanced Tip

Auditing “which subnets are actually private” should always mean reading route tables directly, never trusting a subnet’s name — a subnet literally named “private-subnet-1” with a stray 0.0.0.0/0 route to an IGW is, functionally, a public subnet regardless of what anyone called it.

4Advanced Connectivity: Peering, Transit Gateway, and PrivateLink

As organizations grow beyond a single VPC, three distinct connectivity mechanisms — each with a genuinely different scaling and isolation model — compete for the same architectural role, and choosing the wrong one is one of the most consequential mistakes in enterprise AWS networking.

VPC Peering creates a direct, non-transitive connection between exactly two VPCs — non-transitive meaning a peered connection to VPC B does not grant VPC A any reachability to whatever VPC B is itself peered with. This non-transitivity is the mechanism’s defining limitation: connecting N VPCs with full mesh peering requires roughly N(N-1)/2 individual peering connections, a combinatorial growth that becomes operationally unmanageable well before an organization reaches even a few dozen VPCs.

Transit Gateway solves exactly this scaling problem by acting as a central, regional routing hub — every attached VPC (and Site-to-Site VPN, and Direct Connect connection) peers only with the Transit Gateway itself, and the Transit Gateway’s own route tables control which attachments can reach which others, turning what would be a combinatorial peering mesh into a linear number of attachments with centrally governed routing policy.

PrivateLink (interface VPC endpoints) solves a different problem entirely: rather than connecting whole networks, it exposes a single specific service privately, over ENIs placed directly in your subnets, without requiring any IP address space planning coordination between provider and consumer VPCs at all — the standard pattern for securely consuming a specific AWS service or a partner’s SaaS offering without broad network-level connectivity.

Production Example — Enterprise Hub-and-Spoke

Large enterprises with dozens of application VPCs standardize on a Transit Gateway hub-and-spoke model specifically to avoid the peering-mesh scaling wall, centralizing route table policy (which spoke VPCs can reach a shared services VPC, which can reach the internet egress VPC) in one place rather than distributing that logic across dozens of individual peering connections.

ADR-VPC-04Anti-Pattern
Anti-Pattern

Growing an organization’s multi-VPC connectivity organically through ad hoc VPC Peering as each new need arises.

Why It Fails

Peering’s non-transitivity and combinatorial connection growth mean an organically grown peering mesh becomes both operationally unmanageable and impossible to apply consistent network policy across, well before it becomes technically infeasible.

Better Approach

Adopt a Transit Gateway hub-and-spoke design proactively once more than a handful of VPCs need to communicate, centralizing routing policy rather than discovering the peering-mesh scaling wall during an actual incident.

5High Availability & Reliability

A VPC itself is a regional construct with no single point of failure, but the resources and gateways placed inside it very much can be — and VPC-level HA design is really about correctly distributing those resources across Availability Zones.

Subnets are inherently AZ-scoped — a subnet exists entirely within one Availability Zone and cannot span multiple. Genuine high availability therefore requires deliberately creating subnets (and the resources within them) across at least two, typically three, AZs, with load balancers, Auto Scaling groups, and NAT Gateways all explicitly configured to span that same AZ set, rather than assuming the VPC construct itself provides multi-AZ resilience automatically.

The most common HA misconfiguration at the VPC layer is exactly the single-NAT-Gateway mistake from Chapter 3: a cost-optimization decision to run one shared NAT Gateway for an entire VPC, made without recognizing that it silently reintroduces a single-AZ failure domain for every private subnet’s outbound connectivity, undermining an otherwise carefully designed multi-AZ compute architecture.

1 AZ
Maximum span of any single subnet
3
Commonly recommended minimum AZ count for production HA
Per-AZ
Correct NAT Gateway deployment granularity for true HA

6Performance & Scalability

Network performance inside a VPC is governed by instance-type-level networking capability, Enhanced Networking, and placement strategy far more than by anything configured at the VPC construct itself.

Elastic Network Adapter (ENA) support, tied to instance type and size, determines the maximum achievable network throughput and packet-per-second rate for a given instance — a networking-intensive workload placed on an instance type without adequate ENA-driven bandwidth will bottleneck regardless of how well the surrounding VPC is designed. Cluster placement groups reduce inter-instance latency and increase achievable throughput specifically for instances that need to communicate heavily with each other, by packing them onto physically proximate hardware within a single AZ — a meaningful lever for tightly-coupled, high-throughput workloads like distributed training or HPC clusters.

NAT Gateway bandwidth is itself a scalable-but-not-infinite resource — each NAT Gateway supports a large but finite amount of bandwidth, and workloads with very high sustained outbound throughput requirements should validate NAT Gateway bandwidth limits explicitly rather than assuming unlimited scaling, potentially distributing load across multiple NAT Gateways per AZ if genuinely necessary.

i
Advanced Tip

For very high-throughput inter-VPC communication (large data-transfer pipelines between application and data-processing VPCs), a VPC Peering connection typically offers better throughput characteristics than routing the same traffic through a Transit Gateway, because Transit Gateway introduces an additional per-flow bandwidth cap per attachment — a genuine performance/manageability trade-off worth evaluating explicitly for the highest-throughput flows.

7Security: Defense in Depth Across Stateful and Stateless Layers

The combination of stateful security groups and stateless NACLs is a genuine defense-in-depth design, not redundant duplication — advanced security architectures deliberately use both layers for what each does best.

Security groups, being stateful and instance-scoped, are the right place for fine-grained, application-aware access control — “only the load balancer’s security group may reach port 443 on this instance.” NACLs, being stateless and subnet-scoped, are the right place for broad, subnet-wide guardrails, most notably explicit deny rules — security groups support allow rules only, so any requirement to explicitly block a specific known-bad CIDR range regardless of any other rule must be implemented at the NACL layer, since it’s the only layer capable of expressing a deny.

VPC Flow Logs provide the audit and forensic layer underneath both: a record of accepted and rejected traffic at the ENI, subnet, or VPC level, essential for both security investigation and for validating that security group and NACL rules are actually producing the intended traffic pattern rather than the one an engineer assumed they’d configured.

Security Groups — Best For

  • Fine-grained, application-aware allow rules
  • Stateful return-traffic handling with no extra rules needed
  • Instance-level, tag-driven policy at scale

NACLs — Best For

  • Explicit deny rules against known-bad ranges
  • Broad, subnet-wide guardrails independent of individual instance configuration
  • An additional layer that survives a security-group misconfiguration on a single instance

8Monitoring, Logging & Metrics

VPC observability leans on three purpose-built tools — Flow Logs, Traffic Mirroring, and Reachability Analyzer — each answering a genuinely different question about what’s happening inside a software-defined network.

VPC Flow Logs answer “what traffic was allowed or rejected, and by which layer” at a summary level (source, destination, port, protocol, accept/reject), making them the primary tool for security auditing and for diagnosing “traffic I expected to be blocked is getting through” or vice versa. Traffic Mirroring answers a deeper question — “what does the actual packet content look like” — by copying real traffic from an ENI to a monitoring target, essential for deep packet inspection or intrusion detection use cases that flow-level summaries can’t support. VPC Reachability Analyzer answers a third, entirely different question: “is a path between these two points even theoretically possible, given the current route tables, security groups, and NACLs” — a static configuration analysis tool that can identify a broken path without generating any actual traffic at all, valuable for pre-validating a design change before it ever reaches production.

1

Static Validation

Reachability Analyzer confirms a path is theoretically possible before any traffic flows.

2

Summary Monitoring

Flow Logs continuously record accept/reject decisions for ongoing audit and anomaly detection.

3

Deep Inspection

Traffic Mirroring captures actual packet content for security tooling or forensic investigation.

4

Remediate

Findings from any layer feed back into route table, security group, or NACL corrections.

9Design Patterns & Anti-Patterns

The durable large-scale VPC architectures share a common shape: centralized, explicitly governed connectivity and shared services, rather than organically grown, ad hoc network relationships between independently managed VPCs.

Pattern

Transit Gateway Hub-and-Spoke

Application VPCs attach only to a central Transit Gateway, with routing policy governed centrally rather than distributed across peering connections.

Pattern

Shared VPC / Centralized Egress

A dedicated VPC hosts shared NAT Gateways and internet egress, consumed by multiple application VPCs, centralizing cost and security control over outbound traffic.

Anti-Pattern

Organic Full-Mesh Peering

Ad hoc peering added VPC-by-VPC as needs arise eventually hits the combinatorial scaling wall from Chapter 4 with no clean migration path.

Anti-Pattern

Overlapping CIDR Ranges

VPCs provisioned with overlapping IP address ranges cannot be peered or connected via Transit Gateway without disruptive re-addressing — a mistake made once at VPC creation and paid for indefinitely afterward.

10Advantages, Disadvantages & Trade-offs

VPC’s core trade-off is one of near-total networking flexibility against the genuine architectural discipline required to keep that flexibility from producing an unmanageable, ungoverned network sprawl at scale.

Advantages

  • Complete, software-defined control over addressing, routing, and segmentation with no physical hardware to manage
  • Defense-in-depth security model combining stateful and stateless enforcement layers
  • Rich connectivity options (Peering, Transit Gateway, PrivateLink, Direct Connect, VPN) for every integration scenario
  • Purpose-built diagnostic tooling (Flow Logs, Reachability Analyzer, Traffic Mirroring) unavailable in most on-premises networks

Disadvantages

  • CIDR planning mistakes made at creation time are disruptive and costly to fix later
  • Peering’s non-transitivity creates a real scaling ceiling that requires proactive architectural choices to avoid
  • NAT Gateway and Transit Gateway both introduce their own bandwidth and cost considerations at high throughput
  • The software-defined enforcement model requires unlearning some on-premises networking intuitions (packet capture, physical segmentation)

11Best Practices & Common Mistakes

Nearly every advanced VPC incident traces back to a small, well-known set of planning mistakes made early and discovered late — CIDR overlap, single-AZ NAT dependencies, and organically grown peering meshes chief among them.

Plan non-overlapping CIDR ranges across every VPC in an organization from day one, anticipating future connectivity needs even if none exist yet.
Deploy one NAT Gateway per Availability Zone for genuine multi-AZ outbound resilience, never a single shared NAT Gateway.
Adopt Transit Gateway proactively once VPC count and interconnection needs exceed a handful, rather than after the peering mesh becomes unmanageable.
Use NACLs specifically for explicit deny rules that security groups cannot express, not as a duplicate of security-group logic.
Validate route tables directly when auditing subnet public/private status, never trusting naming conventions alone.
!
Most Common Mistake

Choosing VPC CIDR ranges without any coordination across teams or accounts, discovered only when two VPCs genuinely need to be connected and their address spaces overlap — by far the most expensive and disruptive class of VPC mistake to fix after the fact.

12Real-World & Industry Examples

VPC architecture patterns scale directly with organizational complexity — from a single team’s simple two-tier subnet layout to enterprise-wide, centrally governed multi-account network topologies.

Multi-Account Enterprise Landing Zones

Large enterprises implementing an AWS multi-account strategy standardize on Transit Gateway hub-and-spoke topology precisely because it lets a central network team enforce consistent routing and segmentation policy across dozens or hundreds of application-owning accounts without each team needing deep networking expertise.

SaaS Providers Using PrivateLink

B2B SaaS companies expose their service to enterprise customers via PrivateLink endpoints specifically so customers can consume the service without any CIDR coordination or broad network-level trust relationship — a security and operational simplification both sides value heavily.

Hybrid Cloud via Direct Connect and VPN

Organizations migrating workloads gradually to AWS commonly run a Virtual Private Gateway or Transit Gateway VPN/Direct Connect attachment specifically to give on-premises systems and cloud workloads private, low-latency connectivity throughout a multi-year migration, rather than routing hybrid traffic over the public internet.

“A VPC’s flexibility is its greatest strength and its greatest risk — the architectures that scale well are the ones that impose deliberate structure on that flexibility early, not the ones that discover the need for structure during an outage.”

13Frequently Asked Questions

Q1Why is VPC Peering non-transitive, and can that ever be worked around?
Non-transitivity is a deliberate design choice to keep routing scope explicit and predictable — it cannot be bypassed within peering itself, which is precisely why Transit Gateway exists as the mechanism for genuinely transitive, centrally governed multi-VPC routing.
Q2Can a security group reference another security group instead of a CIDR range?
Yes, and this is a core advanced pattern — referencing a security group ID instead of a static IP range lets a rule automatically apply to any instance carrying that security group, regardless of how many instances scale in or out, without ever needing to update the rule itself.
Q3Does a VPC endpoint (Gateway or Interface) route traffic over the public internet?
No — both Gateway endpoints (used for S3 and DynamoDB) and Interface endpoints (PrivateLink, used for most other AWS services) keep traffic entirely on the AWS private network, never traversing the public internet or requiring an Internet Gateway or NAT Gateway at all.
Q4What is the practical difference between a Virtual Private Gateway and a Transit Gateway VPN attachment?
A Virtual Private Gateway attaches directly to a single VPC, while a Transit Gateway VPN attachment connects on-premises networks to the central Transit Gateway hub, making the on-premises connection immediately available to every VPC attached to that Transit Gateway rather than just one.
Q5Is DNS resolution automatically shared across peered or Transit Gateway-connected VPCs?
Not automatically — DNS resolution across VPC boundaries requires explicitly enabling DNS resolution support on the peering connection, or configuring Route 53 Resolver rules and endpoints for Transit Gateway-connected environments, since private hosted zone resolution does not cross VPC boundaries by default.

14Summary and Key Takeaways

Key Takeaways

  • A VPC is a software-defined network enforced at the hypervisor level — there is no dedicated physical hardware, which is why changes apply in seconds and traditional packet-capture intuition doesn’t fully transfer.
  • Every packet passes through a strict, ordered enforcement chain — security group, NACL, route table, mapping service — and troubleshooting should follow that exact order.
  • “Public” and “private” subnet are purely route-table consequences, not an inherent subnet property — always audit routes directly, never subnet names.
  • VPC Peering is non-transitive and scales combinatorially — Transit Gateway exists specifically to solve that scaling wall with centrally governed routing.
  • Security groups and NACLs are complementary, not redundant — stateful allow-only rules at the instance level, stateless allow/deny rules (including explicit denies) at the subnet level.
  • Multi-AZ resilience must be deliberately designed — subnets, NAT Gateways, and load balancers all need explicit per-AZ redundancy; the VPC construct itself does not provide it automatically.
  • CIDR planning and connectivity topology decisions made early are disproportionately expensive to change later — proactive, centrally governed design consistently outperforms organic growth at scale.