Amazon SNS, Beneath the Fan-Out
A deep, engineer-level walkthrough of how Amazon SNS actually delivers, retries, and filters messages at scale — the pub/sub delivery model, FIFO topic internals, message filtering evaluation, and the patterns that separate a reliable fan-out architecture from a silent message-loss incident.
If you already know that SNS is “pub/sub for AWS,” you know the tagline, not the delivery engine. The engineering substance in SNS lives in how it fans a single published message out to a heterogeneous set of subscribers with independent retry policies, how message filtering happens server-side before a subscriber ever receives anything it doesn’t need, and how FIFO topics enforce ordering and deduplication guarantees that standard topics deliberately don’t provide. This walkthrough assumes you’ve already created a topic and subscribed something to it; the focus is what’s happening underneath, and where production systems actually lose messages or receive them out of order if these internals aren’t understood.
AAdvanced Core Concepts
Skipping “what is pub/sub” — this is the model experienced engineers reach for when reasoning about SNS in production.
Delivery to each subscriber is independent, not transactional
When a message is published to a topic with five subscribers, SNS doesn’t deliver it as a single atomic operation across all five. Each subscription has its own delivery attempt, its own retry policy, and its own success or failure outcome, entirely independent of the others. A message can succeed for four subscribers and exhaust retries for the fifth without any of that affecting the other four — there’s no all-or-nothing guarantee across the fan-out. This independence is a deliberate design choice: it means one slow or broken subscriber can’t block delivery to the rest, but it also means “the message was published” tells you nothing about whether every subscriber actually received it without checking each subscription’s delivery status individually.
Think of publishing to an SNS topic like a store manager announcing a memo to five different departments by five separate couriers. Each courier has their own route and their own chance of getting stuck in traffic — the sales department getting the memo instantly doesn’t mean the warehouse got it at all. The manager saying “I sent it” only confirms it left their hands.
Message filtering happens at the subscription, evaluated server-side before delivery
A filter policy attached to a subscription is evaluated by SNS itself against the message’s attributes (or, with body-based filtering, the message payload) before that message is ever sent to the subscriber — filtering is not something the subscriber does after receiving everything. This server-side evaluation is what makes fan-out architectures efficient at scale: a topic can have a hundred subscribers, each with a narrow filter policy, and each one only ever receives the small subset of messages actually relevant to it, without the topic needing to know anything about subscriber-side logic.
FIFO topics provide ordering and deduplication only within a message group
A FIFO SNS topic guarantees strict ordering and exactly-once delivery, but both guarantees are scoped to a MessageGroupId, not the topic as a whole. Messages in different message groups can be delivered in any relative order to each other, even though messages within the same group are strictly ordered. Deduplication uses either an explicit MessageDeduplicationId or, if content-based deduplication is enabled, a SHA-256 hash of the message body, within a 5-minute deduplication interval — publishing the same content twice within that window is treated as a duplicate and only delivered once.
Independent Per-Subscription Delivery
Each subscriber has its own retry policy and outcome — no cross-subscriber atomicity or shared failure state.
Server-Side Filter Policy
SNS evaluates filter policies before delivery, so subscribers only ever receive messages matching their declared interest.
FIFO Message Group Scoping
Ordering and dedup guarantees apply within a MessageGroupId, not across the whole topic.
Per-Subscription Redrive Policy
A dead-letter queue can be attached per subscription, capturing only that subscriber’s exhausted-retry messages.
IInternal Working
What actually happens between “Publish API call” and each subscriber independently receiving (or not receiving) the message.
When a message is published, SNS first evaluates the topic’s access policy to confirm the publisher is authorized. Once accepted, the message is durably stored and the fan-out process begins: SNS enumerates the topic’s active subscriptions and, for each one, evaluates that subscription’s filter policy against the message’s attributes. Subscriptions whose filter doesn’t match are skipped entirely for that message — no delivery attempt is made, and this skip isn’t logged as a failure since it was never intended to be delivered there.
For each matching subscription, SNS attempts delivery according to that protocol’s delivery mechanics: HTTP/S endpoints receive a POST request and must respond with a 2xx status for the attempt to count as successful; SQS queue subscriptions have the message enqueued directly; Lambda subscriptions invoke the function synchronously from SNS’s perspective (though Lambda itself may process asynchronously depending on its own configuration); email and SMS use their respective delivery channels with very different latency and reliability characteristics than the programmatic protocols.
graph TD
A[Publish API Call] --> B[Access Policy Check]
B --> C[Message Durably Stored]
C --> D[Enumerate Active Subscriptions]
D --> E{Filter Policy Match?}
E -->|No| F[Skip - No Delivery Attempt]
E -->|Yes| G[Attempt Delivery Per Protocol]
G --> H[SQS: Enqueue Message]
G --> I[Lambda: Synchronous Invoke]
G --> J[HTTPS: POST with 2xx Expected]
H --> K[Success]
I --> K
J -->|2xx| K
J -->|Non-2xx or Timeout| L[Retry Policy Engaged]
L -->|Exhausted| M[Dead-Letter Queue if Configured]
Fig 1 — Fan-out delivery pipeline showing per-subscription filtering and independent delivery outcomes
An HTTP/S subscriber that responds with a 2xx status but then fails to actually process the message internally looks identical to a successful delivery from SNS’s perspective — SNS has no visibility into what the endpoint does after acknowledging receipt. Reliable HTTP/S subscribers must not return 2xx until processing has genuinely succeeded or been durably queued internally.
DData Flow & Lifecycle
A message’s delivery lifecycle for a given subscription follows a retry state machine governed by that subscription’s delivery retry policy, which is configurable independently per subscription and per protocol — a critical detail, since it means the same published message can be retried far more aggressively for one subscriber than another.
Immediate Delivery Attempt
SNS attempts delivery to the subscriber’s endpoint as soon as the message passes filtering.
Backoff Retry Phase
On failure, SNS retries according to the subscription’s configured backoff policy (linear, exponential, or a custom min/max delay and retry count).
Extended Retry (HTTP/S only)
HTTP/S subscriptions support an optional extended retry phase beyond the immediate policy, for tolerating longer subscriber outages.
Exhaustion
Once all configured retries are exhausted without success, the message is considered undeliverable to that specific subscription.
Dead-Letter Capture (if configured)
If a redrive policy with a dead-letter queue is attached to that subscription, the exhausted message is sent there; otherwise it is permanently lost with only CloudWatch metrics as evidence it ever failed.
For FIFO topics, this lifecycle interacts with ordering guarantees in an important way: SNS will not deliver a message to a subscribed FIFO-compatible endpoint out of order within its message group, meaning a stuck or slow delivery at the head of a group can delay delivery of subsequent messages in that same group, even while messages in other groups continue flowing normally.
TAdvantages, Disadvantages & Trade-offs
Advantages
- Native fan-out to multiple heterogeneous protocols (SQS, Lambda, HTTP/S, email, SMS) from a single publish call removes the need for custom distribution logic.
- Server-side message filtering reduces unnecessary invocations and processing cost across large subscriber sets.
- FIFO topics provide strict ordering and exactly-once delivery guarantees without building a custom sequencing layer.
- Per-subscription dead-letter queues isolate failure handling so one broken subscriber’s issues don’t require touching the topic or other subscriptions.
Disadvantages / Trade-offs
- No cross-subscriber delivery atomicity — a “successfully published” message provides no guarantee every subscriber actually received it.
- Standard (non-FIFO) topics provide no ordering guarantee at all, and at-least-once delivery means duplicate handling must be built into subscriber logic.
- FIFO topics’ ordering guarantee is scoped to message groups, which requires careful group-key design to actually get the ordering behavior an application needs.
- HTTP/S subscriber reliability depends entirely on correct 2xx-after-real-processing semantics — a naive implementation can silently drop messages while appearing healthy.
PPerformance & Scalability
Standard SNS topics scale to very high publish throughput with no pre-provisioned capacity, since the service is fully managed and elastic. FIFO topics, by contrast, have a defined throughput ceiling per message group (though overall topic throughput scales with the number of distinct message groups in use), which means an application funneling all traffic into a single message group for “simplicity” caps its own achievable throughput regardless of the topic’s overall capacity — a frequent and avoidable bottleneck.
Message filtering’s performance benefit compounds at scale: a topic with a thousand narrowly-filtered subscribers processing a high publish rate only incurs actual delivery cost and subscriber-side processing for the messages each one cares about, rather than every subscriber processing and discarding irrelevant messages itself — pushing that filtering cost to the managed service rather than replicating filter logic in every subscriber’s code.
Robinhood has publicly discussed designing FIFO SNS message group keys around per-account or per-symbol identifiers specifically to maximize parallel throughput while preserving strict ordering guarantees only where genuinely required — illustrating that message-group key design is itself a scalability decision, not just a correctness one.
HHigh Availability & Reliability
SNS itself is a Regional, managed, multi-AZ service with no customer-managed redundancy layer — the reliability engineering that matters is in per-subscription retry and dead-letter configuration, and in cross-Region strategy for genuinely disaster-resilient architectures.
Without a dead-letter queue configured on a subscription, a message that exhausts retries is gone permanently, with CloudWatch’s NumberOfNotificationsFailed metric as the only evidence it happened — there’s no built-in replay mechanism after the fact. This is why production subscriptions handling business-critical events should always have an explicit redrive policy, treating “no DLQ configured” as an active reliability gap rather than an acceptable default. For cross-Region resilience, SNS topics don’t natively replicate across Regions, so multi-Region architectures typically publish to independent topics in each Region from application code aware of the active Region, or use a cross-Region event bus pattern via EventBridge for more complex routing needs.
Reliability pattern used by mature teams
Attach a dead-letter queue to every subscription handling business-critical events, alarm on that DLQ’s message count rather than only on the topic’s failure metric, and build a periodic or on-demand redrive process to reprocess DLQ messages once the root cause is fixed.
SSecurity
SNS access control operates through both IAM policies attached to principals and topic policies — resource-based policies attached directly to the topic — with topic policies being the standard mechanism for cross-account publish or subscribe permissions, analogous to the repository-policy pattern used elsewhere in AWS. A topic policy that’s too permissive (allowing sns:Subscribe from any principal, for example) can let an unintended party subscribe an endpoint they control and receive messages meant for a closed set of internal consumers.
Message content encryption at rest is available via SSE with a customer-managed or AWS-managed KMS key, which matters for topics carrying sensitive payloads, though it’s worth noting that SNS message attributes used for filtering are still evaluated in plaintext by the filtering engine even with encryption enabled — encryption protects data at rest and in transit, not the filtering logic’s visibility into attribute values it needs to route messages correctly.
Scope topic policies to specific principals and specific actions (never a wildcard principal for Subscribe or Publish on a topic carrying sensitive data), enable SSE with a customer-managed KMS key for regulated payloads, and audit topic policies periodically since a policy that was correctly scoped at creation can become overly permissive as an organization’s account structure evolves.
MMonitoring, Logging & Metrics
SNS publishes per-topic CloudWatch metrics including NumberOfMessagesPublished, NumberOfNotificationsDelivered, and NumberOfNotificationsFailed, but these are topic-level aggregates by default — diagnosing which specific subscription is failing requires enabling delivery status logging per protocol (supported for SQS, Lambda, HTTP/S, Firehose, and application endpoints), which writes detailed per-attempt success and failure logs to CloudWatch Logs, including HTTP response codes for HTTP/S subscribers.
Because delivery status logging is opt-in and has its own cost, many teams under-enable it and then have no diagnostic path when a specific subscriber silently stops receiving messages — enabling it, at least at a sampled rate, for any production subscription is the standard recommendation for maintaining actual delivery visibility rather than relying solely on topic-level aggregate metrics that can mask a single failing subscriber among many healthy ones.
| Signal | Source | Primary Use |
|---|---|---|
| Topic-level publish/delivery/failure counts | CloudWatch topic metrics | Aggregate health monitoring, alarming |
| Per-attempt delivery status | Delivery status logging (opt-in, per subscription) | Diagnosing a specific failing subscriber |
| Dead-letter queue depth | CloudWatch SQS metrics on the DLQ | Detecting exhausted-retry messages needing attention |
| Control-plane changes | AWS CloudTrail | Auditing topic policy and subscription configuration changes |
DDeployment & Cloud Architecture
The canonical production fan-out architecture pairs an SNS topic with multiple SQS queue subscriptions rather than direct Lambda or HTTP/S subscriptions wherever durability matters — the “fan-out to SQS” pattern gives each consumer its own durable, poll-based buffer with its own visibility timeout and redrive behavior, decoupling the consumer’s processing speed from the publisher entirely, which raw Lambda or HTTP/S subscriptions don’t provide to the same degree.
graph TD
PUB[Publisher Application] --> TOPIC[SNS Topic]
TOPIC -->|filter: orders| Q1[SQS Queue - Order Service]
TOPIC -->|filter: inventory| Q2[SQS Queue - Inventory Service]
TOPIC -->|filter: analytics| Q3[SQS Queue - Analytics Pipeline]
Q1 --> L1[Lambda Consumer]
Q2 --> L2[Lambda Consumer]
Q3 --> FIREHOSE[Kinesis Firehose]
Q1 -.->|exhausted retries| DLQ1[(Dead-Letter Queue)]
Q2 -.->|exhausted retries| DLQ2[(Dead-Letter Queue)]
Fig 2 — Fan-out-to-SQS pattern with independent filter policies and per-consumer dead-letter queues
Cross-account fan-out — a central “event bus” account publishing to consumers owned by other teams’ AWS accounts — is a common pattern for large organizations standardizing on event-driven architecture, implemented via topic policies granting cross-account subscribe permissions, letting each consuming team own and evolve their own SQS queue and processing logic independently of the publishing team’s release cycle.
PDesign Patterns & Anti-patterns
Pattern
Fan-out to SQS with per-consumer filter policies: every downstream consumer owns its own SQS queue subscribed to the shared topic with a narrowly scoped filter policy, decoupling consumer processing speed and failure handling from both the publisher and every other consumer.
Why It Works
Gives each team independent scaling, retry, and dead-letter behavior without any coordination with the publishing team or other consumers.
Anti-pattern
Publishing all messages into a single FIFO message group for a high-throughput application “to keep ordering simple everywhere.”
Consequence
Throughput is capped by the single-group ceiling, and a slow or stuck delivery at the head of that one group delays every subsequent message in the entire application, even unrelated ones.
Anti-pattern
Running production subscriptions with no dead-letter queue configured, relying on the assumption that delivery “basically always works.”
Consequence
Any message that exhausts retries is permanently and silently lost with only an aggregate CloudWatch metric as evidence, with no way to recover or replay it after the fact.
BBest Practices & Common Mistakes
Prefer SQS subscriptions over direct HTTP/S for durability
SQS gives you a durable buffer with its own visibility timeout and redrive behavior that raw HTTP/S delivery doesn’t provide.
Design FIFO message group keys around real parallelism needs
Use per-entity keys (per account, per order, per resource) rather than a single global group, to avoid capping throughput unnecessarily.
Treating “message published” as “all subscribers received it”
Delivery is independent per subscription — a successful publish call says nothing about individual subscriber outcomes.
Leaving delivery status logging disabled in production
Without it, diagnosing which specific subscriber is silently failing among many healthy ones is nearly impossible from aggregate metrics alone.
RReal-World & Industry Examples
Netflix has publicly described using SNS fan-out to SQS as a core building block of their event-driven microservices architecture, specifically citing the ability for independent teams to subscribe new consumers to existing topics without requiring any change from the publishing team as a key organizational, not just technical, benefit.
Zillow has discussed using SNS FIFO topics with per-listing message group keys to guarantee that property-update events for a given listing are always processed in the correct order downstream, while updates for different listings process fully in parallel — a direct illustration of message-group key design as a throughput and correctness trade-off made deliberately.
Capital One has cited SNS delivery status logging as essential to meeting internal audit requirements for demonstrating that critical financial event notifications were successfully delivered to every required downstream system, replacing what had been an assumption of delivery with a queryable, per-attempt record.
FFrequently Asked Questions
SSummary and Key Takeaways
Key Takeaways
- Delivery to each subscriber is fully independent — a successful publish confirms nothing about individual subscriber outcomes.
- Message filtering is evaluated server-side by SNS before delivery, keeping subscriber-side logic simple and reducing unnecessary cost.
- FIFO ordering and deduplication are scoped to a message group, not the whole topic — group-key design is a throughput and correctness decision.
- Retry policy is configurable per subscription — the same message can be retried very differently across different subscribers.
- Without a dead-letter queue, exhausted-retry messages are permanently and silently lost — treat missing DLQs as an active reliability gap.
- Fan-out to SQS, rather than direct HTTP/S or Lambda subscriptions, is the standard pattern for durable, decoupled consumer processing.
- Enable delivery status logging in production — aggregate topic metrics alone can mask a single silently failing subscriber among many healthy ones.