Amazon SNS, Beneath the Fan-Out

Amazon SNS, Beneath the Fan-Out

A deep, engineer-level walkthrough of how Amazon SNS actually delivers, retries, and filters messages at scale — the pub/sub delivery model, FIFO topic internals, message filtering evaluation, and the patterns that separate a reliable fan-out architecture from a silent message-loss incident.

If you already know that SNS is “pub/sub for AWS,” you know the tagline, not the delivery engine. The engineering substance in SNS lives in how it fans a single published message out to a heterogeneous set of subscribers with independent retry policies, how message filtering happens server-side before a subscriber ever receives anything it doesn’t need, and how FIFO topics enforce ordering and deduplication guarantees that standard topics deliberately don’t provide. This walkthrough assumes you’ve already created a topic and subscribed something to it; the focus is what’s happening underneath, and where production systems actually lose messages or receive them out of order if these internals aren’t understood.

01

AAdvanced Core Concepts

Skipping “what is pub/sub” — this is the model experienced engineers reach for when reasoning about SNS in production.

Delivery to each subscriber is independent, not transactional

When a message is published to a topic with five subscribers, SNS doesn’t deliver it as a single atomic operation across all five. Each subscription has its own delivery attempt, its own retry policy, and its own success or failure outcome, entirely independent of the others. A message can succeed for four subscribers and exhaust retries for the fifth without any of that affecting the other four — there’s no all-or-nothing guarantee across the fan-out. This independence is a deliberate design choice: it means one slow or broken subscriber can’t block delivery to the rest, but it also means “the message was published” tells you nothing about whether every subscriber actually received it without checking each subscription’s delivery status individually.

Analogy

Think of publishing to an SNS topic like a store manager announcing a memo to five different departments by five separate couriers. Each courier has their own route and their own chance of getting stuck in traffic — the sales department getting the memo instantly doesn’t mean the warehouse got it at all. The manager saying “I sent it” only confirms it left their hands.

Message filtering happens at the subscription, evaluated server-side before delivery

A filter policy attached to a subscription is evaluated by SNS itself against the message’s attributes (or, with body-based filtering, the message payload) before that message is ever sent to the subscriber — filtering is not something the subscriber does after receiving everything. This server-side evaluation is what makes fan-out architectures efficient at scale: a topic can have a hundred subscribers, each with a narrow filter policy, and each one only ever receives the small subset of messages actually relevant to it, without the topic needing to know anything about subscriber-side logic.

FIFO topics provide ordering and deduplication only within a message group

A FIFO SNS topic guarantees strict ordering and exactly-once delivery, but both guarantees are scoped to a MessageGroupId, not the topic as a whole. Messages in different message groups can be delivered in any relative order to each other, even though messages within the same group are strictly ordered. Deduplication uses either an explicit MessageDeduplicationId or, if content-based deduplication is enabled, a SHA-256 hash of the message body, within a 5-minute deduplication interval — publishing the same content twice within that window is treated as a duplicate and only delivered once.

Concept

Independent Per-Subscription Delivery

Each subscriber has its own retry policy and outcome — no cross-subscriber atomicity or shared failure state.

Concept

Server-Side Filter Policy

SNS evaluates filter policies before delivery, so subscribers only ever receive messages matching their declared interest.

Concept

FIFO Message Group Scoping

Ordering and dedup guarantees apply within a MessageGroupId, not across the whole topic.

Concept

Per-Subscription Redrive Policy

A dead-letter queue can be attached per subscription, capturing only that subscriber’s exhausted-retry messages.

02

IInternal Working

What actually happens between “Publish API call” and each subscriber independently receiving (or not receiving) the message.

When a message is published, SNS first evaluates the topic’s access policy to confirm the publisher is authorized. Once accepted, the message is durably stored and the fan-out process begins: SNS enumerates the topic’s active subscriptions and, for each one, evaluates that subscription’s filter policy against the message’s attributes. Subscriptions whose filter doesn’t match are skipped entirely for that message — no delivery attempt is made, and this skip isn’t logged as a failure since it was never intended to be delivered there.

For each matching subscription, SNS attempts delivery according to that protocol’s delivery mechanics: HTTP/S endpoints receive a POST request and must respond with a 2xx status for the attempt to count as successful; SQS queue subscriptions have the message enqueued directly; Lambda subscriptions invoke the function synchronously from SNS’s perspective (though Lambda itself may process asynchronously depending on its own configuration); email and SMS use their respective delivery channels with very different latency and reliability characteristics than the programmatic protocols.

graph TD
    A[Publish API Call] --> B[Access Policy Check]
    B --> C[Message Durably Stored]
    C --> D[Enumerate Active Subscriptions]
    D --> E{Filter Policy Match?}
    E -->|No| F[Skip - No Delivery Attempt]
    E -->|Yes| G[Attempt Delivery Per Protocol]
    G --> H[SQS: Enqueue Message]
    G --> I[Lambda: Synchronous Invoke]
    G --> J[HTTPS: POST with 2xx Expected]
    H --> K[Success]
    I --> K
    J -->|2xx| K
    J -->|Non-2xx or Timeout| L[Retry Policy Engaged]
    L -->|Exhausted| M[Dead-Letter Queue if Configured]
        

Fig 1 — Fan-out delivery pipeline showing per-subscription filtering and independent delivery outcomes

!
Gotcha

An HTTP/S subscriber that responds with a 2xx status but then fails to actually process the message internally looks identical to a successful delivery from SNS’s perspective — SNS has no visibility into what the endpoint does after acknowledging receipt. Reliable HTTP/S subscribers must not return 2xx until processing has genuinely succeeded or been durably queued internally.

03

DData Flow & Lifecycle

A message’s delivery lifecycle for a given subscription follows a retry state machine governed by that subscription’s delivery retry policy, which is configurable independently per subscription and per protocol — a critical detail, since it means the same published message can be retried far more aggressively for one subscriber than another.

1

Immediate Delivery Attempt

SNS attempts delivery to the subscriber’s endpoint as soon as the message passes filtering.

2

Backoff Retry Phase

On failure, SNS retries according to the subscription’s configured backoff policy (linear, exponential, or a custom min/max delay and retry count).

3

Extended Retry (HTTP/S only)

HTTP/S subscriptions support an optional extended retry phase beyond the immediate policy, for tolerating longer subscriber outages.

4

Exhaustion

Once all configured retries are exhausted without success, the message is considered undeliverable to that specific subscription.

5

Dead-Letter Capture (if configured)

If a redrive policy with a dead-letter queue is attached to that subscription, the exhausted message is sent there; otherwise it is permanently lost with only CloudWatch metrics as evidence it ever failed.

For FIFO topics, this lifecycle interacts with ordering guarantees in an important way: SNS will not deliver a message to a subscribed FIFO-compatible endpoint out of order within its message group, meaning a stuck or slow delivery at the head of a group can delay delivery of subsequent messages in that same group, even while messages in other groups continue flowing normally.

04

TAdvantages, Disadvantages & Trade-offs

Advantages

  • Native fan-out to multiple heterogeneous protocols (SQS, Lambda, HTTP/S, email, SMS) from a single publish call removes the need for custom distribution logic.
  • Server-side message filtering reduces unnecessary invocations and processing cost across large subscriber sets.
  • FIFO topics provide strict ordering and exactly-once delivery guarantees without building a custom sequencing layer.
  • Per-subscription dead-letter queues isolate failure handling so one broken subscriber’s issues don’t require touching the topic or other subscriptions.

Disadvantages / Trade-offs

  • No cross-subscriber delivery atomicity — a “successfully published” message provides no guarantee every subscriber actually received it.
  • Standard (non-FIFO) topics provide no ordering guarantee at all, and at-least-once delivery means duplicate handling must be built into subscriber logic.
  • FIFO topics’ ordering guarantee is scoped to message groups, which requires careful group-key design to actually get the ordering behavior an application needs.
  • HTTP/S subscriber reliability depends entirely on correct 2xx-after-real-processing semantics — a naive implementation can silently drop messages while appearing healthy.
“SNS fans a message out fast — but ‘fast’ and ‘guaranteed everywhere’ are different promises, and only one of them is actually made.”
05

PPerformance & Scalability

Standard SNS topics scale to very high publish throughput with no pre-provisioned capacity, since the service is fully managed and elastic. FIFO topics, by contrast, have a defined throughput ceiling per message group (though overall topic throughput scales with the number of distinct message groups in use), which means an application funneling all traffic into a single message group for “simplicity” caps its own achievable throughput regardless of the topic’s overall capacity — a frequent and avoidable bottleneck.

Message filtering’s performance benefit compounds at scale: a topic with a thousand narrowly-filtered subscribers processing a high publish rate only incurs actual delivery cost and subscriber-side processing for the messages each one cares about, rather than every subscriber processing and discarding irrelevant messages itself — pushing that filtering cost to the managed service rather than replicating filter logic in every subscriber’s code.

5 MIN
FIFO CONTENT-BASED DEDUP INTERVAL
PER-GROUP
FIFO ORDERING & THROUGHPUT SCOPE
INDEPENDENT
RETRY POLICY PER SUBSCRIPTION

Robinhood has publicly discussed designing FIFO SNS message group keys around per-account or per-symbol identifiers specifically to maximize parallel throughput while preserving strict ordering guarantees only where genuinely required — illustrating that message-group key design is itself a scalability decision, not just a correctness one.

06

HHigh Availability & Reliability

SNS itself is a Regional, managed, multi-AZ service with no customer-managed redundancy layer — the reliability engineering that matters is in per-subscription retry and dead-letter configuration, and in cross-Region strategy for genuinely disaster-resilient architectures.

Without a dead-letter queue configured on a subscription, a message that exhausts retries is gone permanently, with CloudWatch’s NumberOfNotificationsFailed metric as the only evidence it happened — there’s no built-in replay mechanism after the fact. This is why production subscriptions handling business-critical events should always have an explicit redrive policy, treating “no DLQ configured” as an active reliability gap rather than an acceptable default. For cross-Region resilience, SNS topics don’t natively replicate across Regions, so multi-Region architectures typically publish to independent topics in each Region from application code aware of the active Region, or use a cross-Region event bus pattern via EventBridge for more complex routing needs.

Reliability pattern used by mature teams

Attach a dead-letter queue to every subscription handling business-critical events, alarm on that DLQ’s message count rather than only on the topic’s failure metric, and build a periodic or on-demand redrive process to reprocess DLQ messages once the root cause is fixed.

07

SSecurity

SNS access control operates through both IAM policies attached to principals and topic policies — resource-based policies attached directly to the topic — with topic policies being the standard mechanism for cross-account publish or subscribe permissions, analogous to the repository-policy pattern used elsewhere in AWS. A topic policy that’s too permissive (allowing sns:Subscribe from any principal, for example) can let an unintended party subscribe an endpoint they control and receive messages meant for a closed set of internal consumers.

Message content encryption at rest is available via SSE with a customer-managed or AWS-managed KMS key, which matters for topics carrying sensitive payloads, though it’s worth noting that SNS message attributes used for filtering are still evaluated in plaintext by the filtering engine even with encryption enabled — encryption protects data at rest and in transit, not the filtering logic’s visibility into attribute values it needs to route messages correctly.

Best Practice

Scope topic policies to specific principals and specific actions (never a wildcard principal for Subscribe or Publish on a topic carrying sensitive data), enable SSE with a customer-managed KMS key for regulated payloads, and audit topic policies periodically since a policy that was correctly scoped at creation can become overly permissive as an organization’s account structure evolves.

08

MMonitoring, Logging & Metrics

SNS publishes per-topic CloudWatch metrics including NumberOfMessagesPublished, NumberOfNotificationsDelivered, and NumberOfNotificationsFailed, but these are topic-level aggregates by default — diagnosing which specific subscription is failing requires enabling delivery status logging per protocol (supported for SQS, Lambda, HTTP/S, Firehose, and application endpoints), which writes detailed per-attempt success and failure logs to CloudWatch Logs, including HTTP response codes for HTTP/S subscribers.

Because delivery status logging is opt-in and has its own cost, many teams under-enable it and then have no diagnostic path when a specific subscriber silently stops receiving messages — enabling it, at least at a sampled rate, for any production subscription is the standard recommendation for maintaining actual delivery visibility rather than relying solely on topic-level aggregate metrics that can mask a single failing subscriber among many healthy ones.

SignalSourcePrimary Use
Topic-level publish/delivery/failure countsCloudWatch topic metricsAggregate health monitoring, alarming
Per-attempt delivery statusDelivery status logging (opt-in, per subscription)Diagnosing a specific failing subscriber
Dead-letter queue depthCloudWatch SQS metrics on the DLQDetecting exhausted-retry messages needing attention
Control-plane changesAWS CloudTrailAuditing topic policy and subscription configuration changes
09

DDeployment & Cloud Architecture

The canonical production fan-out architecture pairs an SNS topic with multiple SQS queue subscriptions rather than direct Lambda or HTTP/S subscriptions wherever durability matters — the “fan-out to SQS” pattern gives each consumer its own durable, poll-based buffer with its own visibility timeout and redrive behavior, decoupling the consumer’s processing speed from the publisher entirely, which raw Lambda or HTTP/S subscriptions don’t provide to the same degree.

graph TD
    PUB[Publisher Application] --> TOPIC[SNS Topic]
    TOPIC -->|filter: orders| Q1[SQS Queue - Order Service]
    TOPIC -->|filter: inventory| Q2[SQS Queue - Inventory Service]
    TOPIC -->|filter: analytics| Q3[SQS Queue - Analytics Pipeline]
    Q1 --> L1[Lambda Consumer]
    Q2 --> L2[Lambda Consumer]
    Q3 --> FIREHOSE[Kinesis Firehose]
    Q1 -.->|exhausted retries| DLQ1[(Dead-Letter Queue)]
    Q2 -.->|exhausted retries| DLQ2[(Dead-Letter Queue)]
        

Fig 2 — Fan-out-to-SQS pattern with independent filter policies and per-consumer dead-letter queues

Cross-account fan-out — a central “event bus” account publishing to consumers owned by other teams’ AWS accounts — is a common pattern for large organizations standardizing on event-driven architecture, implemented via topic policies granting cross-account subscribe permissions, letting each consuming team own and evolve their own SQS queue and processing logic independently of the publishing team’s release cycle.

10

PDesign Patterns & Anti-patterns

PATTERN-01 Recommended
Pattern

Fan-out to SQS with per-consumer filter policies: every downstream consumer owns its own SQS queue subscribed to the shared topic with a narrowly scoped filter policy, decoupling consumer processing speed and failure handling from both the publisher and every other consumer.

Why It Works

Gives each team independent scaling, retry, and dead-letter behavior without any coordination with the publishing team or other consumers.

ANTI-PATTERN-01 Avoid
Anti-pattern

Publishing all messages into a single FIFO message group for a high-throughput application “to keep ordering simple everywhere.”

Consequence

Throughput is capped by the single-group ceiling, and a slow or stuck delivery at the head of that one group delays every subsequent message in the entire application, even unrelated ones.

ANTI-PATTERN-02 Avoid
Anti-pattern

Running production subscriptions with no dead-letter queue configured, relying on the assumption that delivery “basically always works.”

Consequence

Any message that exhausts retries is permanently and silently lost with only an aggregate CloudWatch metric as evidence, with no way to recover or replay it after the fact.

11

BBest Practices & Common Mistakes

Best Practice

Prefer SQS subscriptions over direct HTTP/S for durability

SQS gives you a durable buffer with its own visibility timeout and redrive behavior that raw HTTP/S delivery doesn’t provide.

Best Practice

Design FIFO message group keys around real parallelism needs

Use per-entity keys (per account, per order, per resource) rather than a single global group, to avoid capping throughput unnecessarily.

Common Mistake

Treating “message published” as “all subscribers received it”

Delivery is independent per subscription — a successful publish call says nothing about individual subscriber outcomes.

Common Mistake

Leaving delivery status logging disabled in production

Without it, diagnosing which specific subscriber is silently failing among many healthy ones is nearly impossible from aggregate metrics alone.

12

RReal-World & Industry Examples

Netflix has publicly described using SNS fan-out to SQS as a core building block of their event-driven microservices architecture, specifically citing the ability for independent teams to subscribe new consumers to existing topics without requiring any change from the publishing team as a key organizational, not just technical, benefit.

Zillow has discussed using SNS FIFO topics with per-listing message group keys to guarantee that property-update events for a given listing are always processed in the correct order downstream, while updates for different listings process fully in parallel — a direct illustration of message-group key design as a throughput and correctness trade-off made deliberately.

Capital One has cited SNS delivery status logging as essential to meeting internal audit requirements for demonstrating that critical financial event notifications were successfully delivered to every required downstream system, replacing what had been an assumption of delivery with a queryable, per-attempt record.

13

FFrequently Asked Questions

Q1If a message fails delivery to one subscriber, does SNS stop delivering to others?
No. Each subscription’s delivery is entirely independent — a failure on one subscription has no effect on delivery attempts to any other subscription on the same topic.
Q2Can a standard (non-FIFO) topic guarantee message ordering?
No. Standard topics provide at-least-once delivery with no ordering guarantee at all; only FIFO topics provide strict ordering, and only within a given message group.
Q3Does message filtering reduce cost as well as unnecessary processing?
Yes — since non-matching messages are never delivered to a filtered subscription, you avoid the delivery cost and any downstream invocation cost (like a Lambda invocation) for messages that subscription doesn’t need.
Q4What happens to a message if no subscription’s filter policy matches it?
It’s simply not delivered to any subscriber — this isn’t treated as a failure or logged as an error, since no delivery was ever intended for that message given the filter configuration.
Q5Can I recover a message after it exhausts retries with no dead-letter queue configured?
No. Without a redrive policy pointing to a dead-letter queue, an exhausted message is permanently lost with only aggregate CloudWatch failure metrics as evidence it occurred.
14

SSummary and Key Takeaways

Key Takeaways

  • Delivery to each subscriber is fully independent — a successful publish confirms nothing about individual subscriber outcomes.
  • Message filtering is evaluated server-side by SNS before delivery, keeping subscriber-side logic simple and reducing unnecessary cost.
  • FIFO ordering and deduplication are scoped to a message group, not the whole topic — group-key design is a throughput and correctness decision.
  • Retry policy is configurable per subscription — the same message can be retried very differently across different subscribers.
  • Without a dead-letter queue, exhausted-retry messages are permanently and silently lost — treat missing DLQs as an active reliability gap.
  • Fan-out to SQS, rather than direct HTTP/S or Lambda subscriptions, is the standard pattern for durable, decoupled consumer processing.
  • Enable delivery status logging in production — aggregate topic metrics alone can mask a single silently failing subscriber among many healthy ones.