Every time your phone buzzes with a message, a price drop alert, a ride-arrival notice, or a friend’s comment on a photo, a small but extraordinarily complex piece of distributed-systems engineering just fired. Push notifications feel instantaneous and effortless to the end user, but underneath that single buzz is a pipeline that has to track billions of devices, know exactly which network path is currently valid for each one, respect two entirely different vendor ecosystems with different rules, retry failures without flooding anyone, and do all of this within a latency budget of a few seconds — at a scale of tens of billions of notifications a day for the largest platforms in the world.

Push notifications did not always exist. In the earliest smartphone era, apps that wanted to alert users had only one option: polling. An app would wake up periodically, open a network connection, ask the server “anything new for me?”, and go back to sleep. This was catastrophic for battery life — imagine every app on your phone independently waking the radio every few minutes. Apple recognized this in 2009 and introduced the Apple Push Notification service (APNs), a centralized, persistent connection that the operating system itself maintained with Apple’s servers. Instead of every app polling, the OS held one long-lived connection, and Apple multiplexed notifications for every app through it. Google followed with Cloud to Device Messaging (C2DM) in 2010, which evolved into Google Cloud Messaging (GCM) in 2012, and was eventually rebuilt and rebranded as Firebase Cloud Messaging (FCM) in 2018 — the current standard for Android push delivery.

The core insight that both platforms converged on is the same: push delivery must be centralized at the OS/vendor level, not left to individual app developers to solve with their own polling or socket logic. Your backend never talks directly to a user’s phone. It talks to Apple or Google, and Apple or Google — who already maintain a persistent, battery-optimized connection to that device — deliver the final hop. This is the single most important architectural fact to internalize before designing anything else: you are building a system that talks to two gatekeeper platforms, not directly to a billion phones.

It’s worth pausing on why a single persistent connection per device, multiplexed across every installed app, is such a big architectural win. Before this model, if you had fifty apps on your phone and each maintained its own always-on socket to its own backend, you would have fifty separate radio wake-ups, fifty separate TCP/TLS handshakes to keep alive, and fifty independent points of battery drain — the radio is, by a wide margin, the most power-hungry component on a mobile device, and keeping it active is far more expensive than keeping the CPU idle. By centralizing this at the OS level, Apple and Google turned an O(n) battery cost (n = number of apps) into an O(1) cost: one connection, one radio wake pattern, shared by every app on the device. This single design decision is why push notifications are viable at all on battery-constrained hardware, and it’s the reason your system, no matter how sophisticated, must always terminate at APNs or FCM rather than trying to reinvent this transport layer.

The other historical thread worth knowing is that early push systems were unreliable and inconsistent across OEMs on Android specifically, because for years many Android manufacturers layered aggressive battery-saving customizations (task killers, background-restriction lists) on top of stock Android that could silently prevent FCM’s underlying persistent connection from staying alive. This fragmentation is part of why FCM’s current architecture leans so heavily on Doze-mode-aware priority levels and why understanding “high priority” versus “normal priority” is not a minor implementation detail — it directly determines whether the OS will wake the device’s network stack at all.

At a billion-device scale, the push system is really several cooperating subsystems, each independently scalable. Let’s name them before we go deeper.

2.1 Component Responsibilities

2.2 Why Separate Fan-Out From Dispatch?

A subtle but important design decision is splitting “explode a logical send into millions of jobs” (Fan-Out) from “speak the wire protocol to a specific gateway” (Dispatch). These have very different scaling characteristics and failure modes. Fan-Out is CPU- and I/O-bound on reading the token registry and writing to queues — it scales with audience size. Dispatch is bound by external rate limits imposed by APNs and FCM — it scales with how fast the gateway will accept traffic, which is largely fixed regardless of how much internal compute you throw at it. If these were combined into one service, a burst in audience size (Fan-Out load) would directly compete for the same process resources as gateway-bound work (Dispatch load), making it much harder to reason about capacity and to apply targeted backpressure. Keeping them separate means you can scale Fan-Out horizontally to handle audience resolution bursts while Dispatch stays sized to exactly what APNs/FCM will tolerate — and a slow gateway never starves the ability to keep resolving and queueing new audiences.

2.3 The Device Token Registry as the System’s Center of Gravity

Every other component in this architecture is, in some sense, in service of keeping the token registry accurate and using it correctly. If it’s stale — holding tokens for uninstalled apps, expired app installations, or devices that rotated their token weeks ago — you pay for it in three ways: wasted dispatch capacity spent on doomed sends, inflated cost (both compute and any per-message vendor billing where applicable), and misleading acceptance-rate metrics that make the system look less healthy than it actually is, or worse, mask a real problem underneath the noise. This is why the design deliberately treats every gateway error response as an immediate, synchronous trigger to update the registry rather than a “clean it up later” batch job — token hygiene is a real-time concern, not a housekeeping afterthought.

Understanding push at scale means understanding the two gatekeepers in real detail, because your entire dispatch layer’s design is dictated by their constraints.

3.1 Apple Push Notification service (APNs)

APNs uses HTTP/2, which allows many concurrent streams over a single long-lived, mutually-authenticated TCP connection. Your dispatch worker authenticates either with a provider certificate or, more commonly today, with a JWT signed with a p8 authentication key tied to your Apple Developer Team ID. Each notification is sent as an HTTP/2 POST to a device-token-specific path, with the JSON payload in the body and metadata (priority, expiration, topic/bundle-id, push type) in HTTP headers.

• Payload limit: 4 KB for standard notifications, 5 KB for VoIP push types — treat as configuration, not a hard-coded constant, since Apple has moved these over the years.
• Connection model: multiplexed HTTP/2 — many notifications per connection, few connections needed even for high throughput.
• Priority: APNs supports “immediate” delivery (priority 10, wakes the device) and “power-considerate” delivery (priority 5, batched by the OS to save battery).
• Feedback: APNs responds synchronously per stream with an HTTP status. A 410 Gone or specific reason string like BadDeviceToken or Unregistered tells you the token is dead and must be purged from your registry immediately — this is your primary signal for token hygiene.

3.2 Firebase Cloud Messaging (FCM)

FCM uses an HTTP v1 REST API (the legacy XMPP/HTTP endpoints are deprecated) authenticated with OAuth2 tokens derived from a service account. Each request targets a registration token, a topic, or a condition (a boolean expression over topics).

• Payload limit: around 4 KB for the data payload.
• Message types: notification messages (OS renders them automatically, even if the app is killed) versus data messages (delivered to the app’s code, which decides what to show — essential for custom rendering, badge counts, or silent background sync).
• Delivery priority: “high” wakes the device and delivers immediately; “normal” is batched, especially under Android’s Doze mode and App Standby power-saving states.
• Feedback: a failed send returns an error code such as UNREGISTERED or INVALID_ARGUMENT. Like APNs, this is your signal to invalidate the stored token.

3.3 The Fundamental Platform Asymmetry

This is a favorite interview probe: the two platforms are not symmetric, and a naive design that treats them identically will misbehave in production.

3.4 Connection Pooling and Throughput Engineering

Because APNs multiplexes many notifications over few HTTP/2 connections, your dispatch workers should maintain a small, warm pool of long-lived, authenticated connections per provider credential rather than opening a new connection per notification — connection setup (TCP handshake, TLS negotiation, HTTP/2 settings exchange) is orders of magnitude more expensive than sending an already-established stream. The practical pattern is: a fixed-size connection pool per worker process, health-checked and replaced on error, with in-flight stream concurrency capped below APNs’ per-connection stream limits so you never trigger a forced connection reset. FCM, being a stateless REST API, doesn’t multiplex in the same sense, but the same principle applies to HTTP connection reuse (keep-alive) and OAuth2 token caching — refreshing an OAuth token on every single request would add unnecessary latency and load on Google’s auth infrastructure at billion-device volumes.

3.5 Handling Silent / Background Pushes Correctly

Both platforms offer a “silent” push type intended to wake the app in the background without showing a visible alert — useful for syncing data ahead of the user opening the app. It’s critical to understand that this category has explicitly weaker guarantees than user-visible pushes: iOS in particular aggressively throttles background pushes based on the app’s recent usage patterns, battery level, and Low Power Mode state, and reserves the right to defer or drop them entirely. A common design mistake is architecting a feature (e.g., “always have fresh content ready when the user opens the app”) around an assumption that silent push is reliable. The safe pattern is to treat silent push as a best-effort optimization layered on top of a pull-based fallback (the app still checks for fresh data on foreground), never as the sole delivery mechanism for anything the product depends on.

Trace a single notification from trigger to phone screen. This end-to-end lifecycle view is exactly what interviewers want you to whiteboard.

4.1 Lifecycle Stages

1. Trigger: an internal event (order shipped, friend request, price alert) or a scheduled campaign initiates the send.
2. Composition: template resolution, localization, personalization tokens filled in, payload built and validated against platform size limits.
3. Targeting: the logical audience is resolved into a token stream, paginated to avoid unbounded memory use — this is where segmentation queries, consent checks, and quiet-hours logic run.
4. Enqueue: jobs land on a durable queue (typically Kafka), partitioned by platform and possibly by shard of device ID for parallelism.
5. Dispatch: platform-specific workers pull batches, open/reuse connections, and send.
6. Acknowledgment: the gateway responds per-message (APNs) or per-batch (FCM), and the worker updates a delivery ledger.
7. Retry or Dead-letter: transient failures (rate limiting, timeouts) go back onto a delayed retry queue with exponential backoff; permanent failures (bad token) trigger token invalidation.
8. Final hop: the OS-level push service delivers to the device if online, or stores the notification (subject to platform-specific TTL) for delivery when the device reconnects.

5.1 Key Trade-offs

Let’s ground this in real numbers. Assume a billion registered devices, split roughly 60/40 between Android and iOS, and a platform that sends an average of 5 notifications per device per day across all campaigns and transactional triggers.

6.1 Where the Bottlenecks Actually Are

• Targeting query fan-out: resolving “all users matching X” against a billion-row store must be paginated and index-backed, never a single unbounded scan.
• Gateway connection limits: APNs and FCM both enforce their own rate limits and connection caps per provider identity — your worker fleet must respect these via token-bucket rate limiting per gateway credential, and shard credentials/apps to parallelize beyond a single limit.
• Queue partitioning: Kafka partitions should be keyed to spread load evenly (e.g., by hash of device ID) while preserving enough ordering guarantees where needed (e.g., don’t reorder a “message deleted” push after “message received”).
• Retry storms: naive immediate retries during a gateway outage can turn a transient blip into a self-inflicted DDoS on your own queue and worker fleet — exponential backoff with jitter is mandatory, not optional.

6.2 Scaling Techniques

6.3 Capacity Planning Worked Example

Suppose product wants to send a single campaign notification to 200 million iOS devices within a 5-minute window (a common “big moment” launch pattern). That’s 200,000,000 / 300 seconds ≈ 666,000 notifications/second sustained for that window, entirely against a single gateway. Since a realistic per-connection throughput on APNs (accounting for stream concurrency limits and network round trips) might be on the order of a few thousand notifications per second per warm connection pool, you need to horizontally scale dispatch workers, each maintaining their own connection pools, to collectively reach that throughput — while making sure your provider credential’s overall rate allowance from Apple isn’t exceeded. This is exactly why the Fan-Out layer needs configurable pacing: product teams should be able to specify “deliver this over 5 minutes” versus “deliver this over 60 minutes,” and the system translates that into an enqueue rate that respects gateway ceilings, rather than dumping the entire audience onto the queue instantly and hoping downstream keeps up.

6.4 Read/Write Amplification From Retries

It’s worth explicitly modeling how retries multiply load. If your permanent-failure rate is 2% (dead tokens) and your transient-failure rate is 5% (timeouts, momentary rate limiting), and each transient failure is retried up to 3 times with backoff, your effective dispatch volume isn’t 100% of the audience — it’s closer to 100% + (5% × up to 3 retries) ≈ 115% of the logical audience size in worst-case gateway conditions. At a billion-device scale this is a meaningfully different capacity number than the naive “one send per device” estimate, and it’s a common source of under-provisioning surprises during incident postmortems.

6.5 CAP Theorem and the Token Registry

The device token registry is a good concrete example for reasoning about CAP theorem trade-offs in an interview. During a network partition between data-center regions, you must choose: reject writes to preserve consistency (CP), or accept writes on both sides and reconcile later, accepting temporary inconsistency (AP). For this specific workload, AP is almost always the right call. A device token being briefly stale after a partition heals is a low-severity problem — worst case, a handful of notifications go to an outdated token and bounce with a clean error that triggers re-registration on the next app launch. Rejecting writes during a partition, on the other hand, would mean actively refusing to register fresh tokens from devices that just installed the app, a much worse and more visible failure. This asymmetry — where the cost of temporary inconsistency is low but the cost of unavailability is high — is exactly the signal that should push you toward an AP-leaning, eventually-consistent design for this particular data store, even though other parts of the same platform (e.g., a payments ledger) might correctly make the opposite choice.

6.6 Cost Optimization at This Scale

At billion-device volumes, both compute and any vendor-side costs compound quickly, so a few disciplined habits pay for themselves. Deduplicating near-identical notifications before fan-out (e.g., collapsing “three people liked your photo” into a single aggregated notification instead of three separate sends) directly cuts dispatch volume. Right-sizing retry policy — capping total retry attempts and retry windows based on the actual half-life of most transient failures rather than retrying indefinitely — avoids paying for work that has a vanishingly small chance of ever succeeding. And regularly auditing the token registry to prune tokens for uninstalled apps (detectable from a pattern of repeated permanent-failure responses) keeps both storage costs and wasted dispatch attempts in check as the registry inevitably accumulates churn over years of operation.

Push delivery sits on the critical path for time-sensitive product experiences (2FA codes, fraud alerts, delivery updates), so the internal pipeline — everything up to the gateway handoff — needs to be designed as if it were a tier-0 system, even though the final hop is inherently best-effort once it leaves your control.

7.1 Reliability Techniques

• Multi-AZ / multi-region deployment of the queue, workers, and token registry so a single data center failure doesn’t stall delivery.
• Durable queueing (replicated Kafka topics) so an in-flight notification survives a worker crash — nothing lives only in a process’s memory.
• Circuit breakers around each gateway integration: if APNs error rates spike, the circuit opens, jobs queue up instead of failing hard, and dispatch resumes once the gateway recovers.
• Dead-letter queues for notifications that exhaust retries, with alerting and periodic reprocessing rather than silent loss.
• Idempotency keys on every notification so retries — even across a full pipeline restart — never double-notify a user for the same logical event.

7.2 Failure Mode Table

7.3 Defining Your Actual SLA

Because the final hop is outside your control, a mature team writes its SLA in terms of what it can actually own: for example, “99.9% of accepted notifications will be handed off to the appropriate gateway within 30 seconds of trigger, for 99.95% of the time our own infrastructure is healthy.” This is a meaningfully different — and honest — commitment than “99.9% of notifications will be delivered to the device,” which no push system on earth can truthfully promise, since it depends on device power state, network connectivity, OS-level user settings, and vendor infrastructure that you don’t operate. Being explicit about this boundary in design docs and stakeholder conversations avoids a very common and painful mismatch between what product/leadership assumes the system guarantees and what it’s actually capable of guaranteeing.

7.4 Disaster Recovery and Backup

Beyond day-to-day availability, plan explicitly for full regional loss. The token registry should be replicated cross-region with a documented, tested failover procedure — not just a theoretical capability. Because the registry is effectively irreplaceable (a lost token cannot be regenerated by your system; it can only be re-obtained the next time the affected device’s app happens to launch and re-register), backups and cross-region replication for this store deserve materially more rigor than for data that can be recomputed or re-derived, such as cached template renders or metrics rollups. Regularly exercising a full regional failover — actually cutting traffic over, not just simulating it on paper — is the only reliable way to know your disaster recovery plan works when you eventually need it for real, since untested runbooks are a well-known source of surprises during actual incidents.

7.5 Consensus and Coordination Concerns

Most of this system is deliberately designed to avoid needing strong distributed consensus, since consensus protocols (leader election, distributed locking) add latency and complexity that this workload’s tolerance for eventual consistency doesn’t require. The one place consensus-like coordination legitimately shows up is in Kafka consumer group partition assignment, where the broker coordinates which worker instance owns which partition — but this is handled by the underlying queue infrastructure, not something the application layer needs to reimplement. A useful interview instinct here is recognizing when a distributed system genuinely needs strong consensus (leader election for a singleton scheduler, for example) versus when eventual consistency and idempotency are sufficient and cheaper — this system leans heavily on the latter, and that’s a deliberate, defensible design choice rather than a shortcut.

Push infrastructure is a high-value target: it can be used to spam users, exfiltrate targeting data, or — worst case — send convincing phishing-style notifications that impersonate the platform.

8.1 Defense in Depth for Campaign Tooling

A frequently underestimated attack surface is not external attackers but internal tooling misuse — a marketing team member with campaign-creation access accidentally targeting “all users” instead of a 10,000-person test segment. The mitigation isn’t purely technical; it’s a combination of guardrails: mandatory audience-size confirmation dialogs above a configurable threshold, staged rollout requirements for any campaign above a certain reach, and automatic circuit-breaking if a single campaign’s send rate exceeds a sane multiple of that team’s historical baseline. Treat “accidental mass notification” as seriously as you’d treat a security incident, because from the user’s perspective, an unwanted 3am notification blast is indistinguishable from a compromise.

Because the final hop is opaque (you hand off to Apple/Google and can’t observe the device directly), your observability has to be unusually rigorous about what it can measure precisely.

9.1 Key Metrics

• Enqueue-to-dispatch latency — time from a notification entering the queue to leaving your dispatch worker.
• Gateway acceptance rate — percentage of sends accepted by APNs/FCM (a 200/success at the API level, distinct from actual device delivery).
• Token invalidation rate — spikes here often reveal a bad app release, a mass reinstall event, or a registry sync bug.
• Retry rate and DLQ volume — a leading indicator of gateway degradation before it becomes user-visible.
• Queue lag — the gap between produced and consumed offsets, the earliest signal of a scaling problem.
• End-to-end p50/p95/p99 latency from trigger to gateway acceptance (device-side delivery confirmation is generally not available, so this is your practical ceiling of visibility).

9.2 Logging & Tracing

Every notification should carry a trace ID stamped at composition time and propagated through targeting, fan-out, and dispatch, so a single notification’s journey can be reconstructed on demand — essential for debugging “why didn’t user X get notified” support escalations, which are common and time-sensitive at this scale.

This system is naturally suited to a cloud-native, multi-region deployment.

• Stateless dispatch workers run as containerized services on an orchestrator (e.g., Kubernetes), autoscaled by queue lag and CPU.
• Regional deployment close to major user populations reduces latency for the internal hops, though the final APNs/FCM handoff is inherently global and vendor-controlled.
• Blue-green or canary rollouts for the dispatch workers, since a bad deploy that mishandles payload formatting can silently degrade delivery across an entire platform.
• Infrastructure as code for queue topics, autoscaling policies, and secrets provisioning, so environment drift doesn’t silently break rate-limit configuration.
• Feature flags around new notification types or targeting logic, allowing gradual rollout and fast kill-switch capability given how visible a bad notification blast is to users.

11.1 Device Token Registry Design

This store needs to comfortably hold a billion-plus rows (accounting for multiple devices per user and historical churn) and support:

• Fast point lookups by user ID (for transactional, per-user sends).
• Efficient range / segment scans for campaign targeting (paginated, index-backed).
• High write throughput for constant token refresh / invalidation churn.

A common real-world approach is a horizontally sharded key-value or wide-column store (the class of system that Cassandra, DynamoDB, or Bigtable represent) partitioned by user ID, with secondary indexes or a separate search-optimized index (e.g., a segment/attribute index) feeding the Targeting Service for campaign-style queries. A single relational database rarely scales cleanly to this row count and write volume without significant sharding work of its own.

11.2 Caching

• Hot-token cache: an in-memory cache (e.g., a distributed cache layer) in front of the registry absorbs read load for frequently-notified users (highly active accounts, popular creators with large follower fan-out).
• Template and rendering cache: compiled, localized templates are cached so the Composer doesn’t re-render identical content per recipient.
• Rate-limit counters: per-gateway-credential token buckets are naturally implemented as cache entries with short TTLs and atomic increment operations.

11.3 Load Balancing

Layer-7 load balancers distribute inbound API traffic across Composer/Gateway instances; internally, Kafka’s partitioning acts as the load-balancing mechanism between Fan-Out and Dispatch — consumer groups scale horizontally simply by adding more worker instances that claim partitions.

The system is naturally decomposed into independently deployable services, each with a narrow contract.

These services communicate asynchronously wherever possible (via the event bus/queue) rather than synchronous request chains, so a slowdown in one stage doesn’t cascade into a timeout storm across the whole pipeline — a classic microservices resilience pattern.

13.1 Patterns That Work

13.2 Anti-Patterns to Avoid

14.1 Operational Runbook Habits Worth Building Early

Teams that operate these systems well tend to converge on a small set of operational habits long before they’re forced to by an incident. They build a dashboard that shows acceptance rate, retry rate, and DLQ volume broken out per platform and per gateway credential, because a problem isolated to one app’s credential set looks very different from a platform-wide outage and calls for a different response. They set explicit alert thresholds on token-invalidation rate, since a sudden spike is one of the earliest signals of either a bad client release or a registry synchronization bug. And they rehearse — through game days or fire drills — what happens when a gateway is degraded but not fully down, since partial degradation (elevated latency, intermittent 5xx responses) is a much more common real-world failure than a clean, total outage, and is exactly the scenario where naive retry logic does the most damage.

14.2 Concurrency Considerations in Dispatch Workers

Within a single dispatch worker process, you’re typically running many concurrent send operations against a shared, bounded connection pool. This is a classic bounded-concurrency problem: too few concurrent in-flight requests underutilizes available gateway throughput, while too many risks exhausting connection pool slots, tripping gateway-side rate limits, or starving other work on the same process (like metrics reporting or health checks). The common solution is a semaphore or worker-pool pattern that caps concurrent in-flight sends per connection to a tuned value, combined with backpressure signals fed from the queue consumer — if the connection pool is saturated, the worker should slow its rate of pulling new jobs off the queue rather than buffering an unbounded number of pending sends in local memory, which risks an out-of-memory crash under sustained backpressure and would lose whatever work was queued in-process at the moment of the crash.

A pattern common across all of these companies is worth naming explicitly: the largest, most reliability-conscious platforms rarely trust APNs/FCM acceptance as their only signal of success. They layer an application-level acknowledgment on top — for a messaging app, this might be the recipient’s client explicitly confirming receipt over its own connection once the app is foregrounded or the socket reconnects; for a delivery-tracking app, it might be reconciling push delivery against whether the user actually opened the app and viewed the updated status shortly after. This “trust but verify” pattern is a direct consequence of the acceptance-versus-delivery distinction covered earlier, and it’s a strong signal in an interview that a candidate understands the system isn’t finished at the gateway handoff — it’s finished when the product outcome the notification was meant to drive actually happens.

A billion-device push notification system is not one big problem — it’s a chain of well-scoped smaller problems, each with a known engineering pattern.

If you take one mental model away from this whole design, let it be this: a billion-device push notification system is not one big problem, it’s a chain of well-scoped smaller problems — resolve an audience without blowing up memory, queue work durably, respect two very different external rate limits, retry safely without amplifying an outage, and keep a massive token store honest in near real time. Each of those sub-problems has a known, well-understood engineering pattern. The skill being tested, in an interview or in production, is recognizing which pattern fits which sub-problem, and being honest about the boundary of what your system can actually promise once a notification leaves your infrastructure and enters Apple’s or Google’s.