Picture this: you go to buy the sneakers, the phone, or the limited-edition action figure you’ve been wanting, and the page says “Out of Stock.” Ten years ago your only option was to keep refreshing the page every hour, hoping to catch it live. Today almost every large e-commerce platform — Amazon, Flipkart, Zara, BestBuy, Nike — offers a small button that says “Notify Me” or “Email me when available.” You click it once, close the tab, and go on with your life. Then, sometimes days later, sometimes seconds later, a push notification, email, or SMS lands in your pocket: “It’s back! Grab it before it’s gone again.”

That tiny feature, from the customer’s point of view, looks trivial. From a systems point of view, it is one of the more interesting real-time, event-driven problems in e-commerce engineering. It touches inventory management, event streaming, fan-out messaging, multi-channel delivery, rate limiting, and extreme scale — all at once.

1.1 A brief history: how stock alerts evolved

Historically, stock alerts started as simple cron jobs: a batch script would run every few hours, query the inventory database for products whose stock count changed from zero to a positive number, join that against a table of “subscribers” for that product, and fire off a batch of emails. This worked fine when a site had thousands of users and dozens of restocks a day. But as e-commerce scaled into hundreds of millions of users and flash-sale culture (Black Friday, festival sales, sneaker drops) became normal, the batch approach became painfully slow and unreliable. A restock during a flash sale might sell out again in 90 seconds — a batch job running every 15 minutes would notify customers about stock that no longer exists, causing anger and lost trust.

This tutorial designs a system firmly in that third era — one that can react within a second or two of a stock change and can do so at the scale of a global marketplace processing over a million relevant requests every single minute.

Let’s define the problem precisely, because in a system design interview, precision here is what separates a strong answer from a vague one.

2.1 Functional requirements

• A logged-in (or guest, via email) customer can click “Notify Me” on any out-of-stock product detail page.
• The moment that product’s stock count moves from 0 to a positive number, every subscribed customer should be notified through their preferred channel(s): push notification, email, SMS, or in-app real-time alert.
• A customer should not be notified twice for the same restock event, and should not be spammed if stock flickers (0 → 1 → 0 → 5) rapidly, e.g., due to concurrent orders during a flash sale.
• Customers can unsubscribe, and subscriptions should expire automatically after a configurable period (e.g., 90 days) to avoid unbounded growth.
• The system must support both single-item restocks and mass restocks (a warehouse receives a huge shipment of 50,000 SKUs at once, e.g., post-holiday replenishment).

2.2 Non-functional requirements

• Scale: the platform must sustain on the order of 1,000,000+ requests per minute (~16,700 requests/second sustained, with bursts 5–10x higher during flash sales) across subscription writes, stock-change events, and notification dispatch combined.
• Latency: end-to-end delivery (stock becomes available → user receives a push/in-app alert) should be under 2–5 seconds at p99 for push/in-app, and under 30–60 seconds for email/SMS (which depend on slower third-party providers).
• Reliability: no lost notifications for high-value subscriptions; the system should be at-least-once for delivery attempts but must dedupe on the client-visible side so users don’t see duplicate alerts.
• Consistency: eventual consistency is acceptable between “actual warehouse stock” and “what the notification system believes,” but the gap should be small (sub-second to low-seconds via CDC).
• Availability: 99.95%+ uptime for the subscription-write path (customers must always be able to register interest) and for the notification-dispatch path.
• Fairness: if 200,000 people are subscribed to a product that restocks with only 500 units, notifications should go out to everyone roughly simultaneously — the system must not favor whoever happens to be processed first in a way that meaningfully changes their odds of purchasing (a fairness/ethics dimension many teams overlook).

2.3 The scale of the problem in numbers

Before diving into architecture, let’s build a shared vocabulary. Each concept below is explained the way you’d explain it to someone who has never built a distributed system before.

3.1 Change Data Capture (CDC)

What: CDC is a technique for watching a database’s internal transaction log (like MySQL’s binlog or PostgreSQL’s WAL — Write-Ahead Log) and turning every insert, update, or delete into a stream of events, without the application having to explicitly “publish” anything.

Why: Without CDC, every service that writes to the inventory table would need to remember to also publish an event — easy to forget, easy to get wrong, and it doubles the chance of a bug causing a write to succeed while the event silently fails. CDC guarantees that if the row changed in the database, an event was produced — no code path can “forget.”

Analogy: Imagine a security camera pointed at a warehouse’s stock ledger book. Every time someone writes a new number in the ledger, the camera notices the pen moving and immediately radios “the number on page 42 changed” to everyone who needs to know — the ledger-writer doesn’t need to remember to call anyone.

Practical example: We use Debezium, an open-source CDC connector, watching the inventory PostgreSQL database. Whenever a row in the inventory_stock table changes such that quantity goes from 0 to a positive value, Debezium emits a structured event to a Kafka topic called stock.updated.

3.2 Event Streaming (Kafka)

What: Apache Kafka is a distributed, append-only log system. Producers write messages (“events”) to named topics; many independent consumers can read the same topic at their own pace, and messages are retained for a configurable window even after being read.

Why: Kafka decouples the inventory system from the notification system. The inventory team never needs to know or care who is listening to stock changes; new consumers (like a future “back-in-stock analytics dashboard”) can be added without touching inventory code at all.

Analogy: Kafka is like a public radio broadcast tower. The DJ (producer) just plays the song (event); anyone with a radio tuned to that station (consumer) hears it, whether it’s 1 listener or 1 million. The DJ never needs to know how many people are listening.

3.3 Fan-out

What: Fan-out means taking one event (one product restocked) and expanding it into many downstream actions (one notification per subscriber, which could be 1 or 500,000 people).

Why it matters here: A single restock event for a wildly popular sneaker can fan out into hundreds of thousands of individual notification jobs within the same second. The system must be designed so this “fan-out storm” doesn’t overwhelm downstream push/email/SMS providers or the databases tracking delivery status.

3.4 Idempotency & Deduplication

What: An idempotent operation produces the same end result no matter how many times it’s applied. Deduplication is the practice of detecting and discarding repeat events/messages before they cause a duplicate side-effect (like sending the same email twice).

Why: Distributed systems built on queues and streams are usually “at-least-once” — a message might be delivered/processed more than once due to retries, consumer rebalances, or network blips. Without idempotency keys, a customer could receive five copies of “it’s back in stock!” for one restock.

Practical example: Every notification job carries a deterministic idempotency key: hash(userId + productId + restockEventId). Before actually sending, the dispatcher checks a Redis SETNX (set-if-not-exists) with a short TTL; if the key already exists, the job is dropped as a duplicate.

3.5 Backpressure & Rate Limiting

What: Backpressure is a mechanism that lets a slow downstream component signal to a fast upstream component “slow down, I can’t keep up,” rather than being overwhelmed and falling over. Rate limiting caps how many requests/notifications a component or user can trigger in a given time window.

Why: Email and SMS providers (SES, Twilio) have hard sending rate limits and per-account quotas. Without backpressure and rate limiting, a fan-out of 500,000 notifications could get the platform’s SES account throttled or even suspended for sending “too fast.”

3.6 Eventual Consistency

What: A consistency model where, given no new updates, all parts of the system will eventually agree on the same value — but at any given instant, some parts may be slightly behind.

Why it’s acceptable here: It’s fine if the notification pipeline’s view of “is this product in stock” lags the actual warehouse database by a few hundred milliseconds. It would not be fine if the actual checkout/payment system used the same loose consistency — that needs strong consistency to avoid overselling. This distinction — strong consistency for money-moving paths, eventual consistency for notification/informational paths — is a key system design insight.

3.7 Consistent Hashing & Sharding

What: Consistent hashing is a technique for mapping keys (like productId) to a fixed set of shards or nodes in a way that, when the number of nodes changes, only a small fraction of keys need to move — unlike naive modulo hashing, where adding one node reshuffles almost everything.

Why it matters here: The Subscription DB and Redis Cluster both grow over time as the catalog and customer base grow. Using consistent hashing (or Redis Cluster’s built-in hash-slot mechanism, which is a variant of the same idea) means we can add new PostgreSQL shards or Redis nodes during a scale-up event without triggering a full data reshuffle that could take the system offline for hours.

Analogy: Imagine a round clock face with 360 marks, and each database shard “owns” an arc of that clock. A product’s ID hashes to a specific mark on the clock, and whichever shard owns that arc stores its data. Adding a new shard just means giving it a small arc carved out of neighbors — everyone else’s arc, and therefore their data, mostly stays put.

3.8 CAP Theorem in this context

What: The CAP theorem states that a distributed system can provide at most two of three guarantees simultaneously during a network partition: Consistency (every read sees the latest write), Availability (every request gets a response), and Partition tolerance (the system keeps working despite network splits between nodes).

How we apply it: For the subscription-write path, we lean toward AP (availability + partition tolerance) — if a customer clicks “Notify Me” during a network blip between regions, we would rather accept the write locally and reconcile later than reject it outright. For the Inventory Service’s actual stock-decrement-on-purchase path (not covered in depth here, but adjacent), the platform instead leans toward CP (consistency + partition tolerance), because overselling a physical product is a much costlier mistake than a slightly delayed notification.

3.9 Exactly-once semantics vs. at-least-once

What: Kafka can be configured for “exactly-once semantics” (EOS) within a single Kafka-to-Kafka pipeline using idempotent producers and transactional writes. However, once a message leaves Kafka and calls an external HTTP API (FCM, SES, Twilio), true exactly-once delivery is no longer achievable, because the acknowledgment from that external call can itself be lost, leaving the caller unsure whether the action actually happened.

Practical example: We enable Kafka’s EOS between the Stream Processor and the notification.request topic (so a Stream Processor crash-and-restart never double-publishes the same subscriber’s notification-request), but rely on application-level idempotency keys (checked via Redis SETNX) for the final hop to external providers, since EOS cannot extend across that boundary.

3.10 Long Polling vs. WebSockets

What: Long polling is a technique where the client holds an HTTP request open until the server has new data to send, then immediately re-opens a new request. WebSockets, by contrast, establish a single persistent, full-duplex connection that both sides can push messages over at any time without re-opening anything.

Why WebSockets win here: For in-app real-time alerts to users actively browsing the site, WebSockets avoid the overhead of constantly re-establishing HTTP connections and let the WebSocket Gateway push a banner the instant a notification-request arrives — typically the fastest channel in the whole system, often delivering in well under a second.

Trade-off: WebSockets require the gateway layer to hold long-lived stateful connections (unlike the stateless request/response services elsewhere in the architecture), which is why the Connection Registry (Redis, mapping userId to the specific gateway node holding their socket) exists — it lets any backend service route a message to the correct node without needing to know which of potentially hundreds of gateway instances holds a given user’s connection.

Below is the full architecture, drawn as a component diagram. Every box names the actual technology or role it plays — including the API Gateway and Load Balancer explicitly, as they sit at the very front door of every request in this system.

4.1 Component-by-component breakdown

Let’s trace exactly what happens inside the system, component by component, for the two core flows: subscribing, and the restock-triggered fan-out.

5.1 Flow A — Customer clicks “Notify Me”

1. The client sends POST /api/v1/subscriptions with the productId and preferred channel(s).
2. The request passes through the CDN (bypassed for POST — CDNs only cache GET/static content), hits the API Gateway which validates the JWT and applies a per-user rate limit (e.g., max 50 subscription requests/minute to stop scripted abuse).
3. The Load Balancer routes the request to a healthy Subscription Service instance using least-connections balancing.
4. The Subscription Service checks for an existing active subscription (to avoid duplicate rows), then writes a new row to the sharded Subscription DB (shard key = hash(productId) % numShards), and asynchronously updates the Redis set subs:{productId} with the userId.
5. The service returns 201 Created to the client, which shows a confirmation toast.

5.2 Flow B — Product restocks and notifications fan out

1. A warehouse management system or the Inventory Service itself updates inventory_stock.quantity from 0 to, say, 500, inside a transaction.
2. Debezium, tailing PostgreSQL’s WAL, immediately emits a structured “row updated” event to the Kafka topic stock.updated, partitioned by productId so all events for one product stay strictly ordered.
3. The Stream Processor (a Kafka Streams application) consumes the event. It checks: was the previous quantity 0 (or null) and is the new quantity greater than 0? If yes, this is a genuine restock transition and the processor proceeds; otherwise it’s a normal stock decrement from a sale, which we ignore for alerting purposes.
4. The processor looks up the Redis set subs:{productId}. If it contains N subscriber userIds, it generates N individual “notification-request” messages, each tagged with a unique restockEventId (so multiple rapid restocks of the same product are distinguishable), and publishes them to the notification.request Kafka topic, partitioned by userId for even worker distribution.
5. The Notification Orchestrator consumes these in parallel across many partitions, checks each user’s channel preferences (push, email, sms, in-app — possibly more than one), and for each chosen channel emits a task to that channel’s dedicated topic (e.g., push.tasks, email.tasks).
6. Each channel worker pool (Push, Email, SMS) pulls from its topic, passes through the Rate Limiter (token-bucket per provider account), performs the idempotency check via Redis SETNX, and calls the external provider API (FCM/APNs, SES, Twilio).
7. The WebSocket Gateway, for users currently connected, receives its own task type and pushes an in-app banner directly over the open socket — this is typically the fastest channel, often under 500ms end-to-end.
8. Delivery status (sent, failed, retrying) is written asynchronously to the Cassandra Notification Log Store for analytics and customer-support lookups.
9. Failed jobs, after a bounded number of retries with exponential backoff, are moved to the Dead Letter Queue for offline inspection or automated re-drive once the downstream provider recovers.

5.3 Sequence diagram — end-to-end restock notification

A subscription record moves through a well-defined lifecycle:

1. Created — user clicks Notify Me; row inserted with status ACTIVE, timestamp, channel preferences, and a default expiry (e.g., createdAt + 90 days).
2. Cached — mirrored into Redis for O(1) lookup during fan-out; cache and DB are eventually consistent, with the DB as source of truth and periodic reconciliation jobs rebuilding the cache from the DB to heal drift.
3. Triggered — a restock event matches this subscription; a notification-request is generated referencing this subscription’s id and the specific restockEventId.
4. Notified — status transitions to NOTIFIED (or, if multi-channel, per-channel sub-status), and the subscription can optionally be marked “consumed” so the same restock doesn’t re-trigger it if flicker occurs after debounce.
5. Expired / Unsubscribed — after the TTL elapses, or on explicit unsubscribe, the record is soft-deleted (status EXPIRED) and later purged by a background job, and it’s simultaneously removed from the Redis set.

6.1 Subscription state machine

A useful design decision here: should a subscription auto-expire after being triggered once, or can a user resubscribe to be notified on every future restock? Most platforms (Amazon included) treat a triggered notification as terminal — if the user still wants alerts for future restocks, they must click “Notify Me” again. This keeps subscriber lists naturally self-pruning and avoids permanently loud subscriber lists ballooning Redis memory forever.

7.1 On the tension between speed and fairness

It’s worth dwelling on that last row, because it’s a genuinely important product-and-engineering conversation, not just a technical footnote. Any asynchronous, at-scale notification pipeline necessarily introduces some delay between “stock became available” and “customer sees the alert and acts on it.” For a product with 50 units and 5,000 subscribers, it is mathematically certain that most subscribers will be notified after the item is already gone, no matter how fast the pipeline is. The right engineering response isn’t to chase impossible zero-latency guarantees; it’s to be transparent in the product experience — showing accurate, live stock counts on the landing page the notification links to, and, for genuinely scarce high-demand items, layering a fair queueing or raffle mechanism (as Nike’s SNKRS app does) on top of the notification system rather than treating “who got notified first” as a de facto purchase-priority mechanism. This is a good example of a case where the best system design answer is not purely technical — it requires recognizing the limits of what any architecture can promise and designing the surrounding product experience to match reality honestly.

The requirement is to comfortably support over one million requests per minute across the system — roughly 16,700 requests/second sustained, with flash-sale bursts reaching 5–10x that (up to ~150,000–170,000 requests/second momentarily). Here’s how each layer scales to meet that.

8.1 API Gateway & Load Balancer tier

Stateless by design, this tier scales horizontally behind an auto-scaling group. At 170k req/s peak, with each gateway node comfortably handling 5,000–10,000 req/s, roughly 20–35 nodes cover peak load, scaled up proactively ahead of known sale events and reactively via CPU/connection-count-based auto-scaling policies otherwise.

8.2 Kafka tier

The stock.updated topic is partitioned (e.g., 128–256 partitions) so that stream processing parallelism scales linearly with partition count. The notification.request topic needs far more partitions (e.g., 512–1024) since fan-out multiplies event volume by average subscribers-per-product, which can be in the tens of thousands during a hot restock. Kafka’s log-based design means write throughput scales near-linearly with broker count and disk throughput, and consumer groups scale by simply adding more consumer instances up to the partition count.

8.3 Redis subscription cache

Deployed as a Redis Cluster with hash-slot sharding across many nodes. A single hot product (a viral restock) can create a “hot key” problem — one Redis slot receiving disproportionate read traffic. We mitigate this by (a) reading the subscriber set once per restock event and caching it briefly in the Stream Processor’s local memory rather than re-reading per-subscriber, and (b) for exceptionally hot products, splitting the subscriber set across multiple sharded keys (e.g., subs:{productId}:{bucket}) so reads parallelize across Redis nodes.

8.4 Fan-out and worker scaling

Worker pools (Push/Email/SMS) auto-scale based on Kafka consumer lag — if the lag on push.tasks grows, Kubernetes’ Horizontal Pod Autoscaler spins up more worker pods, up to the partition count ceiling. Third-party provider rate limits (e.g., FCM allows very high throughput, but SES/Twilio have stricter per-account limits) are respected via the token-bucket Rate Limiter, which queues excess load rather than dropping it, smoothing bursts over a few seconds to a couple of minutes rather than instantaneously blasting a provider.

8.5 Database scaling

The Subscription DB is sharded by productId hash across many PostgreSQL instances (or migrated to a horizontally-scalable store like DynamoDB/Cassandra if write volume grows further), so no single database instance bears the full write load of “Notify Me” clicks nationwide. The Notification Log Store uses Cassandra specifically because its write-optimized, leaderless architecture handles the enormous write volume of “log every notification attempt” far better than a relational database would at this scale.

8.6 The token bucket algorithm, explained

The Rate Limiter’s core algorithm deserves a closer look since it’s a frequently-asked interview topic in its own right. A token bucket is a simple, elegant data structure: imagine a bucket that holds up to $N$ tokens (say, 1,000). Every time we want to send a notification through a given provider, we must first remove one token from the bucket. Tokens are refilled at a steady rate (say, 500 tokens/second) regardless of how many are being consumed. If the bucket is empty when a request arrives, that request waits (or is queued) until a new token becomes available.

Why this shape fits our problem: it naturally absorbs short bursts (the bucket has capacity to send a quick burst up to $N$ tokens instantly) while enforcing a hard steady-state ceiling (the refill rate) that matches exactly what the downstream provider (SES, Twilio) allows contractually. This is superior to a naive fixed-window counter (e.g., “max 30,000 per minute”), which allows a full window’s worth of traffic to burst right at a window boundary, doubling the effective peak rate for a brief moment.

public class TokenBucketRateLimiter {

    private final long capacity;
    private final long refillTokensPerSecond;
    private final AtomicLong availableTokens;
    private volatile long lastRefillTimestamp;

    public TokenBucketRateLimiter(long capacity, long refillTokensPerSecond) {
        this.capacity = capacity;
        this.refillTokensPerSecond = refillTokensPerSecond;
        this.availableTokens = new AtomicLong(capacity);
        this.lastRefillTimestamp = System.nanoTime();
    }

    public boolean tryAcquire() {
        refill();
        long current;
        do {
            current = availableTokens.get();
            if (current <= 0) {
                return false; // no tokens left, caller should queue and retry
            }
        } while (!availableTokens.compareAndSet(current, current - 1));
        return true;
    }

    private void refill() {
        long now = System.nanoTime();
        long elapsedNanos = now - lastRefillTimestamp;
        long tokensToAdd = (elapsedNanos * refillTokensPerSecond) / 1_000_000_000L;

        if (tokensToAdd > 0) {
            long newValue = Math.min(capacity, availableTokens.get() + tokensToAdd);
            availableTokens.set(newValue);
            lastRefillTimestamp = now;
        }
    }
}

In production, this logic runs against a shared Redis instance (using a Lua script for atomicity) rather than local JVM memory, since the Rate Limiter Service runs as many replicated instances and they must all agree on one shared token count per provider account.

• Multi-AZ deployment: every stateful component (Kafka brokers, Redis Cluster, PostgreSQL, Cassandra) is deployed across at least three availability zones, with Kafka replication factor 3 and Cassandra replication factor 3 using a quorum consistency level for reads/writes on critical paths.
• No single point of failure at the edge: API Gateway and Load Balancer tiers run as auto-scaled fleets with health checks; unhealthy nodes are automatically removed from rotation.
• Circuit breakers: calls from the Notification Orchestrator (and channel workers) to external providers (FCM, SES, Twilio) are wrapped in circuit breakers (e.g., Resilience4j in Java) so that if a provider starts failing, we stop hammering it and instead fail fast, queueing jobs for later retry rather than piling up threads waiting on a dying dependency.
• Dead Letter Queues: any notification job that exhausts its retry budget lands in a DLQ topic rather than being silently dropped, enabling manual or automated re-drive once the root cause is fixed.
• Idempotent consumers: since Kafka delivers at-least-once, every consumer (Stream Processor, Orchestrator, channel workers) is written to be safely re-runnable on the same message without duplicating customer-visible side effects.
• Graceful degradation: if Redis subscription cache is fully down, the Stream Processor falls back to a (slower) direct PostgreSQL read path rather than failing the entire restock-notification flow.
• Disaster recovery: Kafka topics have cross-region mirroring (MirrorMaker 2) to a secondary region; PostgreSQL uses continuous WAL archiving and point-in-time-recovery backups; game-day drills simulate full primary-region failure to validate the runbook.

9.1 Consensus & leader election underneath the platform

Several components in this architecture quietly rely on consensus algorithms to stay coordinated, and understanding this is often what separates a candidate who has “used Kafka” from one who understands it. Modern Kafka clusters use KRaft (a Raft-based consensus protocol built into Kafka itself, replacing the older ZooKeeper dependency) to elect a controller node responsible for partition leader assignment and cluster metadata. Similarly, PostgreSQL high-availability setups typically use tools like Patroni, which itself relies on a consensus store (etcd, which uses Raft) to safely elect a single primary and avoid two nodes believing they’re both the primary at once — a dangerous “split-brain” scenario that could cause the Subscription DB to accept conflicting writes on different nodes simultaneously.

Why this matters for stock alerts specifically: if a Kafka partition leader fails mid-fan-out for a viral restock, Raft-based leader election typically completes within a few hundred milliseconds to a couple of seconds, during which writes to that specific partition briefly pause. Designing partition counts and replication factors thoughtfully (replication factor 3, min in-sync replicas of 2) ensures the system tolerates a single broker failure without losing any already-acknowledged stock-change events, and resumes fan-out automatically once a new leader is elected — no human intervention required for the common case of a single-node failure.

9.2 Failure recovery scenarios worth rehearsing

• Debezium/CDC connector crashes: Kafka Connect automatically restarts the connector task on another worker node, resuming from the last committed WAL offset, so no stock-change events are lost — only delayed by the restart time (typically a few seconds).
• A single channel worker pool (e.g., SMS) fails entirely: because channels are isolated via the bulkhead pattern, push and email notifications continue unaffected; SMS tasks simply accumulate in their Kafka topic until the pool recovers, then drain normally (Kafka’s retention window, typically 3–7 days, comfortably covers any realistic outage).
• Redis subscription cache node failure: Redis Cluster automatically promotes a replica to primary for the affected hash slots within seconds; any requests during that brief window fail over to the PostgreSQL fallback path described earlier.

• Authentication & Authorization: the API Gateway validates JWTs (short-lived access tokens plus refresh tokens) before any request reaches the Subscription Service; guest email-based subscriptions require email verification (double opt-in) to prevent someone subscribing another person’s email address without consent.
• Rate limiting & abuse prevention: per-user and per-IP rate limits at the Gateway prevent scripted mass-subscription abuse (e.g., a bot subscribing millions of fake emails to a product to inflate perceived demand or to spam a competitor’s inbox).
• PII protection: emails and phone numbers used for notification are encrypted at rest (column-level encryption in PostgreSQL / Cassandra) and access to raw contact info is scoped to the specific services that need it (Email/SMS workers), not broadly readable across the platform.
• Provider credential isolation: API keys for FCM/APNs/SES/Twilio are stored in a secrets manager (AWS Secrets Manager / HashiCorp Vault), never in code or config files, and rotated on a schedule.
• Unsubscribe compliance: every email includes a one-click unsubscribe link (CAN-SPAM/GDPR requirement); SMS includes “reply STOP to opt out,” both wired back into the Subscription Service to immediately deactivate the relevant record.
• mTLS between internal services: a service mesh (Istio) enforces mutual TLS between the Orchestrator, workers, and databases, so traffic within the cluster is encrypted and mutually authenticated, limiting blast radius if one pod is compromised.
• Input validation: productId and userId inputs are strictly validated to prevent injection attacks against the Subscription DB, and all queries use parameterized statements.

• Metrics (Prometheus + Grafana): subscription write rate, Kafka consumer lag per topic/partition, end-to-end notification latency (p50/p95/p99) from restock event to delivery confirmation, per-channel delivery success/failure rate, Redis hit/miss ratio on the subscription cache, and rate-limiter queue depth per provider.
• Logging (ELK / EFK stack): structured JSON logs from every service, correlated by a shared restockEventId and traceId, so an engineer can search “all logs for this restock event” across every microservice in one query.
• Distributed tracing (Jaeger/Zipkin): a single trace spans from the Debezium event ingestion through the Stream Processor, Orchestrator, Rate Limiter, and channel worker, making it possible to see exactly where time is spent for a slow notification.
• Alerting: PagerDuty/OpsGenie alerts fire on: Kafka consumer lag exceeding a threshold (signals the pipeline is falling behind), notification p99 latency breaching SLA, DLQ growth rate spiking, and third-party provider error-rate spikes (early warning of an FCM/SES/Twilio outage).
• Business dashboards: beyond pure ops metrics, product teams track “notify-to-purchase conversion rate” — of everyone notified, what fraction actually completed a purchase within, say, 1 hour — which is a strong signal of whether the notification speed and channel mix are effective.

All stateless services (Subscription Service, Stream Processor, Notification Orchestrator, channel workers, WebSocket Gateway) run as containerized workloads on Kubernetes, packaged via Docker images built and pushed through a CI/CD pipeline (GitHub Actions or Jenkins → container registry → ArgoCD for GitOps-style deployment).

• Auto-scaling: Horizontal Pod Autoscalers scale channel workers based on Kafka consumer lag (via KEDA — Kubernetes Event-Driven Autoscaling) rather than plain CPU, since these workloads are I/O-bound waiting on third-party providers.
• Canary and blue-green deployments: new versions of the Stream Processor or Orchestrator are rolled out to a small percentage of traffic first, with automated rollback if error rates or latency regress, since a bug in restock-matching logic could otherwise silently under- or over-notify millions of people.
• Multi-region: the system is deployed across at least two AWS regions for global e-commerce brands, with Kafka MirrorMaker 2 replicating topics cross-region, and geo-aware routing at the CDN/DNS layer (Route 53 latency-based routing) directing users to their nearest region.
• Infrastructure as Code: the entire stack (Kafka cluster, Redis Cluster, Kubernetes node pools, IAM policies) is defined in Terraform, enabling reproducible environments and disaster-recovery region rebuilds.
• Cost optimization: Kafka broker and Kubernetes worker node pools use a mix of reserved instances for baseline load and spot/preemptible instances for stateless worker pools that can tolerate interruption (with jobs safely re-queued via Kafka’s consumer-group rebalancing).

Operational maturity for a system like this is measured less by how it behaves on an ordinary Tuesday and more by how gracefully it handles the extraordinary Friday — a viral product drop, a scheduled flash sale, or an unplanned regional outage. This is why capacity planning for this platform is done against a modeled worst-case scenario (a single SKU restocking with several million active subscribers, timed to coincide with a broader site-wide traffic spike) rather than simply extrapolating from average daily load, and why the deployment pipeline treats load and chaos testing as a required, non-optional gate before any major sale event, not an occasional nice-to-have exercise.

13.1 Why sharded PostgreSQL for subscriptions

Subscriptions are structured, relational-shaped data (userId, productId, channel, timestamps, status) that benefit from strong schema guarantees and ACID transactions on individual writes. Sharding by hash(productId) spreads both write load and, more importantly, the read load during fan-out reconciliation across many independent database instances, avoiding a single hot primary.

13.2 Why Cassandra for the notification log

The notification log is an append-heavy, rarely-updated dataset (billions of rows over time: one row per notification attempt per user per channel) queried mostly by simple keys (userId, or notificationId) rather than complex joins. Cassandra’s leaderless, masterless architecture handles enormous write throughput with linear horizontal scalability, which is exactly the shape of this workload — unlike PostgreSQL, which would require increasingly complex sharding to sustain the same write rate.

13.3 Why Redis for the subscription cache and connection registry

Both use cases need sub-millisecond lookups on simple key structures (sets and hash maps) under very high read concurrency during fan-out bursts — exactly Redis’s sweet spot. Redis Cluster’s hash-slot sharding lets us scale horizontally as subscriber-set volume grows.

13.4 Load balancing strategy

The L7 Load Balancer uses least-connections balancing for the Subscription Service (since request processing time can vary slightly based on DB shard latency) and the Kafka partition assignment itself acts as the “load balancer” for the Stream Processor and Orchestrator tiers — each consumer instance is assigned a subset of partitions, which is a natural, built-in load-balancing mechanism for stream-processing workloads.

Below is a representative Java implementation sketch of the Subscription Service’s core endpoint, using Spring Boot, showing the write-through cache pattern discussed above.

@RestController
@RequestMapping("/api/v1/subscriptions")
public class SubscriptionController {

    private final SubscriptionRepository subscriptionRepository;
    private final RedisTemplate<String, String> redisTemplate;

    public SubscriptionController(SubscriptionRepository subscriptionRepository,
                                   RedisTemplate<String, String> redisTemplate) {
        this.subscriptionRepository = subscriptionRepository;
        this.redisTemplate = redisTemplate;
    }

    @PostMapping
    public ResponseEntity<SubscriptionResponse> createSubscription(
            @RequestHeader("X-User-Id") String userId,
            @Valid @RequestBody SubscriptionRequest request) {

        // Prevent duplicate active subscriptions for the same product
        boolean alreadyExists = subscriptionRepository
                .existsActiveByUserAndProduct(userId, request.getProductId());

        if (alreadyExists) {
            return ResponseEntity.status(HttpStatus.CONFLICT)
                    .body(new SubscriptionResponse("Already subscribed", null));
        }

        Subscription subscription = new Subscription(
                UUID.randomUUID().toString(),
                userId,
                request.getProductId(),
                request.getChannels(),
                SubscriptionStatus.ACTIVE,
                Instant.now(),
                Instant.now().plus(90, ChronoUnit.DAYS)
        );

        subscriptionRepository.save(subscription);

        // Write-through: update the fast Redis lookup used during restock fan-out
        String cacheKey = "subs:" + request.getProductId();
        redisTemplate.opsForSet().add(cacheKey, userId);
        redisTemplate.expire(cacheKey, Duration.ofDays(95));

        return ResponseEntity.status(HttpStatus.CREATED)
                .body(new SubscriptionResponse("Subscribed successfully", subscription.getId()));
    }
}

And here is the Kafka Streams logic (Java) that detects genuine restock transitions and fans out notification-request events:

public class RestockDetectionProcessor implements Processor<String, StockChangeEvent, String, NotificationRequest> {

    private ProcessorContext<String, NotificationRequest> context;
    private final RedisSubscriptionLookup subscriptionLookup;

    public RestockDetectionProcessor(RedisSubscriptionLookup subscriptionLookup) {
        this.subscriptionLookup = subscriptionLookup;
    }

    @Override
    public void init(ProcessorContext<String, NotificationRequest> context) {
        this.context = context;
    }

    @Override
    public void process(Record<String, StockChangeEvent> record) {
        StockChangeEvent event = record.value();

        boolean isRestockTransition = event.getPreviousQuantity() <= 0
                && event.getNewQuantity() > 0;

        if (!isRestockTransition) {
            return; // ignore normal sale decrements and other unrelated updates
        }

        String productId = event.getProductId();
        String restockEventId = UUID.randomUUID().toString();

        Set<String> subscriberIds = subscriptionLookup.getSubscribers(productId);

        for (String userId : subscriberIds) {
            NotificationRequest notification = new NotificationRequest(
                    userId, productId, restockEventId, Instant.now());

            context.forward(new Record<>(userId, notification, context.currentSystemTimeMs()));
        }
    }
}

15.1 Patterns used

15.2 Anti-patterns to avoid

19.1 The one idea to remember

The essence of this system is a small, disciplined transformation: an inventory row changing from 0 to a positive number becomes, within seconds, a personal alert on the phone of every customer who cared enough to raise their hand. Everything in the architecture — CDC to guarantee no missed events, Kafka to buffer bursts and preserve order, Redis to answer “who cares?” in microseconds, a rate-limiter to protect the downstream providers that actually reach a human, and a bulkhead per channel so one slow provider doesn’t drag the others down — exists in service of making that transformation happen at a scale, speed, and reliability that a simple polling loop or nightly batch job simply cannot match.