Imagine you need a specific pair of running shoes tonight for a race tomorrow morning. Waiting two days for home delivery does not work, but driving to five different stores hoping one has your size is a waste of an afternoon. “Buy Online, Pick Up In Store”, almost always shortened to BOPIS (sometimes called “click and collect”), solves exactly this problem: you buy the item online, the system tells you which nearby store has it in stock right now and you walk in an hour later to grab it — often from a dedicated pickup counter, without ever browsing the store shelves yourself.

BOPIS is not a brand-new idea — mail-order and catalog showrooms in the mid-twentieth century let customers order from a catalog and collect from a counter. What changed with modern retail is the expectation of real-time accuracy. In the pre-internet showroom model, a mismatch between the catalog and the shelf was an inconvenience discovered on pickup. Today, a customer expects the app to know, to the minute, whether a store fifteen minutes away actually has their size, their colour, in stock — and getting this wrong at scale erodes trust fast. Large retailers like Walmart, Target and Best Buy built BOPIS into a core pillar of their business specifically because it combines the convenience of online shopping with the immediacy of a physical store, and it became especially critical during periods when customers wanted to minimise time spent browsing inside stores.

The engineering challenge behind this simple promise is what this tutorial covers: how do you keep an online storefront’s view of “is this in stock at store #4521” accurate to within seconds, across potentially tens of thousands of stores, each with its own point-of-sale (POS) system quietly selling the same physical inventory to walk-in customers at the very same moment an online shopper is trying to reserve it? And how do you do all of this while absorbing a scale of traffic — a million requests a minute during peak events — that would bring down a naively designed system in seconds?

Why would a retailer invest in building this instead of simply shipping every online order from a central warehouse? The motivation is a mix of customer expectation and hard economics.

But realising these benefits depends entirely on solving a genuinely hard distributed systems problem: keeping online-visible inventory synchronised with real, physical, constantly-changing stock across many independently operating stores. A handful of specific problems make this hard:

• Each store’s point-of-sale system is a separate, often decades-old, system of record for that store’s physical stock — the online platform does not own the ground truth, it has to continuously ingest it.
• A walk-in customer picking an item off the shelf and an online customer reserving that same item happen completely outside any single transaction boundary — there is no database that can lock both events at once.
• Network and system outages at a single store should degrade gracefully (show that store as temporarily unavailable) rather than either falsely showing stock that is not there, or taking down the whole platform’s search experience.
• At flagship-scale, this system must handle enormous traffic bursts — flash sales, popular product drops, holiday shopping — without buckling, which is why this tutorial explicitly designs for a million requests a minute.

It is worth naming explicitly why this particular system design problem is such a rich interview topic. It combines three genuinely distinct hard problems that most single-purpose systems only need to solve one of: a distributed-inventory consistency problem (Section 8), a saga-based multi-service financial transaction problem (Section 5) and a raw horizontal-scale problem at the specific million-requests-a-minute target this tutorial is built around (Section 10). A candidate who addresses all three coherently — and can clearly explain how the design decisions for one (like sharding by store ID) reinforce the others rather than working against them — demonstrates a level of systems thinking that goes well beyond describing any one of these problems in isolation.

3.1 Store Inventory Record

What it is: A per-store, per-SKU record tracking how many units of a specific product are believed to be physically present at a specific store, including how many are soft-reserved for pending online orders. Why it exists: Online availability decisions cannot be made against a single global stock number — the same SKU can be in stock at one store and sold out at another, ten minutes’ drive away.

3.2 Soft Reservation

What it is: A short-lived hold on a specific unit of store stock, made the moment a customer starts checkout, before payment is confirmed. Why it exists: Without it, two customers could both complete checkout for the last unit; a soft reservation buys enough time to complete payment safely and automatically expires if checkout is abandoned. Simple analogy: It is like a waiter placing a “reserved” card on a restaurant table for ten minutes while your party finishes parking the car — if you never show up, the table goes back into the pool.

3.3 Inventory Sync Lag

What it is: The delay between a real-world event at a store (an item sold at the register, a new shipment stocked on the shelf) and that event being reflected in the online platform’s inventory view. Why it exists as a concept to manage: Zero lag is physically impossible across a distributed network of independent store systems; the design goal is to minimise and bound this lag and to design the customer experience to tolerate the small amount that remains.

3.4 Pickup Window

What it is: The time period during which a store commits to holding a prepared order for the customer to collect, after which the reservation is released and the item goes back into sellable stock. Example: A common pickup window is three to five days; after that, an unclaimed order is automatically refunded and re-shelved.

3.5 Store Fulfilment Queue

What it is: The ordered list of pending BOPIS orders a specific store’s staff need to pick, pack and stage for pickup, typically surfaced through a dedicated associate-facing app. Why it exists: Store staff need a clear, prioritised task list — without it, online orders would compete invisibly and chaotically with in-aisle customer service duties.

Let us lay out the full system, from client apps down to the store integration layer. As with any high-scale system, we name the Load Balancer and API Gateway explicitly as their own components — they are not implementation details to wave at, they are the components that make a million-requests-a-minute scenario survivable at all.

4.1 Client Layer

Three distinct clients hit this system: the customer-facing mobile and web apps and the in-store associate app running on POS-connected devices. All three go through the same front door — the Load Balancer and API Gateway — which keeps authentication, rate limiting and routing consistent regardless of which client is calling.

4.2 CDN

What it is: A globally distributed cache for static content — product photos, store location assets, app bundles. Why it exists: At the traffic volumes this system is designed for, serving product images from a single origin would be an immediate bottleneck; the CDN absorbs the overwhelming majority of read traffic before it ever reaches application servers.

4.3 Load Balancer

What it is: The component distributing incoming traffic across many API Gateway and service instances, using health checks to route around unhealthy nodes. Why it exists here specifically: A million requests a minute is roughly sixteen thousand requests a second sustained, with real-world traffic being far spikier than a flat average — the load balancer, paired with aggressive auto-scaling, is what turns “sixteen thousand requests a second” from a terrifying number into a manageable, horizontally distributed workload.

4.4 API Gateway

What it is: The single entry point handling authentication, rate limiting and routing to the correct backend microservice. Why it exists: Centralising rate limiting here is particularly important for BOPIS — it is the layer that protects the Inventory Aggregator Service from being overwhelmed by, for example, a bot repeatedly polling stock levels at a popular store during a product launch.

4.5 Core Microservices

4.6 Store Integration Domain

4.7 Data and Messaging Layer

Postgres, sharded by store region, holds durable order and account records. A Redis Cluster caches the live, per-store inventory view that powers nearly every read in the system — this is the single most heavily hit piece of infrastructure and is discussed in detail in the scalability section. Kafka, partitioned by store ID, carries inventory change events and order-routing events, giving each store’s event stream strict ordering while allowing massive overall parallelism across stores. Elasticsearch backs geospatial and product search for the Store Locator Service.

5.1 The Checkout Saga

Placing a BOPIS order touches Reservation, Order and Payment services, each owning its own data. As with any multi-service financial transaction, we use the Saga pattern with explicit compensating actions:

Step 1: Soft-reserve one unit at the selected store
   -> if fails: item no longer available, ask customer to pick another store

Step 2: Authorise and capture payment
   -> if fails: release the soft reservation

Step 3: Create order record and push to store fulfilment queue
   -> if fails: refund payment AND release the soft reservation

5.2 Java Example: Reservation with Optimistic Locking

public class ReservationService {

    public ReservationResult reserveUnit(String storeId, String skuId) {
        StoreInventory inv = inventoryRepository.find(storeId, skuId);
        if (inv.getAvailableQty() <= 0) {
            return ReservationResult.unavailable();
        }
        int updated = inventoryRepository.decrementIfVersionMatches(
            storeId, skuId, inv.getVersion());
        if (updated == 0) {
            // another request won the race, retry read-modify-write once
            return reserveUnit(storeId, skuId);
        }
        Reservation reservation = reservationRepository.create(storeId, skuId,
            Duration.ofMinutes(10));
        return ReservationResult.success(reservation);
    }
}

5.3 Why Reservations Are Short-Lived and Store-Local

A ten-minute reservation window is deliberately short — long enough for a customer to complete checkout, short enough that an abandoned cart does not lock up scarce stock for long. Critically, the reservation applies to one specific store’s inventory record, not a global stock pool, because BOPIS availability is fundamentally local: an item being reserved at Store A has zero effect on Store B’s availability and modelling it any other way would create unnecessary cross-store contention.

5.4 Idempotency at Checkout

Exactly as in any high-traffic checkout flow, every reservation and order-creation request carries a client-generated idempotency key, so a retried request from a flaky mobile connection cannot create a duplicate reservation or double-charge a customer’s card.

5.5 Handling Partial Failures Gracefully

A subtle but important internal-working detail is how the orchestrator distinguishes between a genuine failure (the payment gateway declined the card) and an ambiguous failure (the request to the payment gateway timed out, with no way to know whether the charge actually succeeded on their end before the connection dropped). For genuine failures, the saga compensates immediately and confidently. For ambiguous failures, the orchestrator does not blindly retry (which risks a duplicate charge) or blindly compensate (which risks releasing a reservation for an order that actually succeeded) — instead it marks the order in a distinct PendingVerification sub-state and relies on the idempotency key to safely query the payment gateway for the definitive outcome of that specific request before deciding whether to proceed or compensate. This distinction between confident failure and ambiguous failure is a detail that is easy to overlook in a first-pass design, but it is exactly the kind of edge case that determines whether a saga implementation is merely theoretically correct or actually safe to run against real payment infrastructure in production.

Let us trace the complete customer journey, from searching for a nearby store through pickup completion.

6.1 Order Lifecycle as a State Machine

BOPIS orders pass through distinct states that must be modelled explicitly, because the branch where a store discovers the item is not physically present — despite what the system believed — is a real, frequent scenario that a simple boolean status cannot represent well.

Here is the core data model. Notice that STORE_INVENTORY is keyed by the combination of store and SKU, which is the central modelling decision of the entire system — stock is never a single global number, it is always store-specific.

A few decisions worth highlighting:

• STORE_INVENTORY.reserved_qty is tracked separately from stock_qty so the available-to-promise number (stock_qty - reserved_qty) can be computed without ever mutating the raw physical count reported by the POS system — this keeps the POS-sourced truth and the platform’s derived availability cleanly separated.
• last_synced lets the system detect and flag stores whose sync feed has gone stale, so it can proactively hide or de-prioritise that store’s inventory rather than serve confidently wrong data.
• PICKUP_CODE is a separate entity from ORDER so it can carry its own single-use used flag independent of the broader order status, keeping the pickup-verification check fast and simple.

This is the subsystem that makes the entire product trustworthy. Get it wrong and customers drive to a store only to find their item was never actually there.

8.1 The Fundamental Challenge: External Systems of Record

Unlike a warehouse the platform fully controls, each store’s POS system is an independent system of record the platform does not own. Some stores run modern, API-capable POS software; many large retail chains still run older POS systems that only support periodic batch exports or proprietary message formats. The Store Inventory Sync Service and POS Adapter exist specifically to absorb this heterogeneity.

8.2 Two Sync Patterns, Used Together

In practice, both patterns run simultaneously: the event-driven push gives the low-latency experience customers expect, while a periodic reconciliation job (running every few minutes per store) compares the platform’s believed stock count against a fresh POS snapshot and corrects any drift — protecting against message loss, a missed event, or a store associate’s manual stock adjustment that never generated an event.

8.3 Java Example: Reconciliation Job

public class InventoryReconciliationJob {

    public void reconcileStore(String storeId) {
        Map<String, Integer> posSnapshot    = posAdapter.fetchCurrentStock(storeId);
        Map<String, Integer> cachedSnapshot = inventoryCache.getAllForStore(storeId);

        for (String skuId : posSnapshot.keySet()) {
            int posQty    = posSnapshot.get(skuId);
            int cachedQty = cachedSnapshot.getOrDefault(skuId, 0);
            if (posQty != cachedQty) {
                inventoryCache.correctStock(storeId, skuId, posQty);
                metrics.recordDriftCorrection(storeId, skuId, cachedQty, posQty);
            }
        }
    }
}

The recordDriftCorrection call matters more than it looks — tracking how often and how large these corrections are, per store, is the primary signal for detecting a store whose event-driven integration has silently degraded.

8.4 The Write Path: From Register Sale to Online Visibility

When a walk-in customer buys the last unit of an item at the register, the POS system emits a sale event. The POS Adapter normalises this into an internal StockDecremented event, published to the store’s Kafka partition. The Store Inventory Sync Service consumes this, updates the Redis-cached availability figure for that store and SKU and republishes a StoreInventoryChanged event that the Inventory Aggregator Service uses to update anything currently showing that store’s availability. This entire chain is designed to complete in well under a second under normal conditions, which is what allows the platform to show near-real-time availability without querying every store’s POS system on every single online page view.

8.5 Handling the Race Between a Walk-In Sale and an Online Reservation

The scenario the whole system is ultimately built around: a walk-in customer and an online customer both go for the last unit within moments of each other. The platform cannot lock the physical shelf, so it accepts that the walk-in sale, recorded directly at the register, is authoritative and immediate, while the online soft reservation is provisional until store staff physically confirm the pick. If store staff cannot find the item (the walk-in customer got there first, and the sale event has not yet propagated), the order moves to the OutOfStockAtStore state shown in the earlier state diagram, triggering an automatic reroute to a nearby store rather than leaving the online customer with a confirmed order for something that no longer exists.

9.1 Key Trade-off: Freshness vs Load on Store Systems

Querying a store’s POS system directly on every online product-page view would give perfectly fresh data but would overwhelm store systems never designed for that request volume. Caching aggressively in Redis protects store systems but introduces some sync lag. The chosen design — event-driven push plus periodic reconciliation, discussed in Section 8 — is itself the resolution of this trade-off, deliberately accepting a small, bounded amount of staleness in exchange for protecting store infrastructure and enabling the scale this tutorial targets.

9.2 Key Trade-off: Showing a Stock Buffer vs Showing Raw Counts

As discussed in the production example above, showing “available minus a small buffer” near zero stock reduces disappointed online customers at the cost of very slightly under-representing availability. This is a genuine business trade-off, not just an engineering one, and the buffer size is typically a tunable parameter the merchandising team adjusts per category based on observed race-condition frequency.

9.3 Key Trade-off: Eventual Consistency for Reads vs Strong Consistency for Writes

This entire architecture deliberately applies different consistency models to different parts of the system, and being able to articulate why is a strong interview signal. The inventory read path (Redis-cached availability shown in search results) is eventually consistent by design — it can lag physical reality by a small, bounded window and the system is built to tolerate and gracefully handle that lag through the OutOfStockAtStore reroute flow. The reservation and payment write path, by contrast, must be strongly consistent within its own transaction boundary — two customers cannot both successfully reserve the same physical unit and a payment cannot be captured twice for one order. Mixing these models correctly, rather than applying one blanket consistency guarantee everywhere, is precisely what allows this system to be both fast enough for a million requests a minute and correct enough to be trusted with real money and real physical goods.

This is the section that most directly separates a “correct” design from one that is actually production-ready at the scale this tutorial requires. Let us ground this in real numbers first, then walk through exactly which components would break under naive assumptions and how the design in Section 4 addresses each one.

10.1 What a Million Requests a Minute Actually Means

One million requests per minute is approximately 16,667 requests per second as a flat average. Real traffic is never flat — a flash sale, a popular product launch, or a holiday shopping peak can easily produce a burst three to five times higher than the sustained average for short windows. A system designed only for the average, not the burst, will fail exactly when it matters most. This tutorial’s target working assumption is therefore closer to 50,000 to 80,000 requests per second at peak burst, with roughly 16,000 to 17,000 requests per second sustained.

10.2 Read-Heavy Traffic Shape

BOPIS traffic is overwhelmingly read-heavy: the vast majority of requests are “is this in stock near me” searches and product-page availability checks, not checkout actions. A realistic split at this scale is roughly 95% reads (search, availability checks, order-status polling) and 5% writes (reservations, order creation, pickup confirmation). This ratio should directly drive architecture decisions — the system should be optimised first and foremost for absorbing massive read volume cheaply.

10.3 Layer-by-Layer: How Each Component Survives This Load

CDN — absorbs the top of the funnel

Product images and largely-static store metadata served through the CDN never reach the origin at all for the vast majority of requests. At this scale, a well-configured CDN should be absorbing well over half of total client-facing traffic before it ever touches the Load Balancer.

Load Balancer — horizontal fan-out

The Load Balancer itself must be a managed, horizontally scalable service (not a single instance) capable of handling tens of thousands of connections per second, distributing across an auto-scaling pool of API Gateway instances. Health checks continuously remove any instance that becomes slow or unresponsive under load, rather than letting it degrade the whole pool.

API Gateway — rate limiting as a survival mechanism

At this request volume, rate limiting stops being a nice-to-have and becomes essential self-preservation. Per-customer and per-IP rate limits at the gateway prevent a single misbehaving client (a bot repeatedly polling a hyped product’s stock) from consuming a disproportionate share of backend capacity and a global circuit breaker at the gateway can shed lower-priority read traffic (like background polling) before ever letting write-path traffic like checkout get starved.

Inventory Aggregator Service — the read hot path

Because reads dominate at a 95:5 ratio, the Inventory Aggregator Service almost never queries a database directly for a stock check — it reads from the Redis Cluster, which is horizontally sharded (partitioned by store ID, consistent with the Kafka partitioning strategy) so no single Redis node becomes a bottleneck. A single well-provisioned Redis node can serve on the order of 100,000+ simple key reads per second, so a cluster of a modest number of shards comfortably absorbs even burst-level read volume for this workload, especially combined with a short-TTL local in-memory cache inside each Inventory Aggregator instance for the very hottest SKUs.

Reservation and Order Services — the write hot path

Writes are the harder 5%, because they require correctness guarantees reads do not. Postgres is sharded by store region so that reservation and order writes for different regions land on entirely different database instances, multiplying total write throughput. Within a shard, the optimistic-locking pattern shown in Section 5.2 keeps individual lock durations extremely short, which is essential — long-held locks are what turn a high-throughput write path into a queueing bottleneck under contention.

Kafka — absorbing bursts without losing events

Partitioning by store ID (Section 4.7) means the total event throughput scales with the number of partitions, and since stores are independent of each other, this scales near-linearly with the number of stores in the system. Kafka’s durable, disk-backed log also means a burst of inventory-change events during a peak sale is absorbed into the queue and processed as fast as downstream consumers can keep up, rather than being dropped or blocking the POS Adapter.

10.4 Auto-Scaling Policy

Stateless services (Auth, Store Locator, Inventory Aggregator, Order) scale horizontally based on a combination of CPU utilisation and, more importantly for a bursty read-heavy system, request queue depth — scaling purely on CPU tends to react too slowly for the kind of sharp traffic spikes a flash sale produces. Target scaling policies should aim to add capacity well before utilisation crosses seventy percent, since new instances take real time (tens of seconds to a few minutes) to boot, warm caches and start serving traffic.

10.5 Graceful Degradation Under Extreme Load

Even a well-scaled system should have a defined degraded mode for the rare moment demand outpaces even burst capacity. A sensible priority order: protect checkout and payment (the 5% write path that generates revenue and cannot be silently dropped) above all else; next protect pickup verification (customers physically standing in a store should never be blocked); and if something has to degrade, let it be search freshness — serving a slightly staler cached availability view, or briefly queueing non-critical search requests, is a far better failure mode than an outage across the board.

11.1 Multi-AZ and Multi-Region Deployment

All stateless services and the Redis and Kafka clusters are deployed across multiple availability zones, so a single zone failure does not take down the platform. Given the national or global footprint of most BOPIS deployments, Postgres shards are typically aligned with geographic regions, so a regional outage affects only stores and orders in that region rather than the entire platform.

11.2 Circuit Breakers for Store POS Integrations

The POS Adapter talks to potentially thousands of independent, unreliable store systems. Each store connection is wrapped in its own circuit breaker, so one store’s misbehaving or offline POS integration cannot exhaust shared connection pools or thread capacity that the sync pipeline needs for every other store.

@CircuitBreaker(name = "posStore-#{storeId}", fallbackMethod = "staleDataFallback")
public StoreStockSnapshot fetchStock(String storeId) {
    return posClient.getCurrentStock(storeId);
}

public StoreStockSnapshot staleDataFallback(String storeId, Throwable t) {
    metrics.recordPosFailure(storeId);
    return inventoryCache.getLastKnownSnapshot(storeId, /* markStale= */ true);
}

11.3 Reservation Expiry as a Distributed Scheduling Problem

With potentially tens of thousands of active ten-minute reservations at any moment across a national footprint, expiring them cannot be a single polling job scanning a full table. A time-bucketed expiry scheduler, or a Redis key with a TTL that triggers a keyspace-expiration event consumed by the Reservation Service, scales this independently of total reservation volume.

11.4 Order Reconciliation Job

As with any saga-based flow, a scheduled reconciliation job scans for orders stuck in an intermediate state (payment captured but never pushed to a store queue, for example) longer than expected and resolves them — either completing or compensating the transaction. This job is a required safety net, not an optional extra, at this system’s scale and complexity.

12.1 Authentication Across Three Client Types

Customers authenticate with standard OAuth2 / JWT flows through the mobile and web apps. Store associates authenticate through a separate, more tightly scoped credential tied to their specific store and role, so a compromised associate credential at one store cannot be used to access order or customer data for other stores. Internal service-to-service traffic uses mutual TLS.

12.2 Pickup Code Security

Pickup codes must resist both guessing and replay. They are generated as sufficiently long random tokens (not sequential order numbers), are single-use (enforced by the used flag on the PICKUP_CODE entity) and expire along with the order’s pickup window. The Pickup Verification Service also rate-limits code-check attempts per order to prevent brute-force guessing at the counter.

12.3 Protecting Store Systems from the Public Internet

Store POS systems, especially older ones, were frequently never designed with internet-facing security in mind. The POS Adapter and Store Inventory Sync Service sit in a tightly controlled network segment, communicating with store systems over a private, authenticated channel (VPN or dedicated link, depending on the store’s connectivity), rather than exposing any store system directly to the public-facing API Gateway.

12.4 Rate Limiting as a Security Control, Not Just a Performance One

At this traffic scale, the API Gateway’s rate limiting doubles as a defence against scraping (competitors monitoring stock levels at scale) and against credential-stuffing attacks on the Auth Service, in addition to its role in protecting backend capacity discussed in Section 10.

12.5 Data Privacy: Location and Purchase History

BOPIS inherently ties a customer’s purchase history to specific physical locations they visit, which is more sensitive than a typical shipping address on file — it can reveal a customer’s regular movements over time. Under privacy regulations such as GDPR and similar regional laws, customers have the right to access and delete this data. In practice, this means the Order Service needs a data-export endpoint assembling a customer’s full BOPIS history and a deletion workflow that anonymises historical order and pickup records (stripping customer-identifying fields while retaining the aggregate transaction facts required for financial audit retention) rather than performing a hard delete that would break required record-keeping.

13.1 The Three Pillars, Applied to BOPIS

Metrics surface that something is wrong, logs explain what happened and distributed tracing (using a correlation ID propagated from the moment a customer starts a store search through to pickup completion) shows exactly where in the chain of six-plus services a problem occurred.

13.2 Business Metrics That Matter

13.3 Alerting Philosophy at Scale

At this traffic volume, alert thresholds must be tuned to avoid noise from normal, expected traffic spikes (a scheduled flash sale is not an incident) while still catching genuine anomalies quickly. A practical approach pages on-call engineers for symptom-based alerts — checkout success rate dropping, p99 latency crossing a hard threshold, a specific store’s sync lag exceeding several minutes — rather than raw infrastructure metrics like CPU percentage, which live on dashboards instead.

13.4 Store-Level Observability Dashboards

Because this system’s correctness fundamentally depends on thousands of independent store integrations, store operations teams need a dedicated dashboard surfacing per-store sync health, drift-correction frequency (from the reconciliation job in Section 8.3) and OutOfStockAtStore incidents — this is what lets a retailer proactively catch a specific store’s failing POS integration before it generates a wave of frustrated customers.

Containerised microservices orchestrated by Kubernetes, deployed across multiple regions to match the geographic distribution of stores and minimise latency for both customers and store-side POS integrations. Each service is independently deployable, which matters especially for the Store Integration Domain — POS adapter code changes for onboarding a newly acquired retail chain’s stores should never require redeploying the customer-facing checkout path.

14.1 Canary Releases for the Reservation and Order Services

Given the financial nature and sheer request volume these services handle, changes are rolled out as canary releases — a new version receives a small percentage of live traffic (starting around one to five percent), with automated rollback triggered if error rates or latency regress, before gradually increasing to full traffic. This is safer than an all-at-once rolling update for a write path this critical and this heavily trafficked.

14.2 Infrastructure as Code and Multi-Region Consistency

All infrastructure — Kubernetes clusters, Redis and Kafka cluster topology, auto-scaling policies, load balancer configuration — is defined declaratively (Terraform or similar) so that every region’s infrastructure stays consistent and standing up capacity in a new region (for geographic expansion) is a repeatable, low-risk process rather than manual reconstruction.

15.1 Why Postgres, Sharded, for Orders

Orders and reservations need ACID guarantees — a double-charged customer or a double-sold reservation is unacceptable. Postgres provides this and sharding by store region distributes write load across many independent database instances, which is essential at this system’s target scale, since a single unsharded primary would become a hard ceiling on write throughput long before reaching a million requests a minute.

15.2 Redis Cluster Design for the Inventory Hot Path

The Redis Cluster is partitioned (sharded) by store ID, matching the Kafka partitioning strategy from Section 4.7 — this consistency means the same store’s data flows through corresponding shards end to end, simplifying reasoning about ordering and making it straightforward to identify and scale a specific hot shard (a store carrying a viral product, for example) independently of the rest of the cluster. Each shard additionally has read replicas to further multiply read capacity, since this system’s read-to-write ratio is roughly 95:5 as established in Section 10.2.

15.3 Read Replicas and CQRS for Search

Store Locator and Inventory Aggregator read paths are effectively a separate concern from the transactional write path, which is a natural fit for a CQRS (Command Query Responsibility Segregation)-flavoured approach: writes go through the Reservation and Order services into Postgres and Redis, while the read-optimised view (denormalised, cached and geospatially indexed in Elasticsearch) is built asynchronously from the same event stream, letting the two sides scale and evolve independently.

15.4 Load Balancing Algorithm Choice

15.5 Capacity Estimation for the Redis Layer

Assume a national retailer with 10,000 stores and 50,000 actively tracked SKUs per store — that is 500 million store-SKU inventory records. At roughly 100 bytes per cached record (stock quantity, reserved quantity, last-synced timestamp and key overhead), that is about 50 GB of total cached data, comfortably fitting across a modest Redis Cluster of a handful of well-provisioned shards with room for replication. Given the 95:5 read-heavy ratio and a target of 50,000-plus requests per second at burst and each Redis shard capable of well over 100,000 simple reads per second, this is a workload the cluster can absorb with headroom, provided (as emphasised throughout this section) that partitioning by store ID prevents any single shard from becoming a disproportionate hot spot.

15.6 Full Capacity Walkthrough

Let us connect the earlier traffic numbers to concrete infrastructure sizing, the way an interviewer would want to see reasoned through step by step. At 16,667 requests per second sustained and roughly 95% reads, that is about 15,800 read requests per second and 830 write requests per second on average, with bursts pushing total traffic toward 50,000 to 80,000 requests per second as established in Section 10.1.

For the read path: if the CDN absorbs roughly half of all client-facing traffic (static assets and cacheable product data), the remaining dynamic read traffic reaching the Inventory Aggregator Service is on the order of 7,000 to 8,000 requests per second sustained, spiking toward 25,000 to 30,000 at burst. A single Inventory Aggregator instance, doing a simple Redis lookup with minimal processing, can reasonably handle a few thousand requests per second; this points to needing on the order of ten to fifteen instances sustained, auto-scaling toward thirty or more at peak burst — a very achievable number for a horizontally-scaled stateless service behind a load balancer.

For the write path: 830 requests per second sustained across reservation and order creation, spread across a sharded Postgres cluster by region. If we assume ten regional shards, that is roughly 80 to 100 writes per second per shard sustained — well within what a single well-provisioned Postgres primary handles comfortably, even accounting for burst multiples of three to five times that figure during a flash sale.

This kind of explicit, step-by-step capacity walkthrough — traffic estimate, split by read / write, divided by expected per-instance or per-shard throughput, resulting in a concrete instance or shard count — is exactly what strong system design interview answers demonstrate, and it is worth practising this reasoning chain independent of the specific numbers used here.

15.7 Concurrency, Locking & Algorithms

15.7.1 Optimistic Locking Under Extreme Contention

The optimistic locking pattern shown in Section 5.2 works well under normal contention, but a viral product at a single popular store can create a genuine hot-key problem — thousands of reservation attempts racing for the same few remaining units within seconds. In this specific scenario, pure optimistic locking with retry can degrade into a thundering herd of failed retries. The mitigation is to route these hot-key decrements through a Redis atomic DECR operation rather than a database row lock, since Redis single-threaded command execution naturally serialises concurrent decrements without the retry storm a database-level optimistic lock produces under this level of contention, then asynchronously persist the confirmed reservation to Postgres.

public class HotKeyReservationHandler {

    public boolean tryReserve(String storeId, String skuId) {
        String key = "inv:" + storeId + ":" + skuId;
        Long remaining = redisTemplate.opsForValue().decrement(key);
        if (remaining != null && remaining >= 0) {
            asyncPersistenceQueue.enqueue(new ReservationRecord(storeId, skuId));
            return true;
        }
        // oversold the cache counter, roll back
        redisTemplate.opsForValue().increment(key);
        return false;
    }
}

15.7.2 Geospatial Search Algorithm for Store Locator

Finding “nearby stores with this item in stock” combines a geospatial query with an availability filter. The Store Locator Service uses Elasticsearch’s native geo-distance query to efficiently find candidate stores within a radius, sorted by distance, then joins this against the Inventory Aggregator’s cached availability for the requested SKU — running the availability filter second, against a much smaller candidate set from the geo query, is significantly cheaper than trying to filter all stores by both criteria simultaneously in a single pass.

15.7.3 Consistent Hashing Across the Sync Pipeline

As emphasised throughout Sections 4, 8 and 15, using the same consistent-hashing scheme (store ID) across Kafka partitioning, Redis Cluster sharding and even load balancer routing for the Store Integration Domain is a deliberate, load-bearing design choice — it means a single store’s complete data journey, from POS event to cached availability, stays within a predictable, traceable set of infrastructure, which dramatically simplifies both debugging and capacity planning for any individual store or region.

15.8 Disaster Recovery & Cost Optimisation

15.8.1 Backup and Recovery Objectives

The Order and Reservation services, handling real financial transactions, target a Recovery Point Objective under one minute and a Recovery Time Objective under five minutes, using continuous write-ahead log shipping to standby replicas in a separate availability zone. The Inventory cache, being derivable from the POS systems and the reconciliation job, can tolerate a much looser recovery objective — in the worst case, a full reconciliation pass across all stores can rebuild it from scratch, which is a meaningfully different (and cheaper to protect) reliability posture than the transactional data.

15.8.2 Cost Optimisation Levers

• CDN offload is the single biggest lever for controlling compute cost at this traffic scale — every request served from the edge is a request that never needs an auto-scaled backend instance to handle it.
• Time-of-day auto-scaling — BOPIS search and checkout traffic follows predictable daily and weekly patterns (evenings, weekends and around major sale events), so scaling down aggressively during predictable low-traffic windows meaningfully reduces sustained compute cost versus static peak-sized provisioning.
• Tiered store integration investment — building the more expensive, low-latency event-driven POS integration first for high-volume flagship stores, while lower-traffic stores rely more heavily on the cheaper periodic-pull pattern, focuses integration engineering cost where it has the most customer impact.
• Reserved capacity for the sustained baseline, burst capacity for peaks — provisioning reserved / committed cloud capacity for the roughly 16,000 requests / second sustained baseline established in Section 10.1, while relying on on-demand auto-scaling only for the burst multiple above that baseline, is typically far cheaper than provisioning peak capacity as a reserved commitment.

16.1 Key External Endpoints

GET  /v1/stores/nearby?lat={lat}&lng={lng}&sku={sku}   -> ranked stores with availability
POST /v1/reservations                                    -> soft-reserve item at a store
POST /v1/orders                                          -> create order, capture payment
GET  /v1/orders/{id}                                     -> fetch order status
POST /v1/pickup/verify                                   -> validate pickup code at counter

16.2 Synchronous vs Asynchronous Communication

Customer-facing search and checkout calls are synchronous REST through the API Gateway, since a human is waiting for an immediate response. Store-side events — stock changes, order acknowledgment, ready-for-pickup status — flow asynchronously through Kafka, which is what allows the system to absorb bursts without forcing every store integration to keep pace with real-time customer traffic.

16.3 API Versioning and Backward Compatibility

External APIs are versioned in the URL path. This matters especially for the POS Adapter’s internal contract with store systems, some of which may be integrated for years without updates — breaking that contract without a clear migration path could silently stop inventory sync for an entire store or chain.

• Partition consistently across Redis and Kafka by the same key (store ID) so the whole pipeline’s scaling and ordering guarantees stay coherent end to end.
• Always run both event-driven sync and periodic reconciliation — event streams can silently drop messages and reconciliation is the safety net that catches drift.
• Design rate limiting at the gateway from day one, not as a reactive fix after the first traffic spike causes an outage.
• Treat OutOfStockAtStore as an expected, first-class state, not an edge case — at scale, it will happen thousands of times a day and the automatic-reroute flow needs to be a smooth, tested path.
• Common mistake: under-provisioning for burst rather than average traffic — a system sized only for the 16,000 requests / second average will fail exactly during the flash sale or product launch that matters most.
• Common mistake: forgetting that store associates are real, busy people juggling in-aisle customers and BOPIS fulfilment — a poorly designed associate app workflow becomes the true bottleneck no amount of backend scaling can fix.
• Common mistake: treating the API Gateway’s rate limiter as a fixed, one-time configuration rather than a continuously tuned control — as real traffic patterns emerge post-launch, thresholds set during initial load testing frequently need adjustment and a gateway that cannot have its limits updated without a full redeploy becomes an operational liability during exactly the high-traffic events it exists to protect against.
• Best practice: instrument the OutOfStockAtStore reroute path with the same rigor as the happy path from day one — since it is a frequent, expected flow at this system’s scale rather than a rare edge case, it deserves first-class dashboards, alerting and on-call runbooks, not an afterthought bolted on after the first production incident reveals nobody understands how often it fires or why.
• Best practice: version the internal contract between the Store Inventory Sync Service and the POS Adapter layer explicitly, even though both are internal services the platform team fully controls — POS integrations are frequently maintained by a different team, on a different release cadence, than the core commerce platform and an implicit, unversioned contract between them is a common source of silent production breakage when one side changes without the other being aware.

19.1 A Closer Look: Target’s Geofenced Drive Up

Target’s Drive Up feature is a good illustration of how the core BOPIS architecture in this tutorial extends naturally: the mobile app triggers a geofence event as the customer approaches, which is simply another event published onto the same order-event stream that already carries reservation, payment and fulfilment events — the In-Store Fulfilment Service reacts to it exactly the way it reacts to a “ready for pickup” transition, just triggering a different downstream action (notify staff to bring the order to the car instead of waiting at a counter).

19.2 A Closer Look: Why Electronics Retailers Lean Harder on BOPIS

Best Buy’s heavy investment in BOPIS reflects a direct connection between average order value and the value of eliminating shipping risk and delay — for a low-cost item, a customer may not mind waiting two days for delivery, but for an expensive laptop or television, both the retailer and the customer have strong incentives to avoid the fraud exposure, damage risk and delay of shipping when a same-day, in-person handoff is available instead. This is a useful lens for interview discussions about which product categories benefit most from BOPIS investment.

19.3 A Closer Look: Grocery BOPIS and Substitution Handling

Grocery BOPIS adds a wrinkle the base architecture in this tutorial does not fully cover: perishable and highly variable stock means the exact item reserved online is sometimes genuinely unavailable by pick time even with excellent sync (a store simply sold its last three ripe avocados to walk-in customers between the reservation and the pick). Grocery-focused BOPIS platforms extend the OutOfStockAtStore state with a substitution workflow — the picker proposes a similar replacement item and the customer approves or declines it, typically through the same notification channel used for ready-for-pickup alerts. This is a direct, practical illustration of how the core state machine from Section 6.1 is designed to be extended for category-specific needs without redesigning the whole order lifecycle.

19.4 Testing Strategy

A system coordinating real-time state across a saga-based checkout flow, a store-partitioned cache and thousands of independent external POS integrations needs a testing approach that goes well beyond isolated unit tests.

19.4.1 Contract Tests for the POS Adapter

Since the POS Adapter integrates with multiple different vendor POS platforms, each integration is covered by contract tests that verify the adapter correctly translates that vendor’s specific event and API formats into the platform’s normalised internal events — this catches a vendor’s API change or an integration regression at build time rather than as a silent, hard-to-diagnose sync failure discovered only through the reconciliation job days later.

19.4.2 Saga Integration Tests, Including the Store-Side Failure Path

Beyond the standard saga compensation-path tests (payment failing after reservation succeeds, and so on), this system specifically needs integration tests exercising the OutOfStockAtStore branch — simulating a store confirming it cannot find a reserved item and verifying the automatic reroute-to-nearby-store flow works correctly end to end, including a second reservation attempt and a second push to a different store’s fulfilment queue. This branch is easy to under-test because it does not occur on the simple happy path, yet at this system’s scale it is a frequent, expected occurrence.

19.4.3 Load Testing Against the Million-Requests-a-Minute Target

Given the explicit scale target for this system, load testing is not optional validation — it is core to the design process. Regular load tests should simulate the realistic 95:5 read / write traffic shape described in Section 10.2, including burst multiples of three to five times sustained average, specifically targeting the Redis Cluster’s hot-key behaviour (Section 15.7.1) and the auto-scaling policy’s reaction time (Section 10.4), since these are the two places most likely to reveal a gap between design intent and real behaviour under load.

Why not just make the online store query each store’s POS system live for every availability check?

At the traffic volumes this system targets, that would overwhelm store POS systems that were never designed for tens of thousands of requests per second and it would make the platform’s availability directly dependent on the reliability of every single store’s local system — a single struggling store’s POS could slow down search for the entire platform. Caching in Redis, kept fresh through event-driven sync and periodic reconciliation, decouples these concerns.

How would you handle a brand-new store being onboarded with no historical sync data?

A newly onboarded store starts in an explicit “onboarding” state, excluded from customer-facing search results until an initial full inventory snapshot is successfully pulled from its POS system and the event-driven sync integration is verified to be flowing correctly — surfacing a store before its data is trustworthy would immediately damage customer trust in the platform.

What happens if a customer never picks up their order?

The order sits in ReadyForPickup until the configured pickup window (Section 3.4) elapses, at which point it automatically transitions to Expired and then Refunded and the reserved unit is released back into sellable store stock, as shown in the state diagram in Section 6.1.

How would this design change if a single store, rather than the whole platform, needed to survive a local traffic spike — say, a celebrity visit driving huge foot traffic and app usage at one location?

Because Kafka, Redis and the load balancer’s consistent hashing are all partitioned by store ID (Sections 4.7, 15.2 and 15.7.3), a single store’s traffic spike is naturally isolated to that store’s shard rather than contending with the rest of the platform’s capacity — the main risk is that one shard becoming disproportionately hot relative to its neighbours, which the per-shard monitoring described in Section 13.4 is specifically designed to surface early, allowing that shard to be scaled or rebalanced independently.

How do you decide the size of the pickup window and does it affect the scalability numbers discussed earlier?

The pickup window is primarily a customer-experience and store-operations decision — long enough that customers are not rushed, short enough that unclaimed reservations do not tie up shelf stock indefinitely — typically landing between three and seven days in practice. It does have a real, if secondary, scalability implication: a longer window means more concurrent ReadyForPickup orders sitting in the system at any moment, which modestly increases the working set size the Order Service and its supporting cache must hold, though this is a far smaller contributor to total system load than the read-heavy search and availability traffic discussed throughout Section 10.

If you had to cut scope for a first version of this system, what would you defer?

A reasonable v1 could launch with a single, modern, API-capable POS integration pattern (deferring support for legacy batch-only POS systems), a fixed pickup window with no dynamic adjustment and manual store-onboarding rather than a fully automated onboarding pipeline. The core reservation saga, the store-partitioned inventory cache and the OutOfStockAtStore rerouting flow are not safely deferrable — they define the fundamental trust the customer places in the “it’s in stock, come get it” promise.