Picture a mid-size clothing retailer with 400 physical stores and a busy website. A customer walks into the downtown store and buys the last size-medium blue jacket on the shelf. Ninety seconds later, a different customer in another city, browsing the same brand’s website, adds that exact jacket — same size, same color, same warehouse-linked SKU — to their online cart. If the website still shows “In Stock,” the company has just sold something it does not have. This is not a rare, unlucky coincidence. At scale, with thousands of transactions happening every minute across stores and the web, this collision happens constantly unless a system is deliberately built to prevent it.

This tutorial is about that system: real-time inventory synchronization between an e-commerce platform and physical retail store point-of-sale (POS) systems. It is one of the most practical, widely-asked system design problems in the industry, because almost every retailer that sells both online and offline has had to solve it, and almost every large tech company (Amazon, Walmart, Target, Best Buy, IKEA, Zara) has invested heavily in getting it right.

A short history of the problem

Before the 2000s, physical retail and online retail were often run as two entirely separate businesses with two separate inventories, two separate warehouses, and sometimes two separate IT departments. A store’s stock count lived in a register or a local server; the website’s stock count lived in a completely different database, often updated only once a day through a nightly batch job. This was acceptable when e-commerce was a small side channel. Customers rarely noticed a mismatch because online volume was low.

As online shopping grew through the 2010s, and especially after the rise of “buy online, pick up in store” (BOPIS), “ship from store,” and same-day delivery, this separation became a serious liability. Retailers realized that treating online and offline inventory as one unified pool — visible and updated everywhere, instantly — was a competitive necessity, not a nice-to-have. Amazon’s acquisition of Whole Foods, Walmart’s massive investment in omnichannel fulfillment, and Target’s “Shipt” same-day delivery service are all, underneath the marketing, inventory synchronization problems solved at enormous scale.

Today, the expectation has shifted from “nightly batch reconciliation” to real-time or near-real-time synchronization, typically with a latency target measured in seconds, not hours. This tutorial explains, from first principles, how such a system is designed, built, scaled, and operated in production.

A short timeline of retail inventory synchronization

Before designing any system, we need to be precise about what problem we are actually solving. “Sync inventory between store and website” sounds simple, but it hides several distinct, hard sub-problems.

The core problem statement

• A sale at any store or on the website reduces the available count everywhere else, within a target latency (commonly 1–5 seconds for high-demand items, up to a minute for long-tail items).
• The system never lets more units be sold than physically exist (no “oversell”), or at minimum minimizes oversell to a controlled, cancel-and-refund rate.
• The system survives network partitions — a store losing internet connectivity for ten minutes should not lose sales data or corrupt the central count.
• The system scales to tens of thousands of transactions per second during peak events like Black Friday, without falling over.

Why this is hard (not just “call an API”)

A junior engineer’s first instinct is often: “Just have the POS call the e-commerce API directly every time something sells.” This breaks down almost immediately at scale, for reasons worth naming explicitly, because interviewers will push on exactly these points.

Scoping the problem for an interview or a real project

Before jumping to a solution, it helps to explicitly agree on scope, because “inventory synchronization” can silently expand to include pricing synchronization, promotions, tax calculation, and fulfillment routing if boundaries aren’t set. For this tutorial, and for a typical system-design interview framing, the scope is deliberately narrowed to: tracking accurate, near-real-time available-quantity counts for each SKU at each location (including a virtual “online” location representing e-commerce-fulfillable stock), propagating changes quickly and reliably in both directions between stores and the web platform, and doing so in a way that degrades gracefully rather than catastrophically under network or infrastructure failure. Pricing, promotions, and full order-fulfillment orchestration are treated as adjacent systems this one integrates with, not as part of the core design.

Functional and non-functional requirements

Explicitly separating functional requirements (what the system must do) from non-functional requirements (the qualities the system must have while doing it) is exactly the kind of structured thinking that distinguishes a strong system-design answer from a vague one, and it’s worth stating both categories out loud early, before diving into architecture, so that everyone in the discussion is designing against the same shared, explicit target rather than silently assuming different priorities.

Business impact of getting this wrong

It’s worth grounding the technical problem in business consequences, because system design interviews increasingly reward candidates who connect architecture decisions to business outcomes.

Before diagramming the architecture, we need shared vocabulary. Each concept below is explained with a real-life analogy, a beginner-level example, and a production example, because these terms will be reused throughout the rest of the tutorial.

Event-Driven Architecture

What: Instead of one system directly calling another and waiting for a response, a system announces “something happened” (an event) onto a shared channel, and any interested system picks it up whenever it is ready.

Analogy: Think of a school bell. The teacher does not walk to every classroom and personally tell each student “class is starting.” The bell rings once; every classroom that cares about that signal reacts independently, at its own pace.

Beginner example: A doorbell camera doesn’t call your phone directly and wait on the line — it publishes a “motion detected” notification, and your phone app, plus any other subscribed device, reacts to it independently.

Production example: When a POS sells an item, it publishes an InventoryDecremented event to a message broker rather than calling the e-commerce API directly. The e-commerce inventory service, the analytics pipeline, and the reorder-alert service all consume that same event independently.

Message Broker / Event Stream

What: A durable, ordered (per-partition) log or queue that sits between producers (things that generate events) and consumers (things that process events), decoupling them in time and space.

Analogy: A post office mailbox. You drop a letter in; you don’t need the recipient to be home. The letter waits safely until they check the mailbox.

Production example: Apache Kafka is the most common choice for this use case — it retains events for a configurable period, supports very high throughput, and guarantees ordering within a partition (e.g., all events for the same SKU land in the same partition and are processed in order).

Eventual Consistency

What: A guarantee that, given no new updates, all copies of a piece of data will eventually converge to the same value — but not necessarily instantly. This is contrasted with strong consistency, where every read always reflects the very latest write.

Analogy: Group chat message delivery. When you send a message, your friends across different networks and devices don’t all see it in the exact same millisecond, but within a short time everyone converges to the same conversation history.

Production example: When a store sells an item, the central inventory count and the website’s cached “In Stock” badge might disagree for 1–3 seconds while the event propagates — that’s an accepted, deliberate trade-off (explored fully in Trade-offs).

Idempotency

What: An operation is idempotent if performing it multiple times has the exact same effect as performing it once. This matters because message delivery systems commonly offer “at-least-once” delivery, meaning duplicates can and will happen.

Analogy: An elevator call button. Pressing it five times doesn’t call five elevators — the system recognizes the request is already registered.

Production example: Each inventory event carries a unique eventId. The consumer checks whether it has already processed that ID (using a deduplication table or cache) before applying the change, so a retried delivery does not double-decrement stock.

Optimistic Concurrency Control

What: A strategy where updates carry a version number; a write only succeeds if the version it expects still matches the current version in the database. If two writers collide, one wins and the other retries against the new version.

Analogy: Editing a shared Google Doc. If two people edit the same paragraph, the system detects the conflict and reconciles it, rather than silently letting one edit erase the other without anyone noticing.

Production example: An inventory row stores quantity = 4, version = 17. A decrement operation says “update quantity to 3 only if version is still 17.” If another concurrent sale already moved the version to 18, this write is rejected and retried — preventing the classic lost-update bug.

CAP Theorem in this context

What: During a network partition, a distributed system must choose between Consistency (every node sees the same data) and Availability (every request gets a response, even if potentially stale). You cannot have perfect versions of both at once during a partition.

Applied here: If a store loses connectivity to the cloud, should the POS refuse to sell anything until it reconnects (favoring consistency), or should it keep selling using its last-known local stock count (favoring availability)? Nearly every real retailer chooses availability — a register that cannot sell during a network blip is a direct, visible revenue loss and a terrible in-store experience. This choice, and its consequences, shapes much of the architecture that follows.

CQRS (Command Query Responsibility Segregation)

What: A pattern where the path used to change data (commands, such as “record a sale”) is architecturally separated from the path used to read data (queries, such as “show current stock”). Each path is optimized independently, using different models, storage engines, or scaling strategies.

Analogy: A restaurant kitchen has one workflow for taking orders (commands flowing from customer to kitchen) and a completely separate workflow for the menu board (a read-optimized view everyone can glance at without disturbing the kitchen). The kitchen and the menu board don’t need to be the same system, even though they represent the same underlying reality.

Production example: Writes flow through the Inventory Sync Service into the sharded database (the command side); reads flow from Redis and read replicas (the query side), which are kept fresh via the write path’s cache-invalidation step, but are never themselves the target of a write.

Backpressure

What: A mechanism that lets a slower downstream component signal to a faster upstream component “slow down, I can’t keep up,” rather than being overwhelmed and failing.

Analogy: A funnel naturally limits how fast liquid can pour through it — pouring faster than the funnel’s throughput just causes overflow, so a careful pourer watches the funnel and slows down.

Production example: If the Inventory Sync Service’s consumer lag starts climbing because the database is struggling to keep up with write volume, the consumer intentionally slows its poll rate (or the orchestrator adds more consumer instances) rather than accepting events faster than they can be durably applied.

Hybrid Logical Clocks

What: A timestamp scheme that combines a physical wall-clock time with a logical counter, giving events a total order that is both roughly time-accurate and immune to the problems of relying on physical clocks alone (which can drift or disagree slightly across machines).

Analogy: Imagine a group of people writing entries in a shared logbook from different rooms. Each entry gets both a wall-clock timestamp and a running sequence number, so even if two people’s watches are a second off from each other, the sequence number still lets everyone agree on the true order of entries.

Production example: In a multi-region deployment, hybrid logical clocks attached to each inventory event let the Conflict Resolution Engine correctly determine that an event from Region A “happened before” a related event from Region B, even when the two regions’ physical clocks are not perfectly synchronized — which they never are, in practice, due to network time protocol drift.

Read-Your-Writes Consistency

What: A weaker, more targeted consistency guarantee than full strong consistency: a specific user (or session) is guaranteed to see their own most recent write, even if other users might briefly see a slightly older value.

Analogy: If you post a comment on a social media app, you expect to see your own comment appear immediately when you refresh, even if a friend on the other side of the world might see it a second later.

Production example: A store manager who just performs a manual stock correction on a handheld device should immediately see that corrected count reflected back to them on the same device, even though the broader system elsewhere is only eventually consistent — this is typically achieved by routing that manager’s subsequent read to the same region/replica that accepted their write, or by having the client optimistically apply its own write locally while waiting for confirmation.

With vocabulary established, here is the full end-to-end architecture. Every box below is a distinct component you would name explicitly in an interview whiteboard session — load balancer, API gateway, message broker, sync service, cache, and database are all called out individually, because interviewers specifically listen for these component names and the reasoning behind each one.

Component-by-component breakdown

1. POS Terminal (store register)

The physical or software cash register at each store. When a cashier scans an item and completes a sale, the POS records a local transaction and needs to communicate a stock decrement outward. It must remain functional even without internet access, because a store’s ability to sell should never depend on cloud connectivity.

2. Store Sync Agent

A lightweight service running on-premises at each store (or in a nearby edge location) that buffers inventory events locally — typically in an embedded database like SQLite or RocksDB — and forwards them to the cloud with retry logic and exponential backoff. If the store’s internet connection drops, events queue locally and flush automatically once connectivity returns, guaranteeing no sale event is lost.

3. Load Balancer

Sits in front of the cloud-facing gateway, distributing incoming traffic from thousands of Store Sync Agents (and the website itself) across many API Gateway instances. It performs health checks and removes unhealthy instances from rotation, ensuring no single gateway node becomes a bottleneck or single point of failure. In this system it typically operates at Layer 7 (HTTP-aware), enabling routing decisions based on path or headers, not just IP and port.

4. API Gateway

The single, controlled entry point into the backend. It handles authentication (verifying the request really comes from a registered store device), authorization, rate limiting (so one misbehaving store cannot flood the system), request validation, and routing to the correct downstream service. It also provides a stable, versioned public contract so internal services can evolve independently.

5. Message Broker (Kafka event stream)

The backbone of the whole system. Every inventory-affecting action — a sale, a return, a restock, a manual count correction — is published as an immutable event here. Kafka’s partitioning (commonly partitioned by SKU or by store+SKU) guarantees that events for the same product are processed in the order they arrived, which is essential for correctness. The broker also decouples producers (stores) from consumers (the sync service), so a slow or temporarily-down consumer never blocks a store from selling.

6. Dead-Letter Queue (DLQ)

Events that repeatedly fail processing (due to malformed data, a downstream outage, or a bug) are routed here instead of being retried forever or silently dropped. This protects the main pipeline’s throughput while preserving the failed events for investigation and reprocessing.

7. Inventory Sync Service

The core business-logic consumer. It reads events off the stream, validates them, applies optimistic-concurrency-controlled updates to the inventory database, and triggers downstream notifications. This is typically a horizontally-scaled stateless service, so more instances can be added during peak load.

8. Conflict Resolution Engine

Handles the case where two updates to the same SKU arrive in a way that could conflict (for example, a store sale and an online sale processed by different service instances at nearly the same moment). Discussed in depth in the dedicated Conflict Resolution section below.

9. Reconciliation Service

A separate, lower-priority batch job that periodically compares the “official” inventory count against a full physical or logical recount (e.g., nightly cycle counts at stores), correcting any drift that accumulated from edge cases, bugs, or manual overrides. Real-time systems reduce drift dramatically but rarely eliminate it entirely — reconciliation is the safety net.

10. Distributed Cache (Redis)

Holds “hot” inventory counts for fast reads — the website’s product pages read from here far more often than from the primary database, because product page views vastly outnumber actual purchases. Cache entries are invalidated or updated immediately whenever the sync service commits a change.

11. Inventory Database

The system of record. Typically a horizontally sharded, replicated relational or NoSQL database (sharded by SKU or store ID) that stores the authoritative, versioned quantity for every SKU at every location, plus a virtual “total available for online sale” aggregate.

12. Event Store

An append-only log of every inventory-affecting event ever processed, kept separately from the mutable current-state database. This gives a full audit trail (crucial for retail — finance and loss-prevention teams need to answer “why does this count look wrong?”) and enables rebuilding state from scratch if needed (event sourcing).

13. Webhook Dispatcher & WebSocket Gateway

Once inventory changes, downstream systems need to know. The Webhook Dispatcher pushes updates to internal services (search indexing, recommendation engines); the WebSocket Gateway pushes live updates to any open storefront sessions, so a customer looking at a product page sees “Only 1 left!” update in real time without refreshing.

14. E-commerce Platform (API, storefront, search)

The consumer-facing side: the Inventory API serves availability data to the storefront and to search/catalog services, which decide whether to show a product as purchasable, low-stock, or sold out.

Let’s zoom into how a single sale actually moves through this system, mechanically, from the moment a cashier scans a barcode.

Step-by-step internal flow

1. Local transaction commit: The POS completes the sale in its local transactional store first. The customer gets their receipt immediately — this step never waits on the network.
2. Event construction: The Store Sync Agent builds an InventoryChangeEvent containing SKU, store ID, delta (-1), a unique event ID (UUID), a timestamp, and a logical version marker.
3. Local durability: The event is written to a local, disk-backed queue before any network call is attempted. This guarantees the event survives even if the store’s power or network drops at this exact instant.
4. Transmission with backoff: The agent attempts to POST the event to the cloud. On failure, it retries using exponential backoff with jitter (covered further in Performance & Scalability) rather than hammering the network immediately.
5. Gateway validation: The API Gateway authenticates the store device’s credentials, checks the payload schema, and enforces per-store rate limits before forwarding the event onward.
6. Publish to stream: The gateway publishes the validated event onto the Kafka topic, partitioned by SKU (or store+SKU) to guarantee ordering for that specific product.
7. Consumption: An Inventory Sync Service instance, subscribed to that partition, picks up the event in order.
8. Idempotency check: The service checks whether this eventId has already been processed (using a fast lookup, often backed by Redis with a short TTL). If yes, it acknowledges and discards the duplicate without reapplying the change.
9. Optimistic write: The service reads the current row (quantity + version), computes the new quantity, and issues a conditional update — succeeding only if the version still matches.
10. Conflict handling: If the conditional update fails because another concurrent event already changed the version, the service re-reads the latest state and retries the calculation (this is the Conflict Resolution Engine at work).
11. Cache update: Once the database write succeeds, the service updates (or invalidates) the corresponding Redis entry so future reads are fast and fresh.
12. Downstream fan-out: The service emits a follow-up “InventoryUpdated” notification for webhooks and WebSocket subscribers, so the storefront and internal systems reflect the new count.
13. Acknowledge and commit offset: Only after all the above succeeds does the consumer commit its Kafka offset, guaranteeing that if the service crashes mid-processing, the event will be redelivered and retried rather than silently lost.

Java example: the event model

public class InventoryChangeEvent {
    private final String eventId;      // UUID, used for idempotency
    private final String skuId;
    private final String storeId;      // "ONLINE" for e-commerce sales
    private final int    quantityDelta; // negative for sale, positive for restock/return
    private final Instant occurredAt;
    private final EventType type;      // SALE, RETURN, RESTOCK, MANUAL_ADJUSTMENT

    public InventoryChangeEvent(String eventId, String skuId, String storeId,
                                int quantityDelta, Instant occurredAt, EventType type) {
        this.eventId = eventId;
        this.skuId = skuId;
        this.storeId = storeId;
        this.quantityDelta = quantityDelta;
        this.occurredAt = occurredAt;
        this.type = type;
    }
    // getters omitted for brevity
}

This class is intentionally immutable — events represent facts that already happened and should never be mutated after creation, only appended to the log.

Java example: idempotent, optimistic-concurrency consumer

@Service
public class InventorySyncConsumer {

    private final InventoryRepository repository;
    private final DeduplicationStore dedupeStore; // backed by Redis, short TTL
    private final CacheClient cache;

    @KafkaListener(topics = "inventory-events", groupId = "inventory-sync-service")
    public void onEvent(InventoryChangeEvent event, Acknowledgment ack) {
        if (dedupeStore.alreadyProcessed(event.getEventId())) {
            ack.acknowledge(); // duplicate delivery - safe to skip
            return;
        }

        boolean applied = false;
        int attempts = 0;
        while (!applied && attempts < 5) {
            InventoryRecord current = repository.find(event.getSkuId(), event.getStoreId());
            int newQuantity = current.getQuantity() + event.getQuantityDelta();

            if (newQuantity < 0) {
                handleOversellAttempt(event, current);
                break;
            }

            // conditional write: succeeds only if version is unchanged
            applied = repository.compareAndSetQuantity(
                    event.getSkuId(), event.getStoreId(),
                    newQuantity, current.getVersion());
            attempts++;
        }

        if (applied) {
            dedupeStore.markProcessed(event.getEventId());
            cache.invalidate(event.getSkuId());
        }
        ack.acknowledge(); // commit offset only after side effects succeed
    }
}

The Outbox Pattern in practice

One subtlety glossed over above deserves its own explanation: how does the Store Sync Agent guarantee that a local sale and the corresponding outbound event are never inconsistent with each other — for instance, the sale commits locally but the process crashes before the event is even queued? The answer is the transactional outbox pattern: instead of writing the sale record and separately, as a second step, queueing the event, both writes happen inside the exact same local database transaction. A background poller then reads unsent rows from the outbox table and forwards them, marking each as sent only after a successful handoff. Because the sale and the outbox row are committed atomically together, there is no window where one exists without the other.

@Component
public class OutboxPoller {

    private final OutboxRepository outboxRepository;
    private final EventTransmitter transmitter;

    @Scheduled(fixedDelay = 500)
    public void flushPendingEvents() {
        List<OutboxRow> pending = outboxRepository.findUnsentBatch(100);
        for (OutboxRow row : pending) {
            try {
                transmitter.sendWithRetry(row.toEvent());
                outboxRepository.markSent(row.getId());
            } catch (Exception ex) {
                // leave unmarked; next poll cycle retries automatically
                break; // stop batch on first failure to preserve rough ordering
            }
        }
    }
}

This design means the local sale transaction, the local outbox row, and the eventual cloud delivery form three independently-recoverable steps, and a crash at any point simply resumes cleanly at the next step on restart — nothing is silently lost, and nothing is double-applied thanks to the idempotency key generated at outbox-row-creation time, not at transmission time.

Handling reservation expiry (BOPIS holds)

Earlier we introduced the idea of a “reserved” state for buy-online-pickup-in-store orders. Implementing this cleanly requires a time-to-live mechanism, since a reservation that’s never picked up must eventually release its held stock back into the available pool. A common implementation uses Redis’s native key expiration combined with keyspace notifications, so an expiry event triggers a corresponding release event back into the main pipeline, rather than requiring a separate polling job to scan for expired reservations.

@Service
public class ReservationService {

    private final RedisTemplate<String, String> redis;
    private static final Duration HOLD_DURATION = Duration.ofHours(72);

    public void reserveForPickup(String skuId, String storeId, String orderId) {
        String key = "reservation:" + storeId + ":" + skuId + ":" + orderId;
        redis.opsForValue().set(key, "HELD", HOLD_DURATION);
        // available count decrements immediately; total physical count is unchanged
    }

    // Triggered automatically by Redis keyspace notification on expiry
    public void onReservationExpired(String expiredKey) {
        ReservationKey parsed = ReservationKey.parse(expiredKey);
        InventoryChangeEvent releaseEvent = new InventoryChangeEvent(
                UUID.randomUUID().toString(), parsed.getSkuId(), parsed.getStoreId(),
                1, Instant.now(), EventType.RESERVATION_RELEASED);
        publishToStream(releaseEvent); // re-enters the same pipeline as any other event
    }
}

The sequence diagram below traces one concrete transaction end-to-end, matching each participant to a named architectural component, so the flow can be read directly off the diagram during a whiteboard interview.

Lifecycle of an inventory record

Beyond a single transaction, it’s worth tracing the full lifecycle of an inventory record from creation to archival:

Buy Online, Pick Up In Store (BOPIS) — a special data flow

BOPIS deserves its own mention because it inverts the usual flow: an online order needs to reserve stock at a specific physical store rather than simply decrementing a global count. This introduces a new state — “reserved” — sitting between “available” and “sold,” with its own expiry (if the customer doesn’t pick up within, say, 72 hours, the reservation releases automatically back into available stock).

This is the section interviewers probe hardest, because it’s where naive designs fall apart. The central question: what happens when two updates to the exact same inventory record race each other?

Where conflicts come from

• Concurrent sales: A store sale and a website sale for the last remaining unit, processed by two different service instances within milliseconds of each other.
• Out-of-order delivery: A restock event and a sale event for the same SKU arrive out of the order they actually happened in, due to retries or partition rebalancing.
• Multi-region writes: In a globally distributed deployment, two regions might both accept a write for the same SKU before replicating to each other.

Techniques used together

1. Partitioning by SKU (avoid conflicts by design)

The single most effective defense is architectural: by partitioning the Kafka topic (and often the database shard) by SKU, all events for a given product are always processed by the same consumer instance, in the order they were published. This eliminates most conflicts before they can happen, rather than resolving them after the fact.

2. Optimistic Concurrency Control (version numbers)

As shown in the Java example earlier, every write includes the version it expects to overwrite. If a concurrent write already advanced the version, the write is rejected and retried against fresh data — this is far more scalable than locking rows, because it doesn’t block other transactions while waiting.

3. Vector Clocks / Logical Timestamps for multi-region

In deployments spanning multiple regions with independent writers, simple version numbers aren’t enough to determine “what happened before what.” Vector clocks (or hybrid logical clocks) let the system distinguish truly concurrent, unrelated writes from causally-ordered ones, and apply a deterministic tiebreak (commonly last-writer-wins by timestamp, with the loser’s effect re-applied as a correction event) when writes are genuinely concurrent.

4. Reservation holds instead of direct decrements

For high-demand items, rather than everyone racing to decrement a shared counter, the system can issue short-lived reservations (e.g., “hold 1 unit for 90 seconds while checkout completes”). This converts a race condition into a queue of holds, each independently resolved, reducing contention on the underlying counter.

“Overselling is not a bug you eliminate — it’s a rate you engineer down to an acceptable business cost.”

Accepting a small, controlled oversell rate

It’s worth being direct about something interviewers respect: in a genuinely real-time, globally distributed system, driving oversell probability to literal zero is either impossible or so expensive (via strict global locking) that it destroys throughput and availability. Most large retailers instead engineer the system to keep oversell to a very small, known rate (for example, under 0.05% of transactions for hot items), and handle it operationally — auto-cancel with an apology discount, or offer a substitute — rather than chasing perfect prevention.

Key trade-off: consistency vs. availability

Nearly every production retail system chooses the first option, treating a rare, small oversell rate as a cheaper cost than lost in-store sales from a frozen register.

Key trade-off: real-time streaming vs. periodic polling

A national retailer with 2,000 stores, each running perhaps 10 registers, generates a continuous stream of transactions that spikes dramatically during events like Black Friday — commonly 50–100x normal peak throughput for a few hours. The architecture must be designed for that peak, not the average.

Horizontal scaling of the sync service

Because the Inventory Sync Service is stateless (all state lives in the database, cache, and Kafka), it scales horizontally simply by adding more consumer instances. Kafka’s consumer group mechanism automatically redistributes partitions across available instances, so scaling out is largely operational rather than a code change.

Partitioning strategy

Choosing the Kafka partition key carefully matters enormously. Partitioning purely by SKU can create “hot partitions” for viral, high-demand products (imagine a single trending sneaker release generating 40% of all traffic on one partition). A common refinement is partitioning by a hash of SKU + storeId, spreading load more evenly, while still guaranteeing ordering for the same SKU at the same store — the granularity that actually matters for correctness.

Retry strategy: exponential backoff with jitter

When the Store Sync Agent’s network call fails, retrying immediately in a tight loop across thousands of stores simultaneously can create a “retry storm” that overwhelms the very system that’s recovering. The standard fix is exponential backoff with random jitter — each retry waits progressively longer, with some randomness added so thousands of stores don’t all retry at exactly the same moment.

public class RetryWithBackoff {

    private static final int  MAX_ATTEMPTS  = 6;
    private static final long BASE_DELAY_MS = 500;

    public void sendWithRetry(InventoryChangeEvent event, HttpClient client) {
        int attempt = 0;
        while (attempt < MAX_ATTEMPTS) {
            try {
                client.post("/inventory/events", event);
                return; // success, stop retrying
            } catch (IOException ex) {
                attempt++;
                long exponential = BASE_DELAY_MS * (long) Math.pow(2, attempt);
                long jitter = ThreadLocalRandom.current().nextLong(0, BASE_DELAY_MS);
                sleepQuietly(exponential + jitter);
            }
        }
        // after MAX_ATTEMPTS, event stays safely in the local durable queue
        // and is retried later by a background flush job
    }
}

Little’s Law applied to capacity planning

Little’s Law (L = λ × W, average items in a system equals arrival rate times average time in the system) is directly useful here: if the sync pipeline must sustain 50,000 events/second at peak, and the target end-to-end processing time is 2 seconds, then the pipeline must comfortably hold roughly 100,000 in-flight events without backing up — which drives concrete decisions about Kafka partition counts and consumer instance counts.

Caching to protect the database

Product page views vastly outnumber purchases — commonly by a factor of 100:1 or more. Serving “is this in stock” reads from Redis instead of the primary database absorbs the overwhelming majority of read traffic, leaving the database free to handle the comparatively rarer write load from actual sales.

Connection pooling

Every Inventory Sync Service instance needs a database connection to apply its writes, and opening a brand-new TCP connection (plus authentication handshake) for every single event would be disastrously slow and would exhaust the database’s max-connection limit almost immediately under real load. Instead, each service instance maintains a small pool of pre-established, reusable connections (commonly managed by a library like HikariCP in the Java ecosystem), borrowing a connection for the duration of a single write and returning it immediately afterward. Sizing this pool correctly is itself a balancing act: too few connections and the service queues up waiting for one to free up, becoming an artificial bottleneck; too many connections and the database itself becomes overwhelmed managing more concurrent connections than it can efficiently schedule work across, since each open connection consumes memory and scheduling overhead on the database server regardless of whether it’s actively doing work at any given moment.

Capacity planning example

Consider a concrete planning exercise for a retailer expecting Black Friday peak traffic of 40,000 inventory events per second, with a target end-to-end sync latency of 2 seconds under load.

The key discipline here is planning for the multiplier, not the average — a system sized only for typical Tuesday-afternoon traffic will fail precisely on the days it matters most, which is why load testing explicitly simulates peak-multiplier scenarios well before the actual event, giving time to find and fix bottlenecks under controlled conditions rather than live ones.

Auto-scaling policy design

Rather than scaling the Inventory Sync Service purely on CPU utilization (a common default that doesn’t actually reflect whether the service is keeping up with its real job), the more meaningful auto-scaling signal is Kafka consumer lag itself — the number of unprocessed messages waiting per partition. A policy that adds instances when lag crosses a threshold (and removes them once lag has been near zero for a sustained period, to avoid flapping) scales the fleet in direct response to the actual backlog, which is precisely the thing the business cares about.

No single point of failure

Every layer in the architecture is deployed with redundancy: the load balancer sits in front of multiple API Gateway instances across at least two availability zones; Kafka runs as a multi-broker cluster with topic replication factor of at least 3, so the loss of one broker doesn’t lose data; the inventory database is replicated with automatic failover; and Redis runs in a clustered, replicated configuration.

Store-level resilience

The Store Sync Agent’s local durable queue is the most important reliability feature in the entire system for the business: it guarantees that a store’s ability to sell is completely decoupled from the availability of the cloud backend. Even a multi-hour cloud outage should never stop a physical register from selling — events simply queue locally and flush once connectivity is restored.

Disaster recovery

Because the Event Store retains a full, ordered history of every inventory change, the entire current-state database can be rebuilt from scratch by replaying events — a technique called event sourcing. This is the ultimate disaster-recovery mechanism: even a catastrophic loss of the primary database is recoverable, given the event log survives (which is why it, too, is replicated across regions).

Graceful degradation

If the Inventory Sync Service falls behind (say, during an unexpected traffic spike), the system should degrade gracefully rather than fail completely: the storefront can fall back to showing slightly stale “last known” stock levels from cache rather than erroring out, while the pipeline catches up in the background.

Defining and meeting SLAs

High availability isn’t a vague aspiration — it needs concrete, measurable targets that different parts of the system can be held to. A typical breakdown separates the availability target for the ingestion path (can stores always submit events successfully, even if slowly) from the freshness target for the read path (how stale can the storefront’s displayed availability be before it’s considered a violation). A realistic set of targets might commit to 99.99% availability for event ingestion (allowing under an hour of full unavailability per year) alongside a 95th-percentile end-to-end sync latency under 5 seconds, with explicit acknowledgment that the tail beyond that percentile can occasionally run longer during extreme peak load without breaching the overall commitment, since chasing five-nines latency across every single event is disproportionately expensive relative to the marginal business value it delivers.

Multi-AZ redundancy within a single region

Even before considering multi-region deployment, every component within a single cloud region is spread across multiple availability zones — physically separate data centers with independent power and networking, connected by low-latency links. The load balancer routes across gateway instances in at least two zones; Kafka brokers and database replicas are similarly distributed, so the loss of one entire data center (a rare but real event — fires, power grid failures, and construction accidents have all caused real-world outages) doesn’t take down the whole regional deployment, only reduces its capacity until the affected zone recovers or is worked around.

Health checks and automatic failover

Every layer implements active health checking rather than relying purely on reactive alerting: the load balancer continuously probes gateway instances and stops routing to any that fail to respond correctly; the database cluster’s replication manager continuously monitors replica health and promotes a new primary automatically if the current primary becomes unreachable; and Kafka’s own internal broker election mechanism reassigns partition leadership away from a failed broker without operator intervention. The overarching design principle is that recovery from any single-component failure should require no human action for the system to keep functioning, even if a human is still paged to investigate and permanently fix the underlying cause afterward.

Runbooks and operational readiness

Automated recovery handles the majority of routine failures, but well-documented runbooks matter for the failure modes that do require human judgment — for instance, deciding whether to manually fail traffic away from a region experiencing partial, ambiguous degradation rather than a clean total outage, where the automated health checks alone might not confidently trigger failover. Well-rehearsed runbooks (ideally tested periodically through planned failover drills, not left untested until a real incident) turn a stressful, error-prone live incident into a calmer, checklist-driven response.

Authenticating store devices

Each Store Sync Agent authenticates using a unique, per-device credential (commonly mutual TLS with a certificate provisioned during device setup, or a scoped API key stored in a hardware-backed secure element). This prevents a compromised or spoofed device from injecting fraudulent inventory events.

Least privilege and scoped access

A store device’s credentials should only permit publishing events for its own store ID — never for arbitrary stores. The API Gateway enforces this authorization check on every request, rejecting any event where the authenticated device’s store ID doesn’t match the event payload’s store ID.

Transport and data protection

All traffic between stores and the cloud travels over TLS. Sensitive fields (if any, such as pricing tied to loyalty programs) are further protected, and the event store — being a permanent audit record — is encrypted at rest.

Protecting against abuse

Rate limiting at the API Gateway protects against both accidental bugs (a misconfigured agent retrying too aggressively) and deliberate abuse (an attacker attempting to flood the pipeline with fraudulent events to manipulate stock visibility, a real risk for high-demand product drops).

Auditability

Because every event is permanently retained in the Event Store with its originating device identity and timestamp, any suspicious inventory manipulation (for example, an insider fraudulently marking stock as sold to cover theft) leaves an unforgeable trail for loss-prevention teams to investigate.

Secrets management for store devices

Thousands of physical devices, spread across locations with inconsistent physical security, represent a meaningfully larger attack surface than a small number of cloud data centers. Device credentials are therefore never embedded as static, long-lived secrets baked into firmware images (which, once leaked from a single compromised store, would let an attacker impersonate every store running that firmware version). Instead, credentials are provisioned per-device during setup, stored in hardware-backed secure storage where available, and rotated on a defined schedule automatically, so a compromised device’s credentials have a bounded window of usefulness to an attacker and can be individually revoked without affecting any other store.

Detecting anomalous inventory manipulation

Beyond preventing unauthorized access, the system benefits from behavioral anomaly detection layered on top of the audit trail — flagging patterns like an unusually high rate of manual stock adjustments from a single device, adjustments clustering suspiciously around a specific employee’s shifts, or a device suddenly reporting sales inconsistent with its historical volume profile. These signals don’t block transactions in real time (that would risk disrupting legitimate sales), but feed into loss-prevention dashboards for investigation, turning the audit trail from a purely reactive forensic tool into a proactive detection capability.

Encryption at rest and key management

The Event Store, being a permanent and complete record of every transaction ever processed, is a particularly sensitive target — a breach here would expose the retailer’s entire historical sales pattern. Encryption at rest, with encryption keys managed through a dedicated key-management service rather than embedded in application configuration, ensures that even a compromised storage volume or backup snapshot doesn’t expose readable data without also compromising the separate key-management layer, adding defense in depth rather than relying on any single control.

Key metrics to track

Distributed tracing

Because one sale touches many services (agent, gateway, broker, sync service, cache, database, webhook dispatcher), attaching a consistent trace ID to each event from the moment it’s created lets engineers follow a single transaction’s full journey across every hop — essential for diagnosing “why did this one sale take 40 seconds to sync” incidents.

Alerting philosophy

Alerts should be tied to customer-facing or business-facing symptoms (rising oversell rate, growing consumer lag crossing a threshold, DLQ volume spike) rather than purely internal signals, so on-call engineers are paged for things that actually matter to the business, and dashboards remain trustworthy rather than noisy.

Multi-region deployment

Large retailers typically deploy the cloud-side components across multiple regions for both latency (stores connect to the nearest region) and disaster recovery. This reintroduces the multi-writer conflict problem discussed earlier, which is why vector-clock-based conflict resolution becomes necessary at this scale rather than optional.

Containerization and orchestration

The API Gateway, Inventory Sync Service, Reconciliation Service, and Webhook Dispatcher are typically packaged as containers and run on an orchestrator (Kubernetes is the common choice), enabling independent auto-scaling of each component based on its own load characteristics — the sync service, for instance, scales based on Kafka consumer lag rather than simple CPU usage.

Blue-green and canary rollouts

Given how business-critical correctness is here, changes to the Inventory Sync Service are typically deployed via canary releases — a new version handles a small percentage of partitions first, with oversell rate and error rate monitored closely before the rollout proceeds to full traffic.

Infrastructure as Code

The full topology — Kafka topic configuration, database schemas, gateway routing rules, autoscaling policies — is defined declaratively (commonly via Terraform), so environments (staging, production, disaster-recovery region) stay consistent and reproducible.

Store-side deployment constraints

Unlike cloud services, the Store Sync Agent runs on modest, sometimes older, in-store hardware, with intermittent connectivity and no dedicated on-site engineer. Deployments to store agents therefore favor small, infrequent, thoroughly tested updates pushed during off-hours, with automatic rollback if a health check fails after an update.

Staged rollout across a fleet of thousands of stores

Pushing a new Store Sync Agent version to every store simultaneously would be reckless — if the new version has a subtle bug, it could affect every register in every location at once, a far worse outcome than a bug in a single cloud service instance behind a load balancer. Instead, updates roll out in waves: first to a small number of low-traffic pilot stores, monitored closely for a defined soak period; then to a broader percentage; then, only once confidence is established, to the full fleet. Each wave’s health is judged against the same metrics discussed in the Monitoring section — error rate, local buffer growth, successful transmission rate — with automatic pause-and-rollback if any wave shows regression before it can spread further.

Handling long-tail, rarely-connected devices

Not every store has equally reliable connectivity — some locations, particularly in regions with less developed infrastructure, may go days between successful cloud syncs. The deployment and versioning strategy has to tolerate a fleet where devices are running a wide spread of agent versions simultaneously, sometimes months apart, rather than assuming a tightly synchronized fleet. This reinforces the earlier point about additive, backward-compatible schema evolution: it isn’t just a nice practice, it’s a hard requirement given the realistic diversity of connectivity across a large physical footprint.

Region failover for the cloud side

If an entire cloud region becomes unavailable, stores normally routed there need to fail over to a healthy region automatically, typically via DNS-based routing with health checks, or a global load-balancing layer sitting above the regional load balancers discussed earlier. Because inventory data for a given SKU might have been most recently updated in the now-unavailable region, the receiving region needs replicated, reasonably fresh data to avoid making decisions against a stale snapshot — which is why cross-region database replication and Event Store replication are treated as core requirements for any multi-region deployment of this system, not optional enhancements.

Choosing the inventory database

The inventory database needs to support high write throughput, strong per-row consistency (for the optimistic concurrency checks), and horizontal scalability. Common real-world choices include a sharded relational database (e.g., PostgreSQL or MySQL, sharded by SKU or store) for strong per-row guarantees, or a distributed NoSQL store like DynamoDB or Cassandra with conditional writes, when the scale demands it.

Sharding strategy

Sharding by SKU keeps all writes for a given product on one shard (simplifying the optimistic-concurrency logic), at the cost of potential hot shards for viral products — the same trade-off discussed in the Kafka partitioning section, and the two are usually aligned so a SKU’s Kafka partition and database shard correspond consistently.

Replication for reads

Read replicas serve less time-sensitive queries (analytics, reporting dashboards, the Reconciliation Service’s audit queries), keeping that load off the primary write path entirely.

Caching layer design

Redis serves the “hot path” read: “is SKU X in stock, and roughly how many are available?” The cache is updated by the sync service the moment a write commits (write-through pattern) rather than relying purely on TTL expiry, minimizing the staleness window customers can observe.

Load balancer configuration

The Layer 7 load balancer in front of the API Gateway fleet performs active health checks (rejecting instances failing to respond correctly), supports connection draining during deployments (so in-flight requests complete before an instance is removed from rotation), and can apply weighted routing during canary releases.

SQL vs. NoSQL for this specific workload

Many large retailers actually run both side by side: a NoSQL store as the primary, high-throughput system of record for the hot write path described throughout this tutorial, alongside a relational data warehouse fed asynchronously (via the Event Store) for the complex analytical and reporting queries that a NoSQL access-pattern model handles poorly. This is itself an application of CQRS at the infrastructure level — different storage engines for different access patterns, kept in sync through the same event pipeline rather than through direct coupling.

Schema design for the inventory table

A minimal but production-realistic schema for the core inventory record includes the SKU identifier, the store identifier (with a reserved sentinel value such as “ONLINE” or “WAREHOUSE” representing the e-commerce fulfillment pool itself), the current quantity, the version number used for optimistic concurrency, a separately-tracked reserved quantity (for BOPIS holds), and a last-updated timestamp used both operationally and for detecting records that haven’t been touched in unexpectedly long periods, which can itself be a useful signal that a store’s sync pipeline has silently stalled.

Index design for the read path

Because the dominant read pattern is “look up availability by SKU across all locations, or by SKU at one specific location,” the primary index is typically a composite key on (SKU, store), with a secondary global index on SKU alone to support the aggregated “total available anywhere” queries the storefront needs for its default, location-agnostic product page view before a customer has specified a store or delivery preference.

Key API contracts

Why microservices, not a monolith, for this problem

Splitting the Inventory Sync Service, Conflict Resolution Engine, and Reconciliation Service into independently deployable services (even though they’re logically related) allows each to scale, deploy, and fail independently — a spike in real-time sync load doesn’t need to affect the (much less time-sensitive) nightly reconciliation job’s deployment schedule or resource allocation.

API versioning

Because store devices update infrequently and can’t all be forced to upgrade simultaneously, the event schema and API contract must support backward compatibility for extended periods — new optional fields are added, never repurposed or removed, and breaking changes are rolled out as a new API version running in parallel with the old one until every store agent has migrated.

Drawing service boundaries

A common early mistake is drawing microservice boundaries around technical layers (a “database service,” a “cache service”) rather than around business capabilities. In this system, the boundaries instead follow distinct business responsibilities: the Inventory Sync Service owns applying validated state changes; the Conflict Resolution Engine owns deciding correctness when writes collide; the Reconciliation Service owns long-run drift correction; and the Webhook Dispatcher owns notifying external consumers. Each of these can be understood, tested, deployed, and scaled on its own terms, and each has a single, clear owner responsible for its correctness — a property that matters enormously as the engineering organization around this system grows past a single team.

Synchronous vs. asynchronous API design

Not every API in this system is asynchronous — the storefront’s availability lookup (GET /inventory/{sku}/availability) is a synchronous, low-latency read, because a customer looking at a product page needs an immediate answer, not an eventual one. The architectural discipline is being deliberate about which interactions are synchronous (fast, cache-backed reads with tight latency budgets) and which are asynchronous (anything that mutates state, which flows through the event pipeline) rather than defaulting to synchronous calls everywhere out of habit, which is precisely the anti-pattern called out earlier regarding the sale path itself.

Rate limiting and quota design per API consumer

Different API consumers warrant different rate-limiting treatment: a store device publishing routine sale events gets a generous, steady quota sized to its realistic maximum transaction rate; a bulk internal reporting job querying historical data gets a much lower-priority quota so it can never crowd out latency-sensitive storefront traffic; and external partner integrations (for instance, a third-party marketplace reading availability data) get their own separately-tracked, more conservative quota with its own monitoring, so one partner’s misbehaving integration can’t degrade service for anyone else sharing the same gateway infrastructure.

Design patterns that work well here

Anti-patterns to avoid

A useful mental test for spotting anti-patterns in a design review is asking, for any proposed shortcut, “does this bypass the idempotency layer, the conflict resolution layer, or the audit trail?” If the answer is yes, the shortcut is very likely trading a small, immediate convenience for a much larger, harder-to-diagnose correctness problem later — exactly the kind of trade that tends to surface, expensively, during the highest-traffic day of the year rather than during ordinary testing.

Testing strategies for distributed inventory systems

Testing a system like this well is meaningfully different from testing a typical CRUD application, because most of the hardest bugs only manifest under concurrency, network failure, or message reordering — conditions that don’t naturally occur in a simple unit test running on a developer’s laptop.

Unit and contract testing

Individual pieces — the idempotency check, the optimistic-concurrency retry loop, the conflict resolution tiebreak logic — are each unit-testable in isolation with straightforward, deterministic inputs. Contract tests additionally verify that the event schema published by store devices and the schema expected by the Sync Service stay compatible as both evolve independently over time, catching accidental breaking changes before they reach production.

Chaos and fault-injection testing

Because so much of this system’s value lies in how it behaves during failure (network partitions, broker outages, duplicate delivery), deliberately injecting these failures in a staging environment is essential rather than optional. Common techniques include randomly killing Sync Service instances mid-processing to verify no events are lost or double-applied, artificially delaying network calls between the Store Sync Agent and the cloud to confirm local buffering behaves correctly, and deliberately delivering duplicate and out-of-order events to confirm the idempotency and ordering guarantees actually hold under adversarial conditions rather than merely the happy path.

Load and peak-simulation testing

Given the earlier capacity-planning discussion around Black Friday-scale traffic, load tests should specifically simulate the peak multiplier scenario, not just steady average throughput, well ahead of the actual event — surfacing bottlenecks (an undersized connection pool, an under-partitioned Kafka topic, an auto-scaling policy that reacts too slowly) while there’s still time to fix them calmly, rather than discovering them live during the highest-revenue hours of the year.

Reconciliation-driven correctness testing

Because the Reconciliation Service independently recomputes and compares inventory state, it doubles as an ongoing correctness test running continuously in production — any drift it detects and corrects is effectively a bug report about the real-time pipeline, and tracking the reconciliation correction rate over time is one of the most honest signals of how well the real-time system is actually performing its job.

Best practices

• Always design events to be idempotent-safe by including a unique event ID from the moment of creation, at the source.
• Prefer additive, backward-compatible schema evolution for anything a large, slowly-updating fleet of devices depends on.
• Make offset/acknowledgment commits happen strictly after side effects succeed, never before.
• Treat the event store as the permanent source of truth and the current-state database as a derived, rebuildable projection.
• Instrument end-to-end latency and consumer lag from day one — these are the metrics that reveal problems before customers do.
• Build the Reconciliation Service early, not as an afterthought — real-time pipelines drift over years of edge cases, bugs, and manual interventions, and reconciliation is the long-term correctness backstop.

Common mistakes

• Assuming the network is reliable and skipping local durable buffering on store devices.
• Choosing a database sharding key that doesn’t align with the message broker’s partitioning key, creating unnecessary cross-shard coordination.
• Over-indexing on strong consistency for a domain (inventory counts) where the business genuinely tolerates brief staleness in exchange for availability.
• Under-provisioning Kafka partitions early, making it expensive and disruptive to repartition later once ordering guarantees depend on the existing layout.
• Neglecting to test the failure paths (network partition, duplicate delivery, out-of-order events) as thoroughly as the happy path.

Organizational best practices

Beyond the purely technical practices above, a system this central to revenue benefits from clear organizational ownership: a single team accountable for the Sync Service and Conflict Resolution Engine’s correctness, a documented on-call rotation with the runbooks discussed in the High Availability section, and a lightweight but real change-review process for anything touching the event schema, given how widely that schema is depended upon across a large device fleet and multiple downstream consumers. Retailers that treat this system as “just another internal service” rather than as the revenue-critical infrastructure it actually is tend to underinvest in exactly the areas — monitoring, staged rollout discipline, reconciliation — that matter most once the system is operating at real scale.

Real-world & industry examples

The common thread

Across every one of these companies, the underlying pattern is the same: event-driven ingestion at the point of sale, asynchronous propagation through a durable message broker, and a central, eventually-consistent inventory service feeding both the storefront and internal fulfillment logic. The specific technology choices differ (Kafka vs. Kinesis vs. proprietary systems), but the architectural shape converges because it’s the shape the problem actually demands, independent of company size, industry vertical, or the particular cloud provider each organization happens to have standardized on internally.

A closer look: why RFID changed the game for Zara

It’s worth explaining why RFID matters architecturally, not just as a retail buzzword. Barcode scanning only generates an inventory event at the moment of a checkout scan — the system has no visibility into stock sitting on a shelf versus stock sitting in a stockroom versus stock that’s actually missing (theft, misplacement, damage) until someone manually counts it. RFID tags, read continuously by fixed or handheld readers throughout the store, generate a much richer, near-continuous stream of location and presence events, dramatically shrinking the gap between “what the system believes exists” and “what physically exists” — directly reducing the reconciliation drift discussed earlier in this tutorial, and making item-level (not just SKU-level aggregate) inventory tracking practical at scale.

A closer look: Walmart’s scale problem

With thousands of stores each carrying on the order of 100,000 distinct SKUs, Walmart’s inventory system is coordinating a state space of hundreds of millions of individual (SKU, store) combinations, any of which can change at any moment. This is precisely the scale at which the architectural choices discussed throughout this tutorial — partitioned event streams, sharded databases, cache-first reads — stop being “good practice” and become the only workable approach; a naive single-database design would collapse under the sheer combinatorial size of the problem long before peak traffic even becomes a factor.

A closer look: Amazon’s grocery integration challenge

Grocery inventory poses an extra wrinkle beyond general retail: perishability and highly variable unit measures (a pound of ground beef isn’t a fixed discrete count the way a t-shirt SKU is). Integrating Whole Foods’ physical grocery inventory into Amazon’s online grocery ordering required extending the event model to handle weight-based and perishable-goods inventory, where “available quantity” is a continuously-updating estimate rather than a simple integer, and where a restock event might need to account for spoilage-driven write-offs as a distinct event type from ordinary sales.

Common interviewer follow-ups — the answers to which show whether a candidate has genuinely internalized the shape of the design, or merely memorized its parts.

Real-time inventory synchronization between e-commerce and physical retail is fundamentally a distributed systems problem disguised as a retail feature. The solution rests on a handful of core ideas repeated at every layer: capture events durably at the source, decouple producers from consumers through a durable ordered stream, make every write idempotent and version-checked, and accept eventual consistency deliberately rather than fighting it — because the business genuinely prefers a highly-available register with occasional brief staleness over a perfectly consistent one that sometimes can’t sell at all.

It’s worth returning, one last time, to the framing question opened at the very start of this tutorial: how does a company guarantee that a jacket sold in a physical store disappears from the website within a handful of seconds, without ever double-selling it? By now, the answer should feel less like a single clever trick and more like a coherent set of disciplined engineering choices, each addressing one specific failure mode, layered together — durable local buffering so a store never depends on the cloud to sell something; an ordered, partitioned event stream so updates propagate quickly without tightly coupling every component to every other; idempotent, version-checked writes so duplicates and races can’t silently corrupt the truth; and a reconciliation safety net so that, over the long run, small inevitable drifts get caught and corrected rather than quietly accumulating. None of these ideas are exotic in isolation; the skill lies in combining them correctly, at the right layer, for the specific correctness and availability trade-offs this particular business problem demands.