Imagine you run an online store. You do not keep every product in your own warehouse. Instead, hundreds of suppliers send you a list of what they have in stock, and your website shows that stock to customers. This list is called an inventory feed. It usually arrives through an API — a way for two computer systems to talk to each other automatically, without a human typing anything.

Now imagine one supplier’s system goes down for two hours, or their API starts sending data very slowly, or it silently stops sending updates altogether. Your website still shows their products as “In Stock” because it has no new information telling it otherwise. Customers keep ordering. You keep accepting money. Then the feed comes back, and you discover you have sold 40 units of a product that only had 5 units left three hours ago. You now must cancel orders, refund money, apologise to customers, and possibly lose them forever.

This tutorial designs a complete system that prevents this from happening. We will build it piece by piece, in simple language, the same way you would explain it to a smart 10-year-old, and then go deep enough that a senior engineer or system design interviewer would be satisfied. Every major decision will include a real analogy, a beginner example, a software example, and how big companies like Amazon, Walmart, and Flipkart actually solve this problem.

Before we design anything, let’s see how this problem grew alongside the internet economy — it did not always exist in this shape.

3.1 What Exactly Is “Overselling”?

Overselling happens when your system accepts and confirms more orders for a product than you actually have available to ship. If a supplier has 5 units of a lamp left, and your system sells 8 because it thought there were still 20 in stock, you have oversold by 3 units.

3.2 Why Does the Supplier Feed Break in the First Place?

• Network failures — the internet connection between you and the supplier’s server drops.
• Supplier server downtime — their server crashes, is under maintenance, or is overloaded.
• Slow responses (latency spikes) — the API technically works but takes 30 seconds instead of 200 milliseconds, causing timeouts.
• Silent data corruption — the feed sends a response, but the numbers are wrong, incomplete, or in the wrong format.
• Rate limiting — the supplier blocks you temporarily because you called their API too many times.
• Authentication expiry — an API key or token expires and nobody renewed it in time.
• Partial batch failures — out of 10,000 products in a feed update, 200 fail validation and are silently dropped, leaving stale stock numbers for just those 200.

3.3 Why This Matters for the Business

3.4 Quantifying the Risk

It helps to put numbers on this problem instead of treating it abstractly. Suppose a product sells an average of 2 units per minute during a promotion, and a supplier’s feed has been stale for 20 minutes because of an outage. If the system naively kept selling based on the last known number, it could sell up to 40 units that may not exist. If the true remaining stock was only 15 units, that is a potential oversell of 25 units — 25 real customers with confirmed orders that cannot be fulfilled. This simple multiplication (sales velocity multiplied by staleness duration) is exactly the calculation the safety stock buffer in Chapter 4.4 and Chapter 8.2 is designed to absorb automatically.

Before drawing a single architectural box, let’s put names to the small vocabulary that shows up in every layer of this design.

4.1 Source of Truth vs. Cached Copy

The supplier’s own database is the real, authoritative “source of truth” for how much stock exists. Your platform keeps a cached copy of that number so that your website is fast and does not depend on the supplier being online at that exact millisecond. The entire design challenge of this tutorial is: how do you keep that cached copy trustworthy, and what do you do when you cannot refresh it?

4.2 Push vs. Pull Integration

• Pull (polling): Your system calls the supplier’s API every few minutes and asks, “What’s your stock right now?” Simple to build, but wastes calls when nothing changed, and there is always a gap between polls.
• Push (webhook): The supplier’s system calls your API the instant something changes. Much fresher data, but it depends on the supplier building and maintaining that webhook reliably.

4.3 Circuit Breaker

A circuit breaker is a safety switch, borrowed from electrical engineering. In electrical wiring, if too much current flows, a fuse “breaks” the circuit before wires melt or a fire starts. In software, if a dependency (like a supplier’s API) starts failing repeatedly, the circuit breaker “opens” and your system stops calling it for a while, protecting itself from cascading failure and wasted resources, and instead falls back to a safe default behaviour.

4.4 Safety Stock Buffer

A safety stock buffer is a small quantity you deliberately hide from customers as a cushion against uncertainty. If a supplier reports 20 units, you might only show 17 as purchasable, keeping 3 as a buffer against feed staleness or counting errors.

4.5 Idempotency

Idempotency means doing an operation multiple times has the same effect as doing it once. This matters because networks can deliver the same feed update twice (duplicate messages), and if your system is not idempotent, it might subtract stock twice for the same event, creating wrong numbers.

4.6 Optimistic Locking

Optimistic locking is a way to safely update a shared number (like stock count) when many people might be trying to update it at the same time, without locking the whole database row for a long time. It checks “has this value changed since I last read it?” before committing a change, and retries if there is a conflict.

4.7 Eventual Consistency

Eventual consistency means that after an update happens, all parts of the system will reflect it — but not necessarily at the exact same instant. Your cache might show old stock for a few seconds after the database has the new number. This is normal and acceptable in most e-commerce systems, as long as the gap is small and reservation logic protects the checkout path.

4.8 CAP Theorem, Applied to Inventory

The CAP theorem states that a distributed system can only fully guarantee two of the following three at the same time: Consistency (every read sees the latest write), Availability (every request gets a response, even during failures), and Partition tolerance (the system keeps working even when parts of the network cannot talk to each other). Because a real network can and will experience partitions, the real choice in practice is between consistency and availability during a partition.

This inventory system deliberately chooses availability over strict consistency for the read path: when the supplier connection is partitioned (down), the product page still loads and still shows a number, rather than freezing or erroring out. It compensates for the resulting consistency risk using the safety buffer, reservation system, and reconciliation job, rather than trying to achieve impossible perfect real-time consistency across an unreliable third-party network.

4.9 Partitioning and Sharding

Partitioning means splitting data or workload into independent chunks so they can be processed in parallel and so a problem in one chunk does not affect the others. In this system, supplier ingestion work is partitioned by supplierId — every supplier is deterministically assigned (using consistent hashing) to one ingestion worker. As the number of suppliers grows, you add more workers and the hashing spreads suppliers across them automatically, without needing to redesign anything.

4.10 Replication and Consensus

Replication means keeping multiple copies of the same data on different machines, so losing one machine does not lose the data. The inventory database uses a primary-replica setup: writes go to a primary node, and that data is copied (replicated) to one or more standby replicas. If the primary fails, a consensus protocol (commonly Raft, used internally by many managed database and coordination systems) is used to have the healthy replicas agree on which one becomes the new primary, avoiding a scenario where two nodes both believe they are in charge and accept conflicting writes — a dangerous condition known as “split brain.”

4.11 Concurrency Control

Concurrency control is how a system safely handles many operations trying to touch the same data at the same time. There are two broad strategies: pessimistic locking, where you lock a row before reading it so nobody else can touch it until you are done (safe, but can create bottlenecks under high traffic), and optimistic locking (introduced in Chapter 4.6), where you do not lock anything up front, but detect conflicts at write time and retry. For a high-traffic flash sale where thousands of customers might try to buy the last few units of a product within the same second, optimistic locking generally performs better because it does not force requests to queue up waiting for a lock; it only pays a retry cost on the (usually rare) occasions where two requests genuinely collide.

Below is the full high-level architecture. Every major infrastructure component — the load balancer, the API gateway, the circuit breaker, the message queue, the cache, and the database — is shown as its own labelled box so you can see exactly where each piece sits in the request path.

5.1 Component-by-Component Explanation

Load Balancer

Sits in front of all your API servers and spreads incoming traffic across many machines so no single server gets overwhelmed. In production this is typically an AWS Application Load Balancer, NGINX, or HAProxy, doing Layer 7 (HTTP-aware) routing with health checks so it automatically stops sending traffic to a server that has crashed.

API Gateway

The single front door for all API traffic. It handles authentication (checking who is calling), rate limiting (stopping abuse), request routing (sending “get product” requests to the Inventory Service and “place order” requests to the Order Service), and can also do request validation and basic logging before traffic ever reaches your business logic.

Circuit Breaker (Feed Health Guard)

Wraps every outbound call to a supplier’s API. It tracks recent success and failure rates per supplier. If a supplier’s failure rate crosses a threshold (for example, more than 50% of calls failing in the last minute), the circuit “opens,” meaning the ingestion service stops calling that supplier for a cooldown period and instead uses fallback logic, described in Chapter 8.

Supplier Feed Ingestion Service

A dedicated microservice whose only job is to fetch or receive inventory updates from suppliers, validate them, and normalise them into your platform’s internal format before publishing them onward. Keeping this separate from the core Inventory Service means a badly-behaved supplier cannot directly slow down or crash the system customers interact with.

Message Queue (Kafka)

A durable, ordered pipe that decouples “receiving a feed update” from “applying a feed update.” The ingestion service publishes an event like “Product X now has 12 units,” and the Inventory Service consumes it whenever it is ready. This means a burst of 50,000 updates from a supplier does not overwhelm the database directly — the queue absorbs the burst and the consumer processes at a controlled pace.

Cache Layer (Redis)

Holds the current stock count in memory so that when a customer loads a product page, the answer comes back in single-digit milliseconds instead of querying the database every time. The cache is refreshed whenever the underlying stock changes in the database.

Inventory Database (PostgreSQL)

The durable source of truth on your side (as opposed to the supplier’s own database, which is the ultimate source of truth). Stores current stock counts, reservation records, and version numbers used for optimistic locking.

Safety Stock Buffer Service

A small rules engine that decides how much of the reported stock to actually expose to customers, based on supplier reliability, feed freshness, and product risk category (fast-moving items get a bigger buffer than slow-moving items).

Reconciliation Service

Runs on a schedule (for example every 15 minutes) and independently re-verifies stock for high-risk or high-velocity products directly against the supplier, catching any drift the real-time pipeline may have missed.

Monitoring and Alerting Stack

Watches feed freshness, circuit breaker state, error rates, and oversell incidents, and pages an on-call engineer when something crosses a dangerous threshold.

Let’s trace what happens, step by step, in the healthy (non-failure) case, before we look at what happens when things break.

6.1 Sequence Diagram of the Normal Flow

6.2 Handling Out-of-Order Events

Networks do not guarantee that messages arrive in the order they were sent. A supplier might publish “stock is now 10” followed quickly by “stock is now 8,” but due to network timing, your Inventory Service could receive the “8” event before the “10” event. If you apply them in the wrong order, you would end up incorrectly showing 10 units instead of the correct, more recent 8.

To prevent this, every inventory event carries a monotonically increasing sequence number or source timestamp from the supplier’s own system. The consumer compares the incoming event’s sequence number against the last-applied sequence number stored for that product. If the incoming event is older than what has already been applied, it is discarded as a stale, out-of-order duplicate rather than being applied and silently corrupting the stock count.

public void applyIfNewer(InventoryUpdateEvent event) {
    InventoryRecord record = inventoryRepo.findByProductId(event.getProductId());

    if (record != null && event.getSequenceNumber() <= record.getLastAppliedSequence()) {
        return; // stale or duplicate event, safely ignore
    }

    record.setSupplierReportedQuantity(event.getQuantity());
    record.setLastAppliedSequence(event.getSequenceNumber());
    inventoryRepo.save(record);
}

6.3 Java Example: An Idempotent Event Consumer

This consumer checks a unique event ID before applying an update, so replayed or duplicate messages from the queue never double-count stock changes.

@Service
public class InventoryEventConsumer {

    private final ProcessedEventRepository processedEvents;
    private final InventoryRepository inventoryRepo;

    public InventoryEventConsumer(ProcessedEventRepository processedEvents,
                                   InventoryRepository inventoryRepo) {
        this.processedEvents = processedEvents;
        this.inventoryRepo = inventoryRepo;
    }

    @KafkaListener(topics = "inventory-updates")
    public void handle(InventoryUpdateEvent event) {
        // Idempotency check: skip if we already processed this exact event
        if (processedEvents.existsByEventId(event.getEventId())) {
            return;
        }

        InventoryRecord record = inventoryRepo.findByProductId(event.getProductId());
        if (record == null) {
            record = new InventoryRecord(event.getProductId());
        }

        record.setSupplierReportedQuantity(event.getQuantity());
        record.setLastFeedTimestamp(event.getTimestamp());
        record.setFeedSource(event.getSupplierId());

        inventoryRepo.save(record);
        processedEvents.markProcessed(event.getEventId());
    }
}

Every inventory record in the system moves through a predictable lifecycle. Understanding this lifecycle is the key to designing correct failure handling.

7.1 Freshness-Aware Read Path

When a customer views a product page or tries to add an item to their cart, the Inventory Service does not just return a raw number — it returns a number adjusted for freshness and buffer rules. This keeps the risk-management logic centralised in one place instead of scattered across every caller.

public int getPurchasableQuantity(String productId) {
    InventoryRecord record = cache.get(productId);
    long ageSeconds = Duration.between(record.getLastFeedTimestamp(), Instant.now()).getSeconds();

    if (ageSeconds > staleThresholdSeconds) {
        return 0; // treat as unavailable until reconciled
    }

    int buffer = bufferPolicy.calculateBuffer(record, ageSeconds);
    return Math.max(0, record.getSupplierReportedQuantity() - buffer);
}

7.2 Handling Partial Batch Failures

Suppliers often send updates as a single large batch — for example, 10,000 products in one API response. A common and dangerous failure mode is a partial batch failure: 9,800 records are valid and 200 fail validation (bad format, missing fields, impossible values like negative stock). A naive system either rejects the entire batch (unnecessarily stalling 9,800 good updates) or accepts everything blindly (letting 200 bad records corrupt the data).

The correct approach is row-level validation with partial acceptance: valid records are processed and published normally, while invalid records are routed to a separate “dead letter” queue for manual or automated review, and the affected products are individually flagged as having a stale or untrusted feed — exactly as if that one supplier connection had failed just for those specific items. This keeps a formatting bug in a tiny fraction of a feed from blocking updates to everything else.

This is the heart of the system. When the supplier’s feed is delayed or fails outright, we need a clear, automatic decision process rather than customers silently being allowed to buy phantom stock.

8.1 The Circuit Breaker State Machine

The circuit breaker tracks three states:

• Closed: Everything is healthy. Calls to the supplier flow through normally.
• Open: Too many recent failures. The breaker stops calling the supplier for a cooldown period and immediately triggers fallback behaviour.
• Half-Open: After the cooldown, the breaker allows a small number of test calls through. If they succeed, it closes again; if they fail, it re-opens.

8.2 What “Fallback” Actually Means Here

Fallback does not mean “keep selling as normal and hope for the best.” It means:

1. Freeze the last known good stock snapshot for that supplier — do not trust any newer, unverified number.
2. Apply an aggressive safety buffer that grows the longer the feed stays down (for example, reduce shown quantity by an increasing percentage every 10 minutes of staleness).
3. Once staleness passes a hard limit (say 30 minutes for fast-moving products, longer for slow movers), mark the product as “Temporarily Unavailable” rather than guessing.
4. Continue accepting orders only up to the buffered quantity, and place any accepted order into a “provisional” state that can be automatically cancelled with an apology and refund if reconciliation later shows it should not have been accepted — this should be rare if buffers are sized correctly.

8.3 Java Example: Circuit Breaker Using Resilience4j

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
        .failureRateThreshold(50)                            // open if 50% of calls fail
        .slowCallRateThreshold(50)
        .slowCallDurationThreshold(Duration.ofSeconds(3))
        .waitDurationInOpenState(Duration.ofMinutes(2))
        .permittedNumberOfCallsInHalfOpenState(5)
        .slidingWindowSize(20)
        .build();

CircuitBreaker breaker = CircuitBreaker.of("supplier-" + supplierId, config);

Supplier<InventorySnapshot> decoratedCall = CircuitBreaker
        .decorateSupplier(breaker, () -> supplierClient.fetchInventory(supplierId));

Try<InventorySnapshot> result = Try.ofSupplier(decoratedCall)
        .recover(throwable -> fallbackService.getLastKnownGoodSnapshot(supplierId));

InventorySnapshot snapshot = result.get();

8.4 Timeout and Retry Strategy

Every outbound call to a supplier must have an aggressive timeout (for example 3 seconds) — never wait indefinitely. Retries should use exponential backoff with jitter (waiting 1s, then 2s, then 4s, with some randomness added) so that many services retrying at once do not all hammer the supplier at the exact same moment, which is called a “thundering herd.”

8.5 Distinguishing Between the Three Failure Types

Not all failures should be treated the same way, because they carry different information:

Confusing “the supplier told us zero” with “we could not reach the supplier” is a common and costly bug — the first should immediately hide the product, while the second should trigger the fallback and buffer logic rather than jumping straight to hiding it, since the item may well still be in stock.

8.6 The Exponential Backoff Formula

A standard exponential backoff with jitter calculates the wait time before the next retry as:

waitTime = min(maxWait, baseWait * (2 ^ attemptNumber)) + random(0, jitterMs)

For example, with a baseWait of 500 milliseconds and a maxWait cap of 30 seconds, the first retry waits around half a second, the second around a second, the third around two seconds, and so on, doubling each time until it hits the cap. The random jitter added on top prevents many failed requests from all retrying at exactly the same moment and overwhelming the supplier the instant it comes back online.

Even with buffers and circuit breakers, the real defence against overselling happens at the exact moment of checkout, using inventory reservation combined with optimistic locking, plus a background reconciliation job that catches anything that slipped through.

9.1 State Diagram of an Order Versus Stock

9.2 Java Example: Optimistic Locking on Stock Reservation

@Entity
public class InventoryRecord {
    @Id
    private String productId;
    private int availableQuantity;

    @Version
    private long version; // JPA uses this for optimistic locking automatically
}

@Service
public class ReservationService {

    private final InventoryRepository repo;

    @Retryable(value = OptimisticLockException.class, maxAttempts = 3)
    @Transactional
    public boolean reserveStock(String productId, int requestedQty) {
        InventoryRecord record = repo.findByProductId(productId);

        if (record.getAvailableQuantity() < requestedQty) {
            return false; // not enough stock, fail fast
        }

        record.setAvailableQuantity(record.getAvailableQuantity() - requestedQty);
        repo.save(record); // JPA checks the version column; throws if it changed underneath us
        return true;
    }
}

The @Version field means that if two customers try to buy the last unit at the same time, only one write will succeed. The database rejects the second write because the version number no longer matches what that request originally read, forcing a retry that will correctly see zero stock remaining.

9.3 Reservation Timeout

When a customer adds an item to their cart or begins checkout, the system should place a short-lived reservation (for example, 10 minutes) rather than permanently deducting stock. If the customer abandons checkout, the reservation automatically expires and the stock returns to the available pool. This prevents “phantom” stock lockups where abandoned carts quietly hoard inventory.

9.4 The Reconciliation Job

On a schedule, a background job re-fetches authoritative stock numbers directly from each supplier for high-risk products (fast sellers, low-stock items, or products flagged during a recent feed outage) and compares them against what the platform currently believes. Any mismatch triggers an automatic correction and, if needed, a review of any orders placed during the mismatch window.

9.5 What Happens When the Reconciliation Job and the Real-Time Pipeline Disagree

Occasionally, the reconciliation job’s fresh call to the supplier will disagree with what the real-time pipeline currently believes. In this case, the reconciliation result — being a fresh, directly-verified read — always wins and overwrites the cached value, since it is by definition more current. The size and direction of the disagreement is also logged as a data point that feeds back into that supplier’s reliability score, which in turn affects how large a safety buffer future orders for that supplier’s products will use. A supplier that frequently disagrees with reconciliation earns a larger, more conservative buffer automatically, without a human needing to notice and manually adjust it.

10.1 Choosing the Database

PostgreSQL (or another strong relational database like MySQL) is the right default for the inventory table because stock counts need strong consistency guarantees, row-level locking, and transactions — features relational databases provide natively. A NoSQL document store could work for the raw supplier feed logs (which are write-heavy and do not need transactions), but the authoritative stock count itself belongs in a system that supports ACID transactions.

10.2 Caching Strategy

• Cache-aside pattern: The Inventory Service reads from Redis first; on a cache miss, it reads from PostgreSQL and populates the cache.
• Write-through updates: Whenever stock changes in the database, the cache is updated in the same operation, so reads never see very stale data.
• Short TTL as a backstop: Even with write-through updates, set a short time-to-live (for example, 60 seconds) on cached stock values, so any missed invalidation self-heals quickly.

10.3 Load Balancing Considerations

The load balancer in front of the API layer should use health checks that go beyond “is the process running” — it should check that the service can actually reach its database and cache, so a server with a broken dependency is pulled out of rotation automatically. For the ingestion pipeline specifically, load balancing supplier polling across multiple ingestion worker instances (using consistent hashing by supplier ID) prevents one slow supplier from starving the workers handling everyone else.

10.4 Database Sharding at Scale

For a platform with millions of products, a single PostgreSQL instance may eventually become a bottleneck even with read replicas. At that scale, the inventory table can be sharded — split across multiple database instances, commonly by a hash of the product ID or by supplier ID, so each shard only needs to handle a fraction of total traffic. This is the same partitioning idea from Chapter 4.9, applied directly to storage rather than to processing workers.

10.5 Data Structures Behind the Scenes

A few specific data structures make this design efficient in practice:

11.1 Service Boundaries

Keeping the Feed Ingestion Service, Inventory Service, and Order Service as separate microservices (rather than one giant monolith) means a bad supplier integration cannot crash checkout, and each service can be scaled independently based on its own load pattern — ingestion load depends on supplier count, while order load depends on customer traffic.

11.2 Example Internal API Contract

GET /internal/inventory/{productId}
Response 200:
{
  "productId":          "SKU-88231",
  "availableQuantity":  14,
  "supplierId":         "SUP-4471",
  "lastFeedTimestamp":  "2026-08-03T09:14:02Z",
  "freshnessState":     "FRESH",
  "bufferApplied":      2
}

POST /internal/inventory/reserve
Request:
{
  "productId": "SKU-88231",
  "quantity":  1,
  "cartId":    "CART-99213"
}

Response 200:
{
  "reservationId": "RES-77120",
  "expiresAt":     "2026-08-03T09:24:02Z"
}

Response 409:
{
  "error":             "INSUFFICIENT_STOCK",
  "availableQuantity": 0
}

Notice the response includes freshnessState and bufferApplied — exposing these internally lets other teams (like the frontend team showing “Only 2 left!” messaging) make informed decisions rather than treating stock as a single, opaque number.

11.3 Choosing the Network Protocol for Supplier Connections

Most suppliers expose plain REST over HTTPS, since it is the easiest for a third-party company to implement and document. For very high-frequency, high-volume internal communication (for example, between the Feed Ingestion Service and the Inventory Service, both owned by your own platform), gRPC is often preferred over REST because it uses a compact binary format and supports strongly-typed contracts, reducing both payload size and the chance of a field being misread. For supplier-to-platform webhooks specifically, every incoming request should carry a cryptographic signature (commonly HMAC-SHA256) so the receiving service can verify the payload genuinely came from the claimed supplier and was not tampered with in transit, a topic covered further in Chapter 15.

12.1 Useful Patterns

12.1.1 A Closer Look at the Saga Pattern for Compensations

When an oversell is detected after the fact, simply deleting the order from the database is not safe — money may have already been captured, a confirmation email may have already been sent, and a warehouse may have already begun picking the item. A saga breaks the cancellation into a sequence of explicit, independently reversible steps: reverse the payment capture, notify the customer with a clear explanation and an apology (often paired with a discount code), release any held stock back to the pool, and update the order’s status to “cancelled: inventory unavailable” for reporting and audit purposes. If any individual step in this sequence fails (for example, the payment reversal API is temporarily down), the saga retries that step independently rather than leaving the whole cancellation half-finished, which is far safer than treating the cancellation as a single all-or-nothing database transaction across systems that do not share a single transactional boundary.

12.2 Anti-Patterns to Avoid

The read path (customers viewing products and checking out) must remain fast and available even when the write path (supplier feed ingestion) is under stress. Some concrete techniques:

• Horizontal scaling of ingestion workers: Partition suppliers across many worker instances using consistent hashing, so adding more suppliers means adding more workers, not slowing down existing ones.
• Batching writes: Group many small stock updates into a single database transaction where possible, reducing per-update overhead.
• Backpressure on the queue: If the Inventory Service consumers fall behind, the message queue absorbs the backlog rather than the ingestion service blocking or crashing.
• Read replicas: For very high read traffic on product pages, a read replica of the inventory database (in addition to the Redis cache) can further reduce load on the primary database used for writes.

13.1 Algorithmic Complexity in the Hot Path

The checkout read path — “how much stock is available right now” — is called far more often than the write path that updates stock, so it deserves the most attention to algorithmic efficiency. A Redis hash-map lookup for a single product’s stock is O(1), meaning the time it takes does not grow as your catalog grows from 10,000 to 10,000,000 products. Contrast this with a naive design that scans a list of all products looking for a matching ID, which would be O(n) and would visibly slow down as the catalog grows — exactly the kind of design mistake that looks fine in a small demo and quietly breaks in production at scale.

13.2 Capacity Planning Example

Suppose your platform expects 5,000 product page views per second at peak, and each view requires one stock lookup. A single Redis node can typically handle well over 100,000 simple reads per second, so one well-provisioned cache node comfortably absorbs this load with significant headroom, while the PostgreSQL database behind it is shielded from ever seeing that traffic directly, since cache hits never reach the database at all. This is a concrete illustration of why the cache-aside pattern in Chapter 10.2 exists — it is not just a nice-to-have, it is the difference between the database needing to handle 5,000 queries per second versus a small fraction of that for cache misses and writes only.

• Multi-AZ deployment: Run the Inventory Service, database, and cache across multiple availability zones so a single data center failure does not take down the whole pipeline.
• Database replication: A primary-replica PostgreSQL setup with automatic failover ensures writes can continue (via a new primary) if the current primary fails.
• Queue durability: Kafka’s replicated partitions mean an ingestion event is not lost even if one broker goes down before the Inventory Service consumes it.
• Graceful degradation: If Redis is unavailable, the system should fall back to reading directly from PostgreSQL (slower, but still correct) rather than failing entirely.
• Per-supplier isolation: One supplier’s total outage should never be able to take down the ingestion pipeline for any other supplier — this is the bulkhead pattern applied at the infrastructure level.

14.1 Disaster Recovery and Backups

Beyond day-to-day availability, the system needs a plan for catastrophic scenarios — an entire region going offline, or accidental data corruption. Automated daily snapshots of the inventory database, combined with point-in-time recovery (the ability to restore the database to its exact state at, say, 3:42 PM yesterday, replaying transaction logs up to that moment), let the team recover from both infrastructure failures and human error, such as a bad deployment that corrupts stock data. A documented recovery time objective (how long recovery should take) and recovery point objective (how much data loss, in time, is acceptable) should be agreed with the business ahead of time, not decided during an actual incident.

14.2 Failure Recovery for the Message Queue

If the Inventory Service consumer crashes mid-processing, Kafka’s consumer offset mechanism ensures that unprocessed messages are not lost — the offset (a marker of “how far through the queue this consumer has read”) is only committed after a message is successfully processed, so a crashed consumer resumes exactly where it left off when it restarts, rather than skipping messages or needing manual intervention.

• Authenticated supplier connections: Every supplier integration uses API keys or OAuth tokens, rotated regularly, never hardcoded in source code.
• Payload validation: Never trust incoming feed data blindly — validate types, ranges, and required fields before it touches the database, protecting against both malicious and accidental bad data.
• Rate limiting inbound webhooks: Prevents a misbehaving or compromised supplier system from flooding your ingestion endpoint.
• Least privilege for service accounts: The Feed Ingestion Service should only have write access to the inventory tables it needs, not broad database access.
• Audit logging: Every stock change should be traceable to the exact feed event or reconciliation job that caused it, which is essential both for security investigations and for resolving customer disputes.

15.1 Verifying Webhook Authenticity

Because webhooks are inbound calls initiated by the supplier’s system, your API endpoint must verify that a request genuinely came from that supplier and was not forged or replayed by an attacker. The standard approach is for the supplier to sign each payload with a shared secret key using HMAC-SHA256, sending the resulting signature in a request header. Your service recomputes the signature independently and rejects the request if the two do not match exactly.

public boolean isValidSignature(String payload, String receivedSignature, String sharedSecret) {
    Mac hmac = Mac.getInstance("HmacSHA256");
    hmac.init(new SecretKeySpec(sharedSecret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
    byte[] computed = hmac.doFinal(payload.getBytes(StandardCharsets.UTF_8));
    String computedHex = HexFormat.of().formatHex(computed);
    return MessageDigest.isEqual(
        computedHex.getBytes(StandardCharsets.UTF_8),
        receivedSignature.getBytes(StandardCharsets.UTF_8)
    );
}

Using MessageDigest.isEqual rather than a plain string comparison matters here — it performs a constant-time comparison, which prevents an attacker from guessing the correct signature one character at a time by measuring how long a mismatched comparison takes, a technique known as a timing attack.

16.1 Key Metrics to Track Per Supplier

16.2 Alerting Thresholds

Alerts should escalate in severity: a warning when a feed is aging past its normal window, a high-priority page when a circuit breaker opens for a high-volume supplier, and an immediate page if any confirmed overselling is detected by reconciliation, since that requires fast customer-facing remediation (refund, apology, or expedited restock).

This system fits naturally into a containerised, cloud-native deployment:

• Kubernetes to run the Feed Ingestion, Inventory, Order, and Reconciliation services as independently scalable deployments, each with its own resource limits and horizontal pod autoscaling rules.
• Managed Kafka (such as Amazon MSK or Confluent Cloud) to avoid operating the message broker cluster by hand.
• Managed PostgreSQL (such as Amazon RDS or Cloud SQL) with automated backups, point-in-time recovery, and multi-AZ failover.
• Managed Redis (such as Amazon ElastiCache) for the caching layer.
• Blue-green or canary deployments for the Inventory Service specifically, since bugs here have direct revenue impact — a canary release lets a new version handle a small percentage of traffic before a full rollout.

17.1 Infrastructure as Code

All of the infrastructure described in this tutorial — the Kubernetes deployments, the Kafka topics, the database instances, the cache clusters — should be defined in code (using a tool such as Terraform) rather than clicked together manually in a cloud console. This makes environments reproducible, lets changes go through code review like any other change, and makes it possible to quickly stand up an identical staging environment for the kind of chaos-engineering failure testing recommended in Chapter 19.1.

17.2 Cost Optimisation

Ingestion workers for low-traffic suppliers do not need to run at the same scale as those handling high-volume suppliers. Autoscaling ingestion worker pools based on actual queue depth and per-supplier call volume, rather than running a fixed large fleet at all times, keeps compute costs proportional to actual load. Similarly, the reconciliation job should prioritise the smaller set of high-risk products identified in Chapter 21 (the FAQ) rather than re-verifying the entire catalog on every run, since supplier API calls are often rate-limited or metered, and unnecessary calls both cost money and consume a shared rate-limit budget that healthy, time-critical checks need.

No architecture is free of trade-offs, and it is worth being explicit about what this design gives up in exchange for what it protects. Every choice below intentionally accepts a small, bounded, well-understood risk in exchange for a much larger benefit in speed, resilience, or simplicity. Being able to explain these trade-offs clearly — not just list the components — is usually what separates a strong system design answer from a merely correct one.

19.1 Best Practices

• Size the safety buffer based on real, per-supplier reliability data, not a single fixed number for every supplier.
• Make every supplier integration independently circuit-broken so one bad partner cannot degrade the whole platform.
• Treat “no update received” the same seriousness as “update received with an error” — silence is itself a failure signal.
• Always log the exact source and timestamp behind every stock number, for support and audit purposes.
• Test failure scenarios deliberately (chaos engineering) — simulate a supplier going fully offline in staging and confirm the buffer and circuit breaker behave as designed before it happens for real.

19.2 Common Mistakes

• Treating the supplier feed as always correct and never validating incoming numbers.
• Applying the fix only at the display layer (hiding stock on the webpage) while still letting the backend accept orders beyond real availability.
• Forgetting to expire stock reservations, silently locking up inventory.
• Not distinguishing between “supplier says zero stock” (a real answer) and “supplier didn’t respond” (an unknown, which should never be treated the same as zero or as unlimited).
• Building monitoring only for uptime of the pipeline itself, and forgetting to monitor the actual business outcome — oversell incidents.
• Hardcoding a single global timeout and retry policy for every supplier, instead of tuning per-supplier settings based on their actual observed latency and reliability characteristics.
• Skipping load testing of the ingestion pipeline itself — a supplier recovering from an outage often sends a large backlog of updates all at once, and if the pipeline was never tested against that kind of burst, it can fall over at exactly the moment it is most needed.

19.3 A Short Checklist Before Going Live

Before launching a supplier integration into production, it helps to walk through a short, concrete checklist rather than relying on memory:

The patterns above aren’t theoretical: they show up in the real marketplace and drop-shipping platforms that keep millions of customers from ever seeing an oversell.

The most common questions that arise when engineers first approach this design, answered directly and without hedging.

Handling supplier inventory feed delays and failures sits at the intersection of e-commerce and distributed systems — let’s condense everything into the ideas worth carrying forward.