Marketplaces started simple. Amazon in 1995 sold books — one item type, one fulfillment path: pick it off a shelf, box it, ship it. eBay, Etsy, and early Shopify stores followed the same mental model: an order is a physical thing moving from a warehouse to a doorstep. That world was elegant precisely because it had exactly one shape of “delivery” to reason about.

Then digital goods entered the same shopping carts. Steam started selling downloadable games in 2003. The App Store arrived in 2008. Software license marketplaces, e-gift cards, streaming subscriptions, PDF templates, and SaaS seat licenses all wanted to sit in the same “Add to Cart” button as a T-shirt or a phone case. Customers didn’t want two separate checkouts — they wanted to buy a laptop and its extended warranty PDF in one transaction.

This forced marketplace architects to solve a problem that didn’t exist in 1995: one order, two fundamentally different fulfillment realities. A digital item can be “delivered” the instant payment clears — no truck, no warehouse, no address required. A physical item needs inventory reservation, picking, packing, carrier hand-off, and days of transit. Both must be represented, priced, taxed, refunded, and tracked inside the same order record.

Today, this pattern is everywhere: Amazon sells physical goods alongside Kindle e-books and digital gift cards in one cart. Best Buy sells laptops alongside downloadable software and geek-squad service plans. G2A and Amazon both sell software license keys next to physical peripherals. This tutorial designs that system from the ground up.

1.1 Why this is a system design problem, not just UI

It’s worth pausing on why this problem is genuinely a system design problem and not just a UI problem. Early attempts by many companies to support digital goods were literally two separate systems glued together at the storefront: a “physical store” and a “digital store” with two different checkout flows, two different order histories, and two different customer support playbooks. Customers hated it — they had to decide up front which “kind” of shopping they were doing. Engineering teams hated it too, because every new feature (coupons, gift cards, loyalty points, subscriptions) had to be built twice.

The shift to a unified checkout with divergent backend fulfillment is the same architectural lesson that shows up again and again in distributed systems: keep the customer-facing contract simple and consistent, and push the complexity of “how” into backend services that can evolve, scale, and fail independently of each other. This is precisely the same philosophy behind CQRS (Command Query Responsibility Segregation), event-driven microservices, and the Saga pattern — all of which we’ll use in this design.

1.2 The multi-vendor multiplier

Another historical driver worth noting: as marketplaces became multi-vendor platforms (think Etsy, Amazon Marketplace, eBay), a single cart could contain items from many different sellers, some shipping physical goods from their own warehouses and others delivering digital downloads instantly. This multiplied the fulfillment-diversity problem — the system now had to coordinate not just two fulfillment types, but potentially dozens of independent fulfillment parties, each with their own SLAs, within one order.

Why is this hard? Because the two fulfillment types pull the system in opposite directions on almost every axis. A design that’s optimal for one is usually poorly matched for the other — and yet the customer’s checkout button has to serve both at once.

The business problem: customers expect one checkout, one payment, one order confirmation email — but internally, the system must fan out into two (or more) independent fulfillment pipelines that succeed or fail independently, without ever double-charging the customer or silently dropping an item.

2.1 The business-metrics dimension nobody mentions

There’s also a business-metrics dimension to this problem that engineers are often not told about until it bites them. Product and finance teams typically want to measure “time to fulfillment” as a KPI. If digital and physical items are lumped into one order-level timestamp, that metric becomes meaningless — averaging a 3-second digital delivery with a 3-day physical delivery produces a number that describes neither reality. The architecture must expose per-fulfillment-type metrics natively, not as an afterthought bolted onto analytics.

2.2 Support tooling depends on this distinction

Similarly, customer support tooling depends heavily on this distinction. A support agent looking at “Order #48213” needs to instantly see: which parts of this order are done, which parts are in flight, and which parts (if any) failed — without needing to understand the internals of the Key Vault or the Warehouse Management System. This means the data model has to be designed for support-and-finance readability from day one, not just for the happy-path checkout flow.

2.3 Regulatory and compliance angle

Finally, there’s a regulatory and compliance angle. Digital goods and physical goods are frequently taxed differently (VAT treatment of digital services in the EU, for example, differs meaningfully from physical goods VAT), have different consumer-protection return windows in many jurisdictions, and have different export-control considerations (a software license might be subject to export restrictions that a physical accessory is not). A system that treats “an order” as one undifferentiated blob cannot correctly apply any of these rules.

Before drawing any architecture, we write down what the system must do and how well it must do it. These requirements are the contract every later decision has to satisfy.

3.1 Functional Requirements

• A cart can contain any mix of digital and physical line items.
• Checkout collects a shipping address only if at least one physical item exists; digital-only carts skip address entirely.
• Payment is captured once, for the full cart total, regardless of item mix.
• Digital items are delivered (license key / download link emailed and shown on order page) within seconds of payment success.
• Physical items are reserved in inventory, routed to a warehouse, picked, packed, and handed to a carrier, with tracking numbers surfaced to the customer.
• Order detail page shows independent status per line item (“License delivered” vs “Shipped, arriving Thursday”).
• Refunds/cancellations are handled per line item, respecting each fulfillment type’s rules (key revocation vs. return-to-warehouse).
• Partial order failures are possible: digital item delivers instantly even if a physical item’s warehouse is temporarily out of stock, and vice versa — payment isn’t rolled back for the whole cart because one leg is delayed.

3.2 Non-Functional Requirements

• Consistency: No customer is ever charged without receiving either the goods or an automatic refund. No license key is ever issued twice for one paid order (idempotency).
• Latency: Checkout API p99 under 300ms; digital delivery p99 under 3 seconds after payment capture.
• Availability: 99.99% for checkout and payment path; fulfillment pipelines can queue and retry, so they can tolerate brief downstream outages.
• Scalability: Must survive flash-sale spikes (e.g., 50x normal traffic during a holiday sale) without losing orders.
• Auditability: Every state transition (payment captured, key issued, item shipped) must be logged immutably for finance and support.

These requirements together point to a single architectural conclusion that shapes everything in the rest of this tutorial: the system needs exactly one strongly-consistent decision point (has this customer paid, and for what) and multiple independently-scaling, independently-failing fulfillment pipelines downstream of that decision point. Every component described from here on exists to implement one side or the other of that split.

The core architectural decision is to treat the order as a single financial/customer-facing entity, but let it fan out into independent fulfillment pipelines per line item, coordinated through an event-driven, microservices architecture with a Saga-based orchestrator sitting between payment and fulfillment.

4.1 The one decision point that shapes everything

Notice a specific architectural shape: everything above the event bus is on the strongly-consistent, latency-sensitive, must-not-fail path (the customer’s money and the order record). Everything below the event bus is on the asynchronous, independently-retryable, eventually-consistent path (the actual delivery of value). That one horizontal line through the diagram is the single most important architectural decision in this design, and every trade-off in the rest of this tutorial derives from it.

Each box in the architecture diagram represents a service with a clear responsibility, a private database or store, and a narrow API. Here is what each one does and why it exists as its own component.

5.1 API Gateway

The single entry point for all client traffic. Responsibilities: TLS termination, authentication (JWT validation), rate limiting per user/IP, request routing to the correct backend service, and request/response transformation. In a hybrid marketplace, the gateway also tags each incoming checkout request with a correlation ID that threads through digital and physical fulfillment pipelines for tracing.

5.2 Load Balancer

Distributes traffic across horizontally scaled service instances using algorithms like round-robin, least-connections, or weighted routing. Performs health checks and removes unhealthy instances from rotation. Critical during flash sales, where the Checkout Service and Payment Service need to scale out fastest.

5.3 Cart Service

Holds the mutable, pre-checkout state of a customer’s cart — a mix of digital and physical SKUs, quantities, and applied promotions. Backed by Redis for low-latency reads/writes, with periodic snapshot to a durable store so an abandoned cart survives a Redis restart.

5.4 Checkout Service

Validates the cart (price freshness, stock availability check — not reservation yet), determines whether a shipping address is required (only if at least one physical SKU is present), computes tax and shipping cost, and produces a “checkout intent” that is handed to the Order Orchestrator.

5.5 Order Orchestrator (Saga Coordinator)

The brain of the system. Implements the order-placement Saga: reserve inventory → capture payment → create order record → publish fulfillment events. Owns compensating actions if any step fails (e.g., release inventory reservation if payment fails).

5.6 Payment Service

Abstracts the external payment gateway (Stripe, Adyen, Braintree). Handles idempotent payment capture (using an idempotency key derived from the order ID so retries never double-charge), and issues refunds per line item later.

5.7 Inventory Service

Manages physical stock counts per warehouse/SKU with reservation semantics (soft-hold during checkout, hard-decrement on payment success). For digital SKUs backed by a finite pool of license keys, this service (or a sibling Key Vault) tracks key availability the same way — as a reservable resource.

5.8 Order Service

The system of record for the order itself: line items, prices, payment reference, and the aggregate/derived order status. Writes to a durable, sharded SQL database and publishes an “OrderCreated” event to the event bus.

5.9 Event Bus (Kafka)

Decouples the Order Service from the two fulfillment pipelines. Digital Fulfillment Service and Physical Fulfillment Service each subscribe independently and process at their own pace — this is what allows one leg to succeed instantly while the other takes days, without either blocking the other.

5.10 Digital Fulfillment Service

Consumes order events, pulls a license key from the Key Vault (or generates one), marks it “issued” atomically, and triggers delivery (email + in-account download link).

5.11 Physical Fulfillment Service

Consumes order events, routes the physical line items to the nearest warehouse with stock via the WMS, coordinates pick/pack, and hands off to a shipping carrier via the Carrier Gateway, receiving tracking updates via webhook.

5.12 Notification Service

Sends order confirmation, digital delivery, shipment, and delivery notifications across email/SMS/push, templated per fulfillment type. It subscribes to the same event bus as the fulfillment services rather than being called synchronously by them — this means adding a new notification channel (e.g., WhatsApp updates) never requires touching Digital or Physical Fulfillment code.

5.13 License Key Vault

A dedicated, tightly access-controlled store for pre-generated or on-demand-generated license keys. Kept as a distinct component (rather than folded into general Inventory) because its access patterns, security requirements (encryption at rest, audit logging of every read), and scaling profile (write-once, read-once-then-locked) are meaningfully different from physical stock counts.

5.14 Warehouse Management System (WMS)

Often a licensed third-party system (e.g., Manhattan Associates, Fluent Commerce) or a large in-house system, responsible for the physical realities of a warehouse: bin locations, pick paths, packing station assignment, and label printing. The Physical Fulfillment Service treats the WMS as an external system with its own API and its own eventual-consistency lag.

5.15 Shipping Carrier Gateway

Abstracts multiple carrier APIs (FedEx, UPS, DHL, regional couriers) behind one internal interface, so the rest of the system doesn’t need to know which carrier fulfilled a given shipment. Handles label generation, rate shopping (picking the cheapest/fastest carrier for a given route), and normalizes each carrier’s very different webhook/tracking event formats into one internal tracking-event schema.

5.16 Search & Catalog Index

An Elasticsearch (or OpenSearch) index that powers product search and category browsing. It is populated asynchronously from the Catalog Service via events, so search results can lag catalog writes by a few seconds — an acceptable trade-off for read-heavy, eventually-consistent search versus a synchronous, strongly-consistent path that would slow down every catalog update.

5.17 Analytics & Monitoring Pipeline

Consumes the same order and fulfillment events (via the event bus) into a data warehouse or stream-processing system (e.g., Kafka Streams, Flink, or a managed pipeline into BigQuery/Snowflake/Redshift) for business intelligence: conversion rates, average fulfillment time per type, revenue by fulfillment type, and fraud pattern analysis.

Below is the sequence of events for a cart containing one digital SKU (a software license) and one physical SKU (a laptop sleeve). Watch how a single checkout call splits into two independent delivery timelines the moment the OrderCreated event is published.

Notice that the digital and physical fulfillment steps happen independently after the event is published — the Order Service does not wait for either to finish before returning a successful checkout response to the customer.

6.1 Order and Line-Item State Machine

The order object has its own status, but that status is a derived summary computed from the individual states of its line items. Each fulfillment type contributes its own set of transitions.

The checkout API must handle three cart shapes with one contract: digital-only, physical-only, and mixed. The key design choice is a single CheckoutRequest where shipping address is optional and validated conditionally.

public class CheckoutRequest {
    private String cartId;
    private String customerId;
    private ShippingAddress shippingAddress; // nullable if no physical items
    private String paymentMethodId;
    private String idempotencyKey;
}

public class CheckoutValidator {

    public void validate(CheckoutRequest request, Cart cart) {
        boolean hasPhysicalItem = cart.getLineItems().stream()
                .anyMatch(item -> item.getFulfillmentType() == FulfillmentType.PHYSICAL);

        if (hasPhysicalItem && request.getShippingAddress() == null) {
            throw new CheckoutValidationException(
                "Shipping address is required: cart contains at least one physical item");
        }

        if (!hasPhysicalItem && request.getShippingAddress() != null) {
            // Not an error, but we simply ignore the address for tax purposes
            // and use billing address or IP-based locale instead.
        }

        if (request.getIdempotencyKey() == null || request.getIdempotencyKey().isBlank()) {
            throw new CheckoutValidationException("Idempotency key is required");
        }
    }
}

Tax computation also branches per line item: physical items use ship-to jurisdiction tax rules, digital items often use a separate “digital goods” tax category (many jurisdictions tax software licenses differently from e-books or SaaS). This is computed per line item and summed, not computed once for the whole cart.

public class TaxCalculator {

    public Money calculateTax(LineItem item, Address billingAddress, Address shippingAddress) {
        if (item.getFulfillmentType() == FulfillmentType.DIGITAL) {
            Address taxJurisdiction = billingAddress; // digital goods taxed by buyer's billing location
            return digitalGoodsTaxRules.compute(item, taxJurisdiction);
        } else {
            Address taxJurisdiction = shippingAddress; // physical goods taxed by ship-to location
            return physicalGoodsTaxRules.compute(item, taxJurisdiction);
        }
    }
}

7.1 Promotions and Discounts Across a Mixed Cart

A coupon like “$10 off your order” or “free shipping over $50” must be evaluated against a cart that mixes SKUs with very different cost structures. The Checkout Service resolves this by applying promotion rules in a defined order: cart-level percentage or flat discounts are first prorated across all line items by price (so refunding one line item later correctly reverses only its share of the discount), and then fulfillment-specific promotions (like “free shipping over $50”) are evaluated only against the subtotal of physical line items, since a “free shipping” promotion is meaningless for a digital-only line.

public class PromotionEngine {

    public void applyCartLevelDiscount(Cart cart, Discount discount) {
        Money subtotal = cart.getSubtotal();
        for (LineItem item : cart.getLineItems()) {
            // Proration: each line item absorbs a share of the discount
            // proportional to its price, so partial refunds stay accurate.
            BigDecimal share = item.getLinePrice()
                    .divide(subtotal.getAmount(), 6, RoundingMode.HALF_UP);
            Money itemDiscount = discount.getAmount().multiply(share);
            item.applyDiscount(itemDiscount);
        }
    }

    public void applyFreeShippingIfEligible(Cart cart, Money threshold) {
        Money physicalSubtotal = cart.getLineItems().stream()
                .filter(i -> i.getFulfillmentType() == FulfillmentType.PHYSICAL)
                .map(LineItem::getLinePrice)
                .reduce(Money.zero(), Money::add);

        if (physicalSubtotal.isGreaterThanOrEqual(threshold)) {
            cart.setShippingFee(Money.zero());
        }
    }
}

This proration approach matters most at refund time: if a customer returns only the physical item from a discounted mixed cart, the refund engine needs to know exactly how much of the original discount was attributable to that line item, not the whole cart — otherwise the business either over-refunds (losing money) or under-refunds (a support escalation and a frustrated customer).

A single ACID transaction across Inventory, Payment, and Order services is not realistic once these are separate microservices with separate databases. The Saga pattern breaks the order-placement flow into a sequence of local transactions, each with a defined compensating action if a later step fails.

8.1 Orchestration vs. Choreography

This example uses an orchestration-based Saga (a central coordinator explicitly calls each step and its compensation) rather than a choreography-based Saga (services react to each other’s events with no central brain). Orchestration is preferred here because the compensation logic for a mixed cart is genuinely complex — it’s easier to reason about, test, and observe with one coordinator than with a web of implicit event reactions.

public class OrderPlacementSaga {

    private final InventoryClient inventoryClient;
    private final PaymentClient paymentClient;
    private final OrderClient orderClient;
    private final EventPublisher eventPublisher;

    public OrderResult execute(CheckoutIntent intent) {
        List<Runnable> compensations = new ArrayList<>();
        try {
            InventoryReservation reservation = inventoryClient.reserve(intent.getPhysicalItems());
            compensations.add(() -> inventoryClient.release(reservation));

            KeyReservation keyReservation = inventoryClient.reserveKeys(intent.getDigitalItems());
            compensations.add(() -> inventoryClient.releaseKeys(keyReservation));

            PaymentResult payment = paymentClient.capture(
                intent.getTotal(), intent.getPaymentMethodId(), intent.getIdempotencyKey());
            compensations.add(() -> paymentClient.refund(payment.getPaymentId()));

            Order order = orderClient.createOrder(intent, reservation, keyReservation, payment);
            compensations.add(() -> orderClient.cancelOrder(order.getId()));

            eventPublisher.publish(new OrderCreatedEvent(order));

            return OrderResult.success(order);

        } catch (SagaStepException ex) {
            // Run compensations in reverse order
            Collections.reverse(compensations);
            compensations.forEach(Runnable::run);
            return OrderResult.failure(ex.getMessage());
        }
    }
}

Digital delivery has one job: turn a paid order line into an emailed/available asset in seconds, exactly once. The core challenge is exactly-once key issuance — a license key must never be handed to two different customers, and a retried event must never issue a second key to the same customer for the same order line.

@Service
public class DigitalFulfillmentService {

    private final LicenseKeyRepository keyRepository;
    private final DeliveryLogRepository deliveryLogRepository;
    private final NotificationClient notificationClient;

    @Transactional
    public void handle(OrderCreatedEvent event) {
        for (LineItem item : event.getDigitalLineItems()) {

            // Idempotency guard: has this exact line item already been fulfilled?
            if (deliveryLogRepository.existsByOrderLineId(item.getLineId())) {
                return; // safe to ignore duplicate event delivery
            }

            LicenseKey key = keyRepository.claimAvailableKey(item.getSkuId());
            if (key == null) {
                throw new OutOfKeysException(item.getSkuId());
            }

            key.markIssued(event.getOrderId(), item.getLineId());
            keyRepository.save(key);

            deliveryLogRepository.save(new DeliveryLog(item.getLineId(), key.getId()));

            notificationClient.sendLicenseKeyEmail(
                event.getCustomerEmail(), key.getKeyValue(), item.getSkuName());
        }
    }
}

The claimAvailableKey call must use a row-level lock (SELECT ... FOR UPDATE SKIP LOCKED) so that concurrent consumers processing the event bus in parallel never claim the same key twice.

SELECT key_id, key_value FROM license_keys
WHERE sku_id = :skuId AND status = 'AVAILABLE'
ORDER BY key_id
LIMIT 1
FOR UPDATE SKIP LOCKED;

9.1 Key Pool vs. On-Demand Generation

There are two common strategies for where license keys come from, and the right choice depends on the software vendor’s own licensing system. A pre-generated pool (as modeled in the schema above) works well when keys are supplied in bulk by a publisher ahead of time — the Digital Fulfillment Service simply claims the next available row. An on-demand generation strategy instead calls an external licensing API synchronously at fulfillment time to mint a brand-new key tied to the specific order and customer; this avoids ever holding unused inventory but introduces a dependency on a third-party API’s availability and latency directly in the critical delivery path, which is exactly why that external call should go through a circuit breaker and a retry-with-backoff policy, with a clear fallback (queue and notify support) if the licensing API is down.

9.2 Delivery Confirmation and Access

Beyond email, digital items should always be retrievable from the customer’s order/account page via a signed, time-limited download URL — email delivery can fail (spam filters), so the account page is the source of truth, and email is a convenience notification.

Physical fulfillment is a multi-stage pipeline: reserve stock → route to warehouse → pick/pack → hand off to carrier → track to delivery. Unlike digital delivery, this pipeline has real-world latency and failure modes (a warehouse can run out of stock after reservation due to a damaged item, a carrier can lose a package).

@Service
public class PhysicalFulfillmentService {

    private final WarehouseRoutingService routingService;
    private final WmsClient wmsClient;
    private final CarrierGatewayClient carrierClient;
    private final OrderStatusClient orderStatusClient;

    public void handle(OrderCreatedEvent event) {
        for (LineItem item : event.getPhysicalLineItems()) {
            Warehouse warehouse = routingService.selectWarehouse(item, event.getShippingAddress());

            PickTask pickTask = wmsClient.createPickTask(warehouse.getId(), item);
            orderStatusClient.updateLineStatus(item.getLineId(), LineStatus.PICKING);

            PackResult packResult = wmsClient.waitForPack(pickTask.getId());
            ShippingLabel label = carrierClient.createLabel(packResult, event.getShippingAddress());

            orderStatusClient.updateLineStatus(item.getLineId(), LineStatus.SHIPPED,
                label.getTrackingNumber(), label.getCarrier());
        }
    }
}

10.1 Handling Split Shipments

A single order can require multiple physical shipments if items are stocked in different warehouses. The line-item-level status model already supports this naturally — each physical line item tracks its own tracking number and status, so a “2 of 3 items shipped” view falls out of the data model for free rather than requiring special-case logic.

10.2 Returns and Reverse Logistics

Physical returns introduce a second, mirrored pipeline: the customer requests a return, receives a prepaid return label (generated through the same Carrier Gateway used for outbound shipments), ships the item back, and the warehouse must receive and inspect it before the refund is authorized. This is modeled as its own state machine attached to the physical line item, distinct from the forward-fulfillment state machine, because a returned item can be accepted, rejected (damaged, wrong item), or partially refunded (restocking fee) — outcomes that have no equivalent on the way out.

public enum ReturnStatus {
    REQUESTED,
    LABEL_ISSUED,
    IN_TRANSIT_TO_WAREHOUSE,
    RECEIVED_AT_WAREHOUSE,
    INSPECTED_ACCEPTED,
    INSPECTED_REJECTED,
    REFUND_ISSUED
}

@Service
public class ReturnService {

    public void receiveAtWarehouse(String returnId, InspectionResult result) {
        Return ret = returnRepository.findById(returnId);
        if (result.isAcceptable()) {
            ret.transitionTo(ReturnStatus.INSPECTED_ACCEPTED);
            refundClient.issueRefund(ret.getOrderLineId(), ret.getRefundAmount());
            ret.transitionTo(ReturnStatus.REFUND_ISSUED);
        } else {
            ret.transitionTo(ReturnStatus.INSPECTED_REJECTED);
            notificationClient.notifyReturnRejected(ret);
        }
        returnRepository.save(ret);
    }
}

Digital line items have no equivalent physical return flow — instead, a “refund” for a digital item is really a revocation: the license key’s status flips to REVOKED, any active entitlement is disabled, and the refund is issued immediately rather than waiting for a physical item to travel back through a carrier network. This asymmetry — instant digital revocation versus multi-day physical inspection — is one of the clearest illustrations of why refund logic must branch by fulfillment type rather than being written once generically.

The schema must express that an order has many line items, each line item has exactly one fulfillment type, and fulfillment detail lives in type-specific tables rather than cramming digital and physical fields into one wide table.

CREATE TABLE orders (
    order_id          BIGINT PRIMARY KEY,
    customer_id       BIGINT NOT NULL,
    status            VARCHAR(30) NOT NULL,   -- derived aggregate status
    total_amount      DECIMAL(12,2) NOT NULL,
    currency          CHAR(3) NOT NULL,
    payment_id        VARCHAR(64) NOT NULL,
    shipping_address_id BIGINT NULL,          -- nullable: digital-only orders have none
    created_at        TIMESTAMP NOT NULL,
    updated_at        TIMESTAMP NOT NULL
);

CREATE TABLE order_line_items (
    line_id           BIGINT PRIMARY KEY,
    order_id          BIGINT NOT NULL REFERENCES orders(order_id),
    sku_id            BIGINT NOT NULL,
    fulfillment_type  VARCHAR(10) NOT NULL,   -- 'DIGITAL' or 'PHYSICAL'
    quantity          INT NOT NULL,
    unit_price        DECIMAL(12,2) NOT NULL,
    tax_amount        DECIMAL(12,2) NOT NULL,
    line_status       VARCHAR(30) NOT NULL,
    UNIQUE (order_id, sku_id)
);

CREATE TABLE digital_fulfillments (
    line_id           BIGINT PRIMARY KEY REFERENCES order_line_items(line_id),
    license_key_id    BIGINT NOT NULL,
    delivered_at      TIMESTAMP NULL,
    download_url_hash VARCHAR(128) NULL
);

CREATE TABLE physical_fulfillments (
    line_id           BIGINT PRIMARY KEY REFERENCES order_line_items(line_id),
    warehouse_id      BIGINT NOT NULL,
    tracking_number   VARCHAR(64) NULL,
    carrier           VARCHAR(30) NULL,
    shipped_at        TIMESTAMP NULL,
    delivered_at      TIMESTAMP NULL
);

CREATE TABLE license_keys (
    key_id    BIGINT PRIMARY KEY,
    sku_id    BIGINT NOT NULL,
    key_value VARCHAR(128) NOT NULL,
    status    VARCHAR(20) NOT NULL,  -- AVAILABLE, ISSUED, REVOKED
    issued_order_id BIGINT NULL,
    INDEX idx_sku_status (sku_id, status)
);

This separation means the Order Service query for “show me this order” is a join across a small, consistent set of tables, while each fulfillment service owns and evolves its own detail table independently — a classic microservice data-ownership boundary expressed at the schema level.

Each service in this architecture exposes a narrow, purpose-built API, and services never reach into each other’s databases directly — all cross-service communication happens through synchronous REST/gRPC calls (for request/response needs) or asynchronous events on the bus (for fire-and-forget fan-out). This is the classic microservices boundary rule: a service’s database is private; its API is public.

Note that POST /v1/orders is marked internal — the client never calls it directly. The Checkout Service is the only caller, and it’s the boundary where the public-facing contract (simple, cart-shaped) is translated into the internal contract (Saga-shaped, with reservation and compensation semantics). This separation lets the internal orchestration logic evolve freely (e.g., adding a third fulfillment type) without ever changing the public checkout API that mobile and web clients depend on.

// Example order response - notice line items carry independent status
{
  "orderId": "ord_9F3K2",
  "status": "PARTIALLY_FULFILLED",
  "lineItems": [
    {
      "lineId": "li_001",
      "sku": "SOFTWARE-LICENSE-PRO",
      "fulfillmentType": "DIGITAL",
      "status": "DELIVERED",
      "deliveredAt": "2026-08-03T10:02:11Z",
      "downloadUrl": "https://cdn.example.com/dl/abc123?exp=1735900000"
    },
    {
      "lineId": "li_002",
      "sku": "LAPTOP-SLEEVE-15IN",
      "fulfillmentType": "PHYSICAL",
      "status": "SHIPPED",
      "trackingNumber": "1Z999AA10123456784",
      "carrier": "UPS",
      "estimatedDelivery": "2026-08-06"
    }
  ]
}

Internally, gRPC is often preferred over REST between the Order Orchestrator and services like Inventory and Payment, since these calls are high-frequency, latency-sensitive, and internal-only — gRPC’s binary protocol and strongly-typed contracts (via protobuf) reduce serialization overhead and catch contract mismatches at compile time rather than at runtime.

Different data in this system has very different freshness requirements, so a single caching policy would either be wastefully conservative or dangerously stale somewhere. The system uses tiered caching, polyglot persistence, and multi-layer load balancing to match each data class to the right storage and delivery technology.

13.1 Caching Strategy

• Catalog data (product descriptions, prices, images): cached aggressively at the CDN edge and in Redis with a TTL of minutes, since catalog changes are infrequent and slightly-stale product pages are low-risk.
• Cart state: stored in Redis as the primary store (not just a cache), since cart data is inherently ephemeral and doesn’t need the durability guarantees of a relational database — a Redis cluster with replication is sufficient, backed by periodic snapshots for abandoned-cart recovery.
• Inventory/key-availability counts: cached only as a fast-path admission check (e.g., “is this SKU likely in stock?”) with a very short TTL or event-driven invalidation — the actual reservation decision always goes to the durable database to avoid overselling from stale cache reads.
• Order status: never cached for writes; reads can be served from a read replica or a short-TTL cache (a few seconds) since customers checking “where’s my order” tolerate slight staleness far better than a customer expecting exactly-once payment or key issuance.

13.2 Database Choices

Order, Payment, and Inventory data need strong consistency and transactional guarantees — a relational database (PostgreSQL or a distributed SQL system like CockroachDB/Spanner for very large scale) is the right fit, sharded by customer ID or order ID for horizontal scale. Catalog and search data, by contrast, is read-heavy and tolerates eventual consistency, making a document store or search index (Elasticsearch) a better fit for that specific access pattern. This is a deliberate example of polyglot persistence: choosing the storage technology per service based on that service’s actual access pattern, rather than forcing one database technology across the whole system.

13.3 Load Balancing in Depth

Load balancing happens at multiple layers in this architecture, each solving a different problem:

No architecture is free — every decision made in the sections above trades one desirable property for another. It’s worth naming these trade-offs explicitly, because interviewers and architecture reviewers both expect a candidate to articulate not just what was chosen, but what was given up, and why that trade was acceptable for this specific problem.

The overarching advantage of this design is that the two fulfillment pipelines fail independently: a warehouse outage never blocks a digital sale, and a Key Vault outage never blocks a physical sale. The overarching disadvantage is operational surface area — instead of one pipeline to monitor, deploy, and reason about, there are now at least two (often more, once returns, notifications, and analytics pipelines are counted), each with its own on-call runbook, its own failure modes, and its own eventual-consistency lag against the order-level view the customer sees.

The hardest scaling moment for this system is a flash sale where thousands of customers try to buy the same limited digital license or the same limited physical SKU within seconds.

• Inventory hotspot mitigation: Shard the inventory/key counter by SKU across multiple database partitions, or use an in-memory atomic counter (Redis DECR) as a fast-path gate before touching the durable database, falling back to the database as the source of truth for the actual reservation record.
• Checkout API horizontal scaling: Stateless Checkout and Order Orchestrator services scale out behind the load balancer; session/cart state lives in Redis, not in-process memory, so any instance can serve any request.
• Payment gateway rate limits: Queue payment capture requests through a bounded worker pool with backpressure, since external payment gateways impose their own rate limits; return “processing” to the client rather than blocking the HTTP request indefinitely.
• Digital delivery throughput: Because digital fulfillment has no physical bottleneck, it can scale near-linearly by adding more Kafka consumer instances in the same consumer group — the main constraint becomes the key-claiming database writes, which is where SKIP LOCKED and sharding matter most.
• Physical fulfillment throttling: Real-world pick/pack throughput is capped by warehouse capacity; the system should surface an honest “processing time” estimate rather than promising instant fulfillment it can’t deliver physically.

15.1 Working Through a Capacity Example

Consider a marketplace running a flash sale where 500,000 customers hit “Buy Now” within a 60-second window for a bundle containing one digital license and one physical accessory. At the API Gateway tier, this is roughly 8,300 requests/second sustained, with realistic bursts several times higher. A few concrete capacity decisions follow directly from this number:

• The Checkout Service should be provisioned (or auto-scaled) to comfortably handle 3-5x the expected peak, since flash-sale traffic is famously bursty rather than smooth, and autoscaling has a cold-start lag of tens of seconds to minutes depending on the platform.
• The license-key claim query (SELECT ... FOR UPDATE SKIP LOCKED) is the single biggest risk of contention at this volume. Sharding the key table by SKU across multiple physical database partitions turns one hot row-lock queue into N independent queues, multiplying effective throughput roughly by the shard count.
• Payment gateway calls are usually the true bottleneck, since most third-party gateways impose their own rate limits (commonly in the low thousands of requests/second per merchant account). The system must queue excess requests rather than reject them outright, returning an honest “your order is processing” state to the client and completing asynchronously — this is a deliberate trade of a few extra seconds of perceived latency for zero lost sales.
• Physical fulfillment naturally absorbs this burst more gracefully than it first appears, because warehouses already operate on a queue (pick tasks) — the WMS integration should simply accept a burst of pick tasks and let its own internal capacity planning smooth the pace, rather than trying to force real-time warehouse throughput to match checkout throughput.

This example illustrates a general principle: not every component in a hybrid fulfillment system needs to scale to the same peak number. Digital delivery must scale to match checkout throughput almost 1:1, because customers expect it instantly. Physical fulfillment does not, because customers already expect it to take time — that expectation gap is exactly what the event-bus decoupling exploits.

Availability is not a single global number for this system — different parts have different SLOs. Checkout must be highly available because a failure there is directly lost revenue. Downstream fulfillment can absorb short outages via queues, retries, and dead-letter handling.

• Multi-AZ deployment for all stateless services (API Gateway, Checkout, Orchestrator, Fulfillment services) so a single availability zone failure doesn’t take down checkout.
• Database replication: Order and Inventory databases use synchronous replication within a region for durability, with async cross-region replicas for disaster recovery.
• Event bus durability: Kafka topics are replicated across brokers (replication factor 3), so an OrderCreated event is never lost even if a broker fails mid-publish.
• Dead-letter queues: If Digital or Physical Fulfillment repeatedly fails to process an event (e.g., Key Vault down), the event moves to a DLQ after N retries with exponential backoff, alerting on-call rather than silently dropping the order.
• Circuit breakers around the external Payment Gateway and Carrier Gateway calls, so a slow third party degrades gracefully (queue and retry) instead of exhausting connection pools system-wide.

Security in a hybrid marketplace has to protect two very different classes of value: bearer-token-like digital assets (license keys, download URLs) and physical shipments to real-world addresses. Each has its own threat model, and the system must defend against both without conflating them.

• License key protection: Keys at rest are encrypted (e.g., AES-256) in the Key Vault; keys in transit to the customer use TLS and short-lived signed URLs rather than embedding raw keys in long-lived emails wherever possible.
• Idempotency keys on checkout and payment capture prevent duplicate charges from client retries or network blips.
• Fraud detection for digital goods: Digital items are attractive to card-testing fraud (small, instantly resellable). Apply velocity checks (e.g., N purchase attempts per card per hour), device fingerprinting, and step-up authentication (3-D Secure) for high-risk digital purchases.
• Fraud detection for physical goods: Address-mismatch scoring (billing vs. shipping), reshipping-mule detection (freight-forwarder address databases), and manual review queues for high-value shipments.
• Least-privilege service accounts: Digital Fulfillment Service can read/write the Key Vault but has no access to the Payment Service’s credentials; Physical Fulfillment Service can call the Carrier Gateway but not the Key Vault.
• PII minimization: Shipping address is stored only in the Order/Shipping domain, encrypted at rest, and purged per data-retention policy after the legally required window.

17.1 Compliance Considerations

Payment data itself should never touch application servers directly — payment card details are tokenized at the client (via the payment gateway’s hosted fields or SDK) so the Payment Service only ever handles a token, keeping the system’s PCI-DSS scope as small as possible. Digital goods delivered internationally may also need export-compliance screening (comparing the customer against restricted-party lists) before key issuance, which the Digital Fulfillment Service should treat as another Saga-style reservation gate, similar to inventory reservation, rather than a bolt-on afterthought.

17.2 Key Rotation and Revocation

Because license keys represent ongoing value (unlike a one-time physical shipment), the Key Vault needs a revocation path independent of the order lifecycle — for example, if a batch of keys is discovered to have been leaked before sale, all unissued keys in that batch must be invalidated instantly, and all issued-but-unused keys flagged for monitoring. This requires the key status field to support states beyond simple “available/issued,” including “revoked” and “flagged,” with the licensing/activation server (if one exists) checking key status on every activation attempt rather than trusting a key permanently once issued.

A hybrid marketplace has to be observable at three levels: business (are we losing sales?), platform (are the services healthy?), and per-order (why did this specific customer have a bad experience?). The metrics below cover all three, with correlation IDs threading a single customer’s journey across every hop.

Every event carries a correlation/trace ID from the original checkout request, propagated through Kafka message headers, so a single order’s full journey (checkout → payment → digital delivery → physical shipment) can be reconstructed in a distributed tracing tool (e.g., Jaeger, Zipkin) for debugging a specific customer complaint.

Each service is packaged as an independently deployable container, orchestrated via Kubernetes, with separate deployments and horizontal pod autoscalers per service so, for example, Digital Fulfillment can scale independently of Physical Fulfillment based on their very different load patterns.

• CI/CD: Each service has its own pipeline; canary or blue-green deployments for the Checkout and Payment services specifically, given their criticality — a bad deploy there directly blocks revenue.
• Infrastructure as Code: Terraform or CloudFormation defines the Kafka cluster, database clusters, and Kubernetes namespaces, so environments (staging, production) stay reproducible.
• Multi-region: For a global marketplace, digital fulfillment can run active-active across regions easily since it has no physical dependency; physical fulfillment is inherently regional (tied to warehouse locations) and routes orders to the correct region’s WMS based on shipping address.
• Cost optimization: Digital Fulfillment workloads are bursty and CPU-light — good fit for serverless/spot-instance autoscaling; Physical Fulfillment integrates with steadier, always-on WMS systems.

The design in this tutorial isn’t inventing new ideas — it’s composing well-known patterns to fit a specific problem shape. Naming them explicitly makes the design easier to communicate in interviews and easier to defend in reviews.

20.1 Patterns Used

• Saga pattern for distributed order placement across services.
• Event-driven architecture / Publish-Subscribe to decouple fulfillment pipelines from the order-creation path.
• Strategy pattern for fulfillment-type-specific logic (a FulfillmentStrategy interface with DigitalFulfillmentStrategy and PhysicalFulfillmentStrategy implementations).
• Outbox pattern to atomically persist an order and its corresponding event within the same local transaction, avoiding dual-write inconsistency between the database and the event bus.
• Idempotent receiver pattern for payment capture and key issuance.

public interface FulfillmentStrategy {
    void fulfill(LineItem item, OrderContext context);
}

public class DigitalFulfillmentStrategy implements FulfillmentStrategy {
    public void fulfill(LineItem item, OrderContext context) {
        // claim key, deliver
    }
}

public class PhysicalFulfillmentStrategy implements FulfillmentStrategy {
    public void fulfill(LineItem item, OrderContext context) {
        // route to warehouse, pack, ship
    }
}

public class FulfillmentDispatcher {
    private final Map<FulfillmentType, FulfillmentStrategy> strategies;

    public void dispatch(LineItem item, OrderContext context) {
        strategies.get(item.getFulfillmentType()).fulfill(item, context);
    }
}

20.2 Anti-patterns to Avoid

• God Order Service: Letting the Order Service directly know how to issue license keys and talk to carrier APIs. This couples unrelated domains and makes independent scaling and deployment impossible.
• Single mutable “status” field with no line-item granularity: Forces awkward hacks like encoding two statuses into one string.
• Synchronous fan-out: Making the checkout HTTP request wait for both digital delivery and physical warehouse routing to complete before responding — this directly couples checkout latency to the slowest possible fulfillment path.
• Distributed transactions (2PC) across microservice databases: Technically possible but operationally fragile and blocking; the Saga pattern with compensations is the industry-preferred alternative at this scale.

These are the small habits that separate a system that survives its first flash sale from one that gets rewritten a year later. Most of them are cheap on day one and painfully expensive to retrofit later.

• Do model fulfillment type as a first-class field on the line item, not inferred from SKU category elsewhere in code.
• Do make every step in the Saga and every event consumer idempotent — assume at-least-once delivery everywhere.
• Do keep the checkout response fast by returning as soon as payment is captured and the order is recorded, then let fulfillment proceed asynchronously with clear status polling/webhooks for the client.
• Don’t assume every order has a shipping address — null-check religiously in downstream code, including tax, notifications, and returns processing.
• Don’t let a stock-out on the physical leg roll back or delay the digital leg — evaluate and compensate each leg independently.
• Don’t forget refund asymmetry: refunding a physical item might require waiting for the return to arrive at the warehouse before releasing funds, while refunding a digital item can require immediate key revocation to prevent further use.
• Do instrument fulfillment-type-specific SLA dashboards from the very first release — retrofitting “time to digital delivery” versus “time to physical shipment” as separate metrics after they’ve been conflated in a single field is a much larger migration than building them separately up front.
• Do design the notification templates per fulfillment type from the start, even if the initial copy is nearly identical — a “your order shipped” template and a “your license is ready” template will diverge quickly as the product grows (tracking links vs. download buttons, different support FAQs linked, different escalation paths).
• Don’t couple the Digital and Physical Fulfillment services to each other directly, even for convenience (for example, having Physical Fulfillment call Digital Fulfillment synchronously to “check if the whole order is done”). Any cross-fulfillment coordination should go through the Order Service’s aggregate status, keeping the two pipelines genuinely independent.
• Do write integration tests that specifically exercise mixed carts, not just pure-digital or pure-physical carts. The interesting bugs in this system — a partial Saga failure, an incorrect proration on a discounted mixed cart, a null shipping address for a digital-only order — only surface when both fulfillment types are present in the same test case.

Nothing validates a design like seeing production systems using the same shape. Each of the companies below has independently converged on some version of “unified checkout, split fulfillment” at very different scales and product mixes.

These are the questions that come up repeatedly in interviews, design reviews, and post-incident retrospectives on hybrid marketplaces. Each answer traces back to a decision made earlier in this tutorial.

If you remember only one idea from this tutorial, make it this: the order is one, but the fulfillment is many. Every other decision — from the Saga to the schema to the notification templates — is a downstream consequence of taking that principle seriously.

The throughline across every section of this tutorial is the same idea applied at different layers of the stack: keep what the customer sees simple and unified, and let the complexity of “how it actually gets to them” live in independent, decoupled backend pipelines that can each scale, fail, and evolve on their own terms. Whether you’re designing the database schema, the event contracts, the Saga compensations, or the notification templates, that one principle — separate the promise from the delivery mechanism — is what keeps a hybrid fulfillment marketplace maintainable as it grows from a handful of SKUs to millions of orders a day.

24.1 Where to go from here

If you are studying this design for an interview, redraw the architecture diagram from memory and then deliberately try to break it: ask what happens when the Payment Service is up but the Order Service database is failing over, what happens when a duplicate OrderCreated event arrives at both fulfillment consumers, and what happens when a customer cancels a mixed order after the digital key has been delivered but the physical item has not yet shipped. If you can answer each of those from the primitives introduced in this tutorial — the Saga log, idempotent consumers, per-line-item state, and typed compensations — you have genuinely internalized the design.

If you are building a system like this for real, start smaller than this tutorial suggests: a modular monolith with a clean line-item fulfillment abstraction and a real event bus in front of two well-separated fulfillment modules will take you very far, and it will let you carve out true microservices only when a specific pipeline actually earns its own operational overhead. Whichever path you take, keep coming back to the question that quietly drives every architectural choice in these twenty-four sections: does this decision let the two fulfillment realities live independently, or does it accidentally couple them together again?