Imagine you want to buy a specific model of wireless headphones. Ten different sellers on a marketplace — or even ten different websites entirely — might be selling that exact same product, each at a different price, with different shipping costs and delivery dates. A price comparison system is the piece of engineering that watches all of those sellers at once, figures out who is cheapest right now, and shows that answer to a shopper before they even finish typing the product name.

Think of it like a group of friends who each go check a different store for the same toy, then run back and shout the price to you the moment they find out. The faster your friends run, and the more often they check, the more accurate and “real-time” your information is. A price comparison system is that same idea, except the “friends” are software services, the “stores” are seller systems or external websites, and the “running back” happens over networks in milliseconds instead of minutes.

1.1 A brief history

1.2 Why “real-time” matters more today than ever before

Two decades ago, prices on physical shelves rarely changed more than once a day, so a comparison engine that refreshed once every few hours was perfectly adequate. Today, many large sellers run automated repricing software that can adjust a price dozens of times an hour in response to a competitor’s move, inventory levels, or even the time of day. If a comparison system still worked on an hourly refresh cycle, shoppers would routinely see prices that had already changed several times since the last update — and worse, a shopper might click “buy” on a price that is no longer honored, creating a poor experience and eroding trust in the platform. This shift from “prices change occasionally” to “prices change continuously and automatically” is the single biggest reason the real-time architecture in this article exists at all.

It also matters because comparison shopping has become a default consumer habit rather than a niche behavior. Shoppers routinely open multiple tabs, use browser extensions, or rely on a marketplace’s own “other sellers” panel before completing a purchase. A platform whose comparison feature feels sluggish or unreliable pushes shoppers toward a competitor’s app instead, which turns “make prices update fast” from a nice-to-have engineering goal into a direct, measurable revenue concern for the business.

Let’s state the problem precisely, the way an interviewer would expect you to restate it before designing anything:

2.1 Why is this hard?

At first glance it sounds simple: “just look up the price and show it.” The difficulty comes from scale, freshness, and disagreement between sources happening simultaneously:

2.2 What makes this different from a typical CRUD application

A standard create-read-update-delete application usually assumes one trusted writer per record and a comfortably small number of concurrent readers. This system inverts both assumptions at once: there are thousands of independent, only partially trusted writers (each seller) contending to update the same small set of records (offers for a given product), while there are simultaneously millions of readers expecting an answer in well under a second. That combination — many untrusted concurrent writers, massive concurrent readers, and a tight latency budget — is precisely what pushes the design away from a simple synchronous request/response model and toward the event-driven, cache-heavy architecture explored throughout this article.

2.3 Goals of the system

• Freshness: A shopper should see a price that is at most a few hundred milliseconds to a few seconds old.
• Correctness: The displayed “lowest price” must actually be the lowest currently valid, in-stock offer — not a stale or unavailable one.
• Scale: Support millions of concurrent shoppers browsing, searching, and comparing, and thousands of sellers pushing updates concurrently.
• Extensibility: New sellers, new product categories, and new currencies/regions should be easy to onboard.
• Resilience: A single slow or broken seller feed must never degrade the experience for products from other sellers.

2.4 Putting numbers on the problem

Before designing anything, it helps to reason about rough scale, because the right architecture for ten sellers looks nothing like the right architecture for a hundred thousand sellers. Consider a mid-to-large marketplace with the following back-of-envelope numbers:

Notice the ratio: read traffic outweighs write traffic by roughly 400 to 1. This single observation drives most of the architecture — it tells us to optimize the read path aggressively (caching, replicas, CDN) while making sure the write path is durable and correctly ordered rather than raw-throughput-optimized, since it simply doesn’t need to be as fast in absolute terms as the read path does.

2.5 Explicit non-goals

Just as important as stating goals is stating what this system deliberately does not try to solve, since a design that tries to do everything usually does nothing well:

• This system does not guarantee the shopper will always pay the exact displayed price — checkout systems perform a final authoritative price check, and the comparison system’s job is to be a very good, very fast estimate, not the final source of truth at the moment of payment.
• This system does not attempt hard real-time guarantees (bounded worst-case latency) — it targets a strong best-effort freshness SLA instead.
• This system does not itself set prices or run repricing algorithms for sellers — that logic lives in each seller’s own systems; this platform only ingests, compares, and displays.

Before diving into architecture, let’s define the building blocks in plain language. Each of these terms will come up repeatedly, so understanding them now makes the rest of the design click into place.

3.1 Product Catalog

What: A master, canonical record of every distinct product the platform knows about (e.g., “Sony WH-1000XM5 Headphones, Black”). Why: Without one canonical product, the system cannot know that Seller A’s “Sony XM5 Wireless Headset Black” and Seller B’s “Sony WH1000XM5 (Black)” are the same item. Analogy: Like a library’s master catalog card that all copies of a book — hardcover, paperback, different publishers — are linked back to. Example: Amazon’s ASIN, or Google Shopping’s GTIN/product ID, are canonical product identifiers.

3.2 Seller Offer

What: A specific seller’s listing for a canonical product — including their price, stock, shipping cost, and delivery estimate. Why: Many offers map to one product; the comparison engine’s job is to rank offers, not products. Analogy: If the product is “a taxi ride from A to B,” an offer is one specific driver’s quoted fare for that ride right now. Example: Ten sellers each have an “offer” row for the same headphones, each with its own price field.

3.3 Price Normalization

What: Converting all incoming prices into one comparable unit — same currency, same tax treatment, same shipping inclusion — before comparing them. Why: $50 + $10 shipping from Seller A is not directly comparable to €45 with free shipping from Seller B until both are converted to the same basis. Analogy: Converting recipe measurements from cups and grams to one unit before comparing which recipe uses more sugar. Example: A normalization service converts every offer to “total landed cost in USD” before ranking.

3.4 Event-Driven Ingestion

What: Sellers (or scraping/polling adapters) emit “price changed” events into a message queue rather than the platform constantly asking “did anything change?” Why: Push-based ingestion scales far better than polling millions of products on a fixed schedule. Analogy: A notification on your phone versus manually refreshing an app every 5 seconds to check for news. Example: A Kafka topic named price-events that seller adapters publish to.

3.5 Real-Time Push (WebSockets / Server-Sent Events)

What: A persistent connection between the client and server that lets the server push new prices to the browser without the browser asking again. Why: Enables shoppers watching a product page to see a price update appear live, without refreshing. Analogy: A live sports scoreboard app that updates itself, versus one you must manually refresh. Example: A WebSocket channel product:12345:prices that pushes new offer data to any subscribed browser tab.

3.6 Dead-Letter Queue

What: A separate holding queue for messages that repeatedly fail processing, rather than being retried forever or silently dropped. Why: Without one, a malformed event could either block an entire partition’s processing indefinitely, or simply vanish with no trace when discarded — both are bad outcomes for a system that must account for every price change. Analogy: A postal service’s “undeliverable mail” bin — items that couldn’t be delivered aren’t destroyed, they’re set aside for a human to investigate. Example: If a price event references a product ID that doesn’t exist in the catalog, after a few retries it lands in a dead-letter topic where an automated job or engineer can inspect and resolve it.

3.7 Read-Your-Own-Writes Consistency

What: A guarantee that after a specific actor makes a change, that same actor immediately sees their own change reflected, even if the wider system is still eventually consistent for everyone else. Why: Without it, a seller who just updated their price in a dashboard might refresh the page and see their old price, reasonably concluding the update failed. Analogy: Editing your own social media post and immediately seeing your edit, even though other viewers’ cached copies of your feed might take a moment longer to update. Example: The seller dashboard routes a seller’s own read requests to the primary database or a guaranteed-caught-up replica for a short window after they submit a change, rather than to an arbitrary potentially-lagging replica.

3.8 Eventual Consistency

What: A guarantee that all parts of the system will eventually agree on the latest price, but might briefly disagree right after a change. Why: Strong (immediate, all-or-nothing) consistency across a globally distributed, high-throughput system is prohibitively slow; eventual consistency trades a tiny, bounded staleness window for massive scalability. Analogy: Several people editing copies of a shared shopping list that sync up a few seconds later — briefly out of sync, but they converge. Example: A cache might serve a price that is 400ms old while the database already has the newer value.

3.9 Idempotency

What: A property where applying the same operation multiple times has the same effect as applying it once. Why: Message queues like Kafka typically guarantee at-least-once delivery, meaning the same price event can be delivered and processed more than once after a retry — without idempotency, a duplicate delivery could double-apply a discount or corrupt a counter. Analogy: Pressing an elevator call button five times doesn’t call five elevators — the operation “request this floor” is idempotent. Example: Each price event carries a unique event ID; the Normalization Service checks a short-lived deduplication cache before applying it, so a re-delivered event is silently ignored the second time.

3.10 Backpressure

What: A signal or mechanism that tells an upstream producer to slow down because a downstream consumer cannot keep up. Why: Without it, a burst of seller updates (say, during a major sale event) could overwhelm the Normalization Service faster than it can process, causing unbounded memory growth or crashes. Analogy: A dam’s spillway gates that open only as fast as the river downstream can safely carry the water away. Example: Kafka naturally provides backpressure because consumers pull at their own pace; the queue simply grows (bounded by retention) rather than the consumer being force-fed data it can’t process.

3.11 Rate Limiting & the Token Bucket Algorithm

What: A control that caps how many requests a client (here, a seller’s feed or a shopper’s API client) can make in a given time window. Why: Protects shared infrastructure from being overwhelmed — intentionally or accidentally — by any single caller. Analogy: A bucket that refills with tokens at a steady rate; every request consumes one token, and once the bucket is empty, requests must wait for it to refill. Example: Each seller is allotted, say, 100 price-update calls per second; a token bucket at the API Gateway enforces this cleanly, allowing short bursts while capping the sustained average rate.

3.12 Exponential Backoff & Jitter

What: A retry strategy where each failed request waits progressively longer before retrying (e.g., 1s, 2s, 4s, 8s), with a small random “jitter” added to each wait. Why: Without backoff, a failing downstream service gets hit with an immediate flood of retries, making the outage worse (a “retry storm”). Jitter prevents many clients from retrying at exactly the same moment and re-creating the same spike. Analogy: If a shop is closed, you don’t keep knocking every second — you wait a bit longer each time, and different people naturally knock at slightly different times rather than in perfect unison. Example: The Seller Feed Ingestion Service uses exponential backoff with jitter when a seller’s webhook endpoint fails, rather than hammering it every second.

3.13 Circuit Breaker

What: A safety mechanism that stops calling a failing dependency for a cooldown period after repeated failures, instead of retrying forever. Why: Prevents a slow or broken downstream dependency from exhausting threads/connections in the caller, which would otherwise cascade the failure elsewhere. Analogy: An electrical circuit breaker that trips and cuts power rather than letting a short circuit start a fire. Example: If a seller’s price API times out five times in a row, the ingestion adapter “opens” the circuit for that seller for 60 seconds before trying again, rather than queuing indefinitely.

3.14 Consistent Hashing

What: A hashing technique that maps both data (e.g., product IDs) and servers (e.g., cache/database nodes) onto the same conceptual ring, so each piece of data is owned by the nearest node on the ring. Why: When a node is added or removed, only the data between that node and its neighbor needs to move — not the entire dataset, as would happen with naive modulo hashing. Analogy: Assigning house addresses along a circular street, where adding one new house only affects its immediate neighbors’ numbering, not the whole street’s. Example: The cache tier uses consistent hashing on product ID so that resizing the Redis cluster from 8 to 10 nodes only reshuffles roughly 20% of keys, not 100%.

3.15 Connection Pooling

What: Reusing a fixed set of already-open database or network connections instead of opening a brand-new connection for every request. Why: Establishing a new TCP/TLS connection has real overhead (round trips, handshake cost); reusing connections dramatically reduces latency and resource usage under high request volume. Analogy: A taxi rank with a fixed number of cars circulating continuously, versus building a brand-new car for every single passenger. Example: The Price Comparison Service maintains a pool of, say, 50 persistent connections per instance to the database and Redis, rather than opening and closing a connection on every single request.

Below is the high-level architecture. Every box is explicitly labeled with the component it represents — including the API Gateway and Load Balancer layers — so you can see exactly where each responsibility lives.

4.1 Component breakdown

4.2 Why microservices instead of a monolith?

It’s fair to ask whether all of this really needs to be split into separate services — couldn’t one well-written application handle catalog, ingestion, normalization, and comparison together? For a small catalog with a handful of sellers, a monolith is genuinely simpler to build and operate, and is often the right starting point. But as the numbers in the Problem section show — hundreds of thousands of sellers, tens of millions of products, wildly different load patterns between writes and reads — a monolith becomes a liability for three concrete reasons. First, the ingestion pipeline needs to scale independently and burst aggressively during sales events, while the comparison service needs steady, always-on horizontal scale; bundling them means over-provisioning one to satisfy the other. Second, a bug or slowdown in, say, currency conversion logic should never be able to take down the ability to serve already-cached comparisons — separate services provide a natural fault boundary. Third, different teams typically own different domains (catalog quality, seller integrations, ranking algorithms) and independent services let them deploy on their own schedules without a shared release train blocking everyone.

4.3 Component responsibilities at a glance

This table is worth building explicitly in an interview setting, because it demonstrates you’ve thought not just about what each piece does when everything works, but what specifically breaks — and what still keeps working — when it doesn’t.

4.4 Seller-side ingestion in detail

The “Seller Feed Ingestion Service” box in Figure 4.1 is itself a small system with three distinct intake mechanisms, since not every seller integrates the same way:

Webhooks are strongly preferred wherever a seller supports them, since they are push-based and near-instant. Scheduled polling is the fallback for sellers whose systems only expose a pull-based API — polling frequency here is often tiered by seller sales volume, so a top seller might be polled every 10 seconds while a low-volume seller is polled every few minutes. Scraping is used only as a last resort, is rate-limited aggressively to respect the target site, and is treated as inherently less reliable — its output is weighted with lower confidence in the anomaly-detection step described in the Security section.

Let’s trace exactly what happens, component by component, when a seller changes a price.

5.1 Step-by-step lifecycle

1. Seller emits a change. A seller’s repricing bot decides to lower the price of a product from $79.99 to $74.99. It either (a) calls a webhook the platform exposed to them, (b) the platform’s polling adapter notices the change on its next scheduled poll of the seller’s feed, or (c) — only as a last resort for sellers with no API — a compliant scraper adapter detects the change on the seller’s public page.
2. Ingestion & validation. The Seller Feed Ingestion Service receives the raw payload, validates its shape and authenticity (signed webhook, API key, or scraper checksum), and converts it into one internal canonical event format regardless of source. Invalid or suspicious events (e.g., price of $0, or a 90% price drop that looks like a data error) are flagged for a sanity-check step rather than published directly.
3. Publish to queue. The validated event is published to a Kafka topic partitioned by product ID, guaranteeing all events for the same product are processed in order by the same consumer.
4. Normalization. The Price Normalization Service consumes the event, converts currency if needed, applies tax rules for the shopper’s likely region, folds in shipping cost, and produces a “landed cost” figure — the number that will actually be compared against other sellers’ offers.
5. Write-through to cache and database. The normalized offer is written to the primary database (durable source of truth) and to the Redis cache (fast read path) in the same logical step, along with a version/timestamp so stale writes can be detected and discarded if they arrive out of order.
6. Search index update. The updated price is also pushed into Elasticsearch so that browse/search/sort-by-price experiences reflect the new price without waiting for a batch reindex.
7. Push to active viewers. If any shoppers currently have that product page open, the Notification Service pushes the new price over their open WebSocket/SSE connection so the price visibly updates on their screen without a page refresh.

5.2 End-to-end sequence diagram

5.3 The lifecycle of a single offer, as a state machine

It’s useful to think of every individual offer (one seller’s price for one product) as moving through a small set of well-defined states, rather than as a single opaque “price” field. This makes error handling and debugging far more tractable, because at any moment you can ask “which state is this offer stuck in?”

5.4 Error handling & retry strategy

Failures happen at every hop of this pipeline, so each step needs an explicit strategy rather than hoping errors don’t occur. If the Normalization Service cannot reach the currency-rate provider, it retries with exponential backoff and jitter (introduced in Core Concepts) rather than retrying immediately in a tight loop. If retries are exhausted, the event is written to a dead-letter queue — a separate Kafka topic holding events that could not be processed — so it is never silently dropped, and an on-call engineer or automated job can inspect and reprocess it later. If the database write succeeds but the cache write fails, the system relies on the cache’s own short TTL to self-heal — the stale cache entry will simply expire and be refreshed from the now-correct database value within seconds, which is a deliberate and acceptable trade-off given the very short TTLs used for hot products. If, conversely, the database write itself fails after the event has already been marked as consumed from the queue, the consumer must not blindly commit its offset — instead, the commit only happens after a successful write, so a crash or database outage mid-processing simply results in the event being re-delivered and reprocessed on restart, which is safe precisely because the write itself is idempotent.

It’s also worth designing explicitly for the case where an entire downstream dependency — say, the search index — is unavailable for an extended period while the database and cache continue working normally. Rather than blocking the whole pipeline on the slowest dependency, each downstream write (database, cache, search index, push notification) is treated as an independent, separately-retried step. A search index outage delays only search-result freshness, while direct product-page price lookups (served from cache and database) continue to update normally and immediately — a good illustration of why decoupling the fan-out into independent steps, rather than one large all-or-nothing transaction, keeps a partial outage from becoming a total one.

public class RetryWithBackoff {

    public static <T> T executeWithRetry(
            Callable<T> task, int maxAttempts) throws Exception {

        int attempt = 0;
        while (true) {
            try {
                return task.call();
            } catch (Exception ex) {
                attempt++;
                if (attempt >= maxAttempts) {
                    throw ex; // exhausted retries, send to dead-letter queue
                }
                long baseDelayMs = (long) Math.pow(2, attempt) * 100;
                long jitterMs = ThreadLocalRandom.current().nextLong(50, 150);
                Thread.sleep(baseDelayMs + jitterMs);
            }
        }
    }
}

It helps to separate the system into two distinct data flows that run continuously and independently: the write path (sellers pushing price changes in) and the read path (shoppers pulling comparisons out). They share storage but have very different performance requirements.

6.1 Read path walkthrough

1. Shopper opens a product page or searches for a product category.
2. Request hits the CDN — for a specific hot product page, edge caching may serve a very recent snapshot directly.
3. Cache miss (or dynamic comparison request) routes through the API Gateway and Load Balancer to the Price Comparison Service.
4. The service checks Redis first; on a cache hit, it returns immediately.
5. On a cache miss, it queries the database, ranks the offers, writes the result back into Redis, and returns it.
6. The client opens a WebSocket subscription for that product so future price changes are pushed live.

6.2 Why three different data stores for one piece of data?

It can seem redundant that a single price update ends up written to the database, the cache, and the search index — three copies of essentially the same fact. This redundancy is deliberate, because each store is optimized for a different access pattern that the others handle poorly. The database is the durable source of truth, optimized for correctness and point lookups by product/seller key. The cache is optimized purely for raw read speed on the hottest keys, trading some memory cost for sub-millisecond access. The search index is optimized for a completely different query shape — free-text search combined with filtering and sorting across millions of products at once, which neither a key-value cache nor a traditional row-oriented database index handles efficiently. Accepting the operational cost of keeping three stores in sync is the price paid for each individual read — whether it’s “get this one offer,” “check the cache,” or “search and sort 50,000 headphones by price” — being fast.

6.3 Lifecycle beyond the first write

A price doesn’t just get written once and sit static — offers continuously expire and get refreshed even without an explicit seller update. Every cached offer carries a TTL; if a seller’s feed goes quiet for longer than a defined threshold (say, 30 minutes with no update and no successful poll), the offer is proactively marked “unconfirmed” and deprioritized in ranking, rather than being trusted indefinitely just because it was correct the last time anyone heard from that seller. This protects shoppers from acting on a price that might have silently gone stale because a seller’s integration quietly broke rather than because nothing changed.

7.1 Key trade-off: consistency vs. latency

This is fundamentally a CAP theorem trade-off (explored further in Advanced Topics). Choosing to serve reads from cache — accepting a small staleness window — is a deliberate choice of availability and low latency over strict consistency. The alternative (always reading the primary database directly) would guarantee freshness but collapse under read load and increase latency for every shopper, every time.

7.2 Trade-off: push (webhooks) vs. poll (scheduled fetch)

Choosing how sellers deliver price changes is itself a significant trade-off decision, not an implementation detail:

In practice, production systems use both, favoring webhooks for large, technically sophisticated sellers and falling back to polling for the long tail of smaller sellers — with polling also serving as a periodic reconciliation check even for webhook-integrated sellers, to catch any events a broken webhook sender might have silently dropped.

7.3 Trade-off: normalization accuracy vs. processing speed

The Price Normalization Service could, in principle, apply extremely precise, region-specific tax rules for every possible shopper jurisdiction before publishing a price — but doing so for every single incoming event would add meaningful processing latency to the write path, directly hurting the freshness metric the whole system is built around. Most production designs instead normalize to a reasonably accurate default (e.g., the seller’s home jurisdiction tax treatment, or a common baseline region) immediately for speed, and apply final, fully precise, shopper-specific tax and duty calculations only at the point the shopper actually views a specific offer or proceeds to checkout — trading a small amount of display-time approximation for a much faster, more scalable write path, while guaranteeing the number that actually matters (what the shopper is charged) is always computed precisely at the moment it counts.

7.4 Trade-off: build vs. buy for infrastructure components

Not every piece of this architecture needs to be built in-house. Message queuing, search indexing, and caching are all available as fully managed cloud services (e.g., managed Kafka offerings, managed Elasticsearch/OpenSearch, managed Redis). Using managed services trades some cost and a degree of control for dramatically reduced operational burden — no need to run upgrade cycles, capacity planning, or on-call rotations for the underlying infrastructure. Most teams building a system like this choose managed infrastructure for the “commodity” pieces (queue, cache, search) and reserve custom engineering effort for the parts that actually differentiate the product — the normalization rules, matching logic, and ranking algorithm that make the comparison genuinely useful.

Two very different scaling problems exist side by side: scaling the number of sellers/products (write-side) and scaling the number of shoppers (read-side).

8.1 Scaling the write side

• Partitioning by product ID: Kafka topics are partitioned so that all updates for a given product route to the same partition, preserving per-product ordering while allowing horizontal scale-out across products.
• Backpressure-aware consumers: The Normalization Service scales out consumer instances independently as ingestion volume grows, without needing to touch the ingestion layer.
• Rate limiting per seller: The Seller Feed Ingestion Service enforces per-seller quotas so one misbehaving repricing bot cannot flood the pipeline.

8.2 Scaling the read side

• Multi-tier caching: CDN edge cache for the most popular products, Redis for everything else, database only on cold misses.
• Read replicas: Database read replicas absorb the residual read traffic that misses the cache, especially for less popular (“long-tail”) products.
• Horizontal service scaling: The Price Comparison Service is stateless, so it scales horizontally behind the Load Balancer simply by adding instances.

8.3 Handling hot products (the “thundering herd”)

During a flash sale, one product might receive far more traffic than average — a “hot key” problem. Mitigations include: caching at multiple layers (CDN + Redis), using a short client-side/edge TTL with background refresh so the cache never fully expires under load, and request coalescing (if 1,000 requests arrive for the same missing cache key simultaneously, only one actually queries the database while the rest wait for that result).

8.4 Worked example: sizing the comparison service with Little’s Law

Suppose product launches and marketing campaigns push the Price Comparison Service to a sustained peak of 80,000 requests per second, and each request — including the cache lookup and ranking logic — takes an average of 25 milliseconds to complete. Little’s Law ($L = lambda times W$) tells us the average number of requests “in flight” inside the service at any instant is $80{,}000 times 0.025 = 2{,}000$ concurrent requests. If each service instance is configured to safely handle 200 concurrent requests before latency starts degrading, this tells us we need at least $2{,}000 div 200 = 10$ healthy instances at peak — and capacity planning should target perhaps 15–20 instances to leave headroom for an instance failing health checks or a sudden 30% traffic spike. This is exactly the kind of quick, defensible capacity estimate an interviewer wants to see derived on the spot, rather than an arbitrary guessed number.

8.5 Capacity planning for seasonal peaks

Retail-adjacent systems like this one have a distinctive traffic shape: relatively steady baseline demand punctuated by predictable, enormous spikes around major shopping events. Rather than treating every spike as a surprise to be handled purely by reactive auto-scaling — which can lag behind a sudden 20x jump by the time new instances boot and warm up — mature teams model these events explicitly ahead of time. Historical traffic from the previous year’s equivalent event, adjusted for expected year-over-year growth, becomes the baseline capacity target; pre-warming caches, pre-scaling the service tier an hour before a known sale start time, and pre-approving temporary quota increases with the cloud provider are all standard practice. Auto-scaling then handles the residual, harder-to-predict variance on top of that pre-provisioned baseline, rather than being asked to handle the entire spike reactively from a cold start.

8.6 Database read scaling: replicas and sharding together

For the residual traffic that does reach the database (cache misses on long-tail products), a single layer of scaling is rarely enough at this volume. Sharding by product ID spreads data — and therefore write load — across many independent database clusters, while read replicas within each shard absorb read traffic without burdening the shard’s write leader. The combination means a single very popular shard can still be read-scaled independently by adding more replicas to just that shard, without needing to re-shard the entire dataset.

A price comparison feature that is down, or shows wrong prices, directly costs sales and erodes shopper trust — so this system is designed assuming components will fail, and focuses on containing the blast radius.

9.1 Key reliability techniques

9.2 Disaster recovery: RTO and RPO

Two metrics anchor any disaster recovery plan. Recovery Time Objective (RTO) is how long the system may reasonably be down before the business impact becomes unacceptable — for this system, a reasonable target might be under five minutes for a full regional failover. Recovery Point Objective (RPO) is how much data (in this case, how many seconds of price events) the business can tolerate losing in a worst-case disaster — given asynchronous cross-region replication, an RPO of a few seconds is typical, meaning at most a handful of the very latest price events might need to be replayed from the Kafka log after failover, since Kafka retains events for a configurable retention window and can be re-consumed from an earlier offset.

9.3 Backup strategy

Beyond live replication, the primary database takes regular automated snapshots (e.g., every few hours) plus continuous write-ahead log shipping, allowing point-in-time recovery — restoring the database to its exact state at any recent moment, not just to the last snapshot — which matters if a bad deployment corrupts data rather than a hardware failure destroying it.

9.4 A graceful degradation ladder

Rather than treating availability as binary (fully up or fully down), it helps to design an explicit ladder of degraded-but-still-useful states the system can fall back through as dependencies fail, each rung less ideal than the one above it but still far better than showing an error:

1. Full experience: Live comparison with real-time push, fully fresh prices.
2. Degraded push: WebSocket layer is down; client falls back to polling the REST API every few seconds instead of receiving instant pushes.
3. Degraded freshness: Normalization Service is delayed; the API serves the last successfully processed prices with a visible “may be slightly outdated” indicator rather than blocking.
4. Cache-only mode: Database is unreachable; the API serves whatever is currently in cache, refusing only genuinely uncached (cold) product lookups.
5. Read-only maintenance mode: Only in a severe, multi-component outage does the system stop accepting new seller price updates entirely while continuing to serve the best available cached comparisons to shoppers.

Designing this ladder in advance — rather than improvising during an actual incident — means on-call engineers have a clear, pre-agreed set of fallback behaviors to trigger, and shoppers experience a gradually degraded but still functional product instead of a hard outage.

9.5 Chaos engineering

Rather than waiting for a real outage to discover a weakness, mature teams deliberately inject controlled failures into a staging or even production environment — killing a random service instance, adding artificial network latency to a dependency, or simulating a full region outage — to verify that circuit breakers, retries, and failover actually behave as designed under real conditions, rather than only in theory. Running these “game days” regularly catches assumptions that silently broke as the system evolved.

This system ingests data from thousands of external, only-partially-trusted seller systems, and serves data to millions of anonymous shoppers — so it has meaningful attack surface on both sides.

10.1 Seller-side (inbound) security

• Signed webhooks / API keys: Every inbound seller update must be authenticated (HMAC signature or per-seller API key) so an attacker cannot inject fake price data.
• Anomaly detection: A 95% price drop or a price of $0.01 gets flagged for review rather than published instantly — preventing both honest bugs and deliberate price manipulation from reaching shoppers.
• Per-seller rate limiting: Prevents both accidental floods and deliberate denial-of-service attempts from a compromised seller account.

10.2 Shopper-side (outbound) security

• API Gateway authentication & rate limiting: Protects against scraping/abuse of the comparison API itself.
• TLS everywhere: All client-server and service-to-service traffic is encrypted in transit.
• Least privilege between services: The Notification Service, for example, should only be able to read prices — never write them — limiting the damage if it were ever compromised.

10.3 Multi-factor authentication for the seller portal

While the machine-to-machine ingestion API relies on signed webhooks and API keys, the human-facing seller dashboard — where a seller’s staff can log in and manually adjust prices, view analytics, or manage their integration settings — requires multi-factor authentication (MFA) for login, given that a compromised seller account could otherwise be used to push fraudulent low prices designed to attract fraudulent orders, or to disrupt a competitor’s listing through unauthorized access. Enforcing MFA specifically for any account with write access to pricing data is a proportionate control given the direct financial impact a compromised account could cause.

10.4 Web application firewall & DDoS protection

Because the read-facing API is public and high-traffic, it sits behind a Web Application Firewall (WAF) that filters known attack patterns (SQL injection attempts, malformed requests, known bad IP ranges) before traffic even reaches the API Gateway, and behind a dedicated DDoS mitigation layer at the network edge that can absorb and scrub volumetric attacks — a real risk for any publicly popular price comparison feature, since it is an attractive target for scraping-bot floods disguised as legitimate shopper traffic.

10.5 Encryption at rest and seller credential management

All persisted data — the primary database, cache snapshots, and backups — is encrypted at rest, not just in transit, so that a stolen disk or backup file is not directly usable. Seller API keys and webhook signing secrets are stored in a dedicated secrets manager rather than in application configuration files, with automated periodic rotation, so a leaked credential has a bounded window of usefulness to an attacker rather than remaining valid indefinitely.

10.6 Secure API design for the seller-facing gateway

The seller-facing ingestion API applies the same rigor as any public API: strict request schema validation (rejecting malformed payloads before they reach business logic), per-seller API key scoping (a key for Seller A cannot be used to submit prices for Seller B’s products), and comprehensive audit logging of every price submission — who submitted it, when, and from what source — which becomes essential evidence if a pricing dispute or suspected manipulation needs to be investigated after the fact.

10.7 Least privilege between internal services

Enforcing this table at the infrastructure level — via service-to-service authentication and scoped database roles, not just application-level convention — means that even if one service were compromised, the attacker’s reach is limited to that service’s narrow, explicitly granted permissions.

Because “the wrong price silently shown to a shopper” is a correctness bug that produces no errors or crashes, observability here must go beyond typical uptime/latency metrics — it must actively watch for correctness drift.

11.1 Distributed tracing

A single price update traverses seven or more services (ingestion, queue, normalization, cache, database, search, push). Distributed tracing (e.g., OpenTelemetry with a trace ID attached to every event) lets engineers see exactly where a specific update spent its time — critical for debugging “why did this price take 8 seconds to show up?”

11.2 SLIs, SLOs, and error budgets

Rather than vague goals like “be fast” and “be reliable,” mature teams define concrete Service Level Indicators (SLIs) — measurable signals such as P95 read latency or P95 freshness lag — and set a Service Level Objective (SLO) target for each, such as “P95 freshness lag under 2 seconds, measured over a rolling 30-day window.” The gap between 100% and the SLO becomes the error budget — for a 99.9% availability SLO, roughly 43 minutes of budget exists per month. As long as the team is within budget, they can ship new features and take reasonable risks; once the budget is exhausted, the team shifts focus to stability work until the budget replenishes. This turns “reliability” from a subjective argument into an objective, shared number everyone can look at.

11.3 Alerting philosophy: alert on symptoms, not causes

A common mistake in observability design is creating an alert for every possible internal cause — high CPU on one service, one slow database query, one dropped connection — which quickly produces so much noise that on-call engineers start ignoring alerts altogether. A more durable approach is to alert primarily on user-facing symptoms defined by the SLOs above — freshness lag breaching its threshold, error rate on the public API exceeding its budget, cache hit ratio dropping sharply — and treat internal-cause metrics (CPU, individual query latency, consumer lag on one specific partition) as diagnostic data to consult only once a symptom-level alert has already fired. This keeps the signal-to-noise ratio high and ensures every page an engineer receives at 3 a.m. genuinely reflects something a shopper or seller would notice.

11.4 Structured logging

Every service emits structured (JSON) logs rather than free-text strings, with consistent fields — traceId, productId, sellerId, eventVersion, service, latencyMs — across every service. This uniformity is what makes it possible to query “show me every log line related to product X in the last five minutes, across all seven services it touched” in a centralized log aggregation tool, rather than manually grepping through each service’s individual log files.

Each component is deployed as an independently scalable, containerized service (Docker, orchestrated by Kubernetes), which allows the write-heavy ingestion pipeline and the read-heavy comparison service to scale on entirely different schedules.

12.1 CI/CD pipeline

A typical pipeline for the Price Comparison Service runs through several automated gates before any code reaches production: unit tests, integration tests against a test instance of the cache/database, a container image build pushed to a private container registry, automated security scanning of that image for known vulnerabilities, deployment to a staging environment for smoke tests, and only then a canary rollout to production. Each gate can automatically halt the pipeline and roll back if it fails, meaning a broken change is caught well before it can affect real shoppers.

12.2 Container registry & image management

Every service is built into a versioned, immutable container image and pushed to a private container registry. Kubernetes then pulls a specific, pinned image version for each deployment — never “latest” — which guarantees that a rollback always restores an exact, previously-known-good state rather than an ambiguous moving target.

12.3 Environment parity

Staging environments mirror production as closely as is cost-effective — same service topology, same Kafka topic structure, same cache configuration — just at smaller scale. This “environment parity” principle catches an entire category of bugs (“it worked in staging but broke in prod”) that arise purely from configuration or topology differences rather than code differences.

12.4 Cloud region selection & data residency

Beyond pure latency optimization, region selection is also shaped by data residency requirements — many jurisdictions require that personal or transactional data about their residents be stored within regional boundaries. Since this system is largely about product and pricing data rather than personal shopper data, residency constraints are lighter here than in, say, a payments system, but seller account details and any shopper-identifying analytics data still need to respect the relevant regional requirements, which factors directly into which cloud regions host which pieces of the primary database.

13.1 Database choice & sharding

The primary database favors a model that handles high write throughput and horizontal scale well — many production systems use a distributed NoSQL or NewSQL store (e.g., a wide-column or document store) sharded by product ID, with each shard replicated for durability and read scaling.

13.2 Why consistent hashing?

Consistent hashing minimizes the number of products that must be reshuffled when a shard is added or removed — only a fraction of keys move, rather than nearly all of them, which matters enormously when the dataset spans hundreds of millions of offers.

13.3 Caching strategy

This system uses write-through caching for hot/popular products combined with cache-aside with short TTLs for the long tail — balancing freshness for the products that matter most against memory cost for the millions of rarely-viewed ones.

13.4 Load balancing approaches

• Round-robin / least-connections at the service tier, spreading requests evenly across stateless Price Comparison Service instances.
• Consistent hashing at the cache tier, so the same product ID always routes to the same Redis node — maximizing cache hit rate.
• Geo-based (GSLB) at the global tier, directing shoppers to their nearest healthy region.

13.5 Indexing strategy

The primary database indexes offers by (productId, sellerId) as the primary lookup key, with a secondary index on sellerId alone to support “show me all of this seller’s listings” queries used by seller dashboards. The Elasticsearch index, by contrast, is built for very different access patterns — full-text search on product titles/descriptions, faceted filtering (brand, category, price range), and sorting by computed landed cost — which is precisely why a separate search index exists alongside the primary database rather than trying to serve both access patterns from one store.

13.6 Handling replication lag

Because read replicas apply changes asynchronously, a replica can lag behind its leader by anywhere from single-digit milliseconds under normal load to a few seconds under heavy write pressure. For most comparison reads this is entirely acceptable given the system’s overall eventual-consistency stance. However, immediately after a shopper’s own action — for instance, right after a seller updates their own listing in a seller dashboard — the system routes that specific “read your own write” request to the primary (or to a replica confirmed to be caught up) rather than to an arbitrary replica, so the seller isn’t confused by seeing their own just-made change appear to have not taken effect.

13.7 Connection pooling at the database tier

With potentially thousands of Price Comparison Service instances, naively allowing each instance to open its own large pool of direct database connections would exhaust the database’s maximum connection limit almost immediately. A connection pooler sitting between the service tier and the database (such as a dedicated pooling proxy) multiplexes many logical application connections onto a smaller number of actual database connections, which is what makes horizontal scaling of the stateless service tier practical without redesigning the database tier every time capacity is added.

Each service exposes a small, focused API. Below is a simplified Java interface for the core comparison lookup, and the internal event contract used between ingestion and normalization.

@RestController
@RequestMapping("/api/v1/products")
public class PriceComparisonController {

    private final PriceComparisonService comparisonService;

    public PriceComparisonController(PriceComparisonService comparisonService) {
        this.comparisonService = comparisonService;
    }

    // Returns the ranked list of current valid offers for a product
    @GetMapping("/{productId}/offers")
    public ResponseEntity<OfferComparisonResponse> getOffers(
            @PathVariable String productId) {

        OfferComparisonResponse response =
                comparisonService.getRankedOffers(productId);

        return ResponseEntity.ok(response);
    }
}

// Canonical internal event published to the Kafka "price-events" topic
public class PriceChangeEvent {
    private String productId;
    private String sellerId;
    private BigDecimal rawPrice;
    private String currency;
    private BigDecimal shippingCost;
    private boolean inStock;
    private long eventVersion; // monotonically increasing, prevents out-of-order writes
    private Instant observedAt;

    // getters and setters omitted for brevity
}

14.1 Microservice boundaries

Services are split by business capability, not by technical layer — the Product Catalog Service owns everything about “what is this product,” while the Price Comparison Service owns everything about “what does it cost right now.” This means each team can deploy, scale, and evolve their service independently, as long as the event and API contracts between them stay stable.

14.2 API versioning

The public-facing API is versioned explicitly in the URL path (/api/v1/...), and breaking changes are only ever introduced in a new version (/api/v2/...), never by silently altering the existing contract. Both versions are typically kept running in parallel for a deprecation window (often 6–12 months) with clear communication and monitoring of v1 traffic, so client applications (mobile apps that can’t be force-updated instantly, third-party integrations) have a realistic runway to migrate before the old version is retired.

14.3 A token bucket rate limiter

Below is a simplified implementation of the token bucket rate limiter (introduced in Core Concepts) as applied at the Seller API Gateway to cap how often any single seller can push price updates.

public class TokenBucketRateLimiter {

    private final int capacity;
    private final double refillRatePerSecond;
    private double availableTokens;
    private long lastRefillTimestamp;

    public TokenBucketRateLimiter(int capacity, double refillRatePerSecond) {
        this.capacity = capacity;
        this.refillRatePerSecond = refillRatePerSecond;
        this.availableTokens = capacity;
        this.lastRefillTimestamp = System.nanoTime();
    }

    public synchronized boolean tryConsume() {
        refill();
        if (availableTokens >= 1) {
            availableTokens -= 1;
            return true;
        }
        return false; // seller must wait, request is rejected with HTTP 429
    }

    private void refill() {
        long now = System.nanoTime();
        double secondsElapsed = (now - lastRefillTimestamp) / 1_000_000_000.0;
        availableTokens = Math.min(capacity, availableTokens + secondsElapsed * refillRatePerSecond);
        lastRefillTimestamp = now;
    }
}

14.4 Additional API endpoints

15.1 Patterns used

15.2 Saga pattern — where it does (and doesn’t) apply

The Saga pattern coordinates a sequence of local transactions across services with compensating actions if a later step fails — commonly used for, say, an order checkout flow spanning payment, inventory, and shipping services. It’s worth explicitly noting this pattern is not heavily used within the price comparison pipeline itself, since updating cache, database, and search index for a price change are independent, idempotent, retryable writes rather than a multi-step business transaction requiring rollback semantics — recognizing when a pattern doesn’t fit is as valuable in an interview as knowing when it does.

15.3 Domain-specific anti-pattern: the “God Ranking Function”

A subtle anti-pattern specific to comparison systems is letting the ranking logic accumulate an ever-growing pile of special-case rules over time — one condition for a specific seller’s negotiated placement, another for a specific category’s shipping quirks, another for a one-off promotional override — until the function becomes an unreadable, untestable tangle that nobody fully understands or trusts. The fix is treating ranking as a composable pipeline of small, independently testable scoring rules (base price, seller trust, delivery speed, promotional weight) combined through an explicit, documented formula, rather than a monolithic function accumulating exceptions. This keeps the ranking logic auditable — an important property given how directly it affects both shopper trust and seller revenue.

15.4 Anti-patterns to avoid

• Always show a “last updated” indicator next to prices so shoppers understand freshness rather than assuming absolute real-time accuracy.
• Validate price sanity automatically before publishing — a $0 or negative price is almost always a bug, not a real offer.
• Design idempotent consumers everywhere in the pipeline, since at-least-once delivery (the norm for Kafka) means the same event can arrive twice.
• Decouple product matching from price comparison — matching is a separate, harder ML/rules problem and should not block the comparison service’s critical path.
• Load test the read path at 10x expected peak — sudden traffic surges (viral products, flash sales) are the norm, not the exception, for this kind of feature.

16.1 Additional practices worth adopting

• Version every schema change to the internal event format, and support reading both the old and new version for a transition period, so a partial rollout of the Normalization Service doesn’t break on events from a not-yet-updated Ingestion Service.
• Treat the seller onboarding flow as a first-class product, not an afterthought — the easier it is for a new seller to integrate correctly, the fewer malformed or unreliable feeds the platform has to defensively handle later.
• Separate “hot” and “long-tail” caching policies explicitly — applying one universal cache TTL to both a viral bestseller and an obscure niche product wastes memory on the long tail or under-serves freshness on hot items; segment cache policy by observed traffic.
• Run regular reconciliation jobs that compare a sample of live prices against a fresh, direct check of the seller’s source, catching silent pipeline drift (a stuck consumer, a silently failing webhook) before shoppers notice it themselves.
• Make staleness visible, not just internally measured — surfacing a lightweight “updated moments ago” indicator in the UI turns an internal SLO into user-visible trust.

16.2 Common mistakes beyond trusting seller data

• Treating cache invalidation as an afterthought: Bolting caching onto an already-built read path late in development often leads to inconsistent invalidation logic scattered across the codebase; designing the cache strategy alongside the data model from day one avoids this.
• Under-provisioning for known peak events: Flash sales and major shopping holidays are predictable in advance — capacity planning should explicitly model these rather than relying purely on reactive auto-scaling that may not react fast enough for a sudden 50x spike.
• Skipping load testing of the write path: Teams often load-test the read path heavily (since it’s higher volume) but under-test the write path’s behavior during a coordinated multi-seller repricing event, which is precisely when correctness matters most.

18.1 CAP theorem in this system

The CAP theorem states a distributed system can only fully guarantee two of Consistency, Availability, and Partition tolerance at once. Since network partitions are a fact of life at scale, the real choice is between CP (consistent but may reject requests during a partition) and AP (available but may briefly serve stale data). This system deliberately chooses AP for the read path — shoppers get a fast answer that’s very likely correct, rather than occasionally getting no answer at all while the system re-establishes strict consistency.

18.2 Replication & consensus

Database shards use leader-based replication (e.g., Raft-style consensus) so that writes go to one leader per shard and are replicated to followers, giving durability and read scaling while avoiding the complexity of full multi-leader conflict resolution for what is, per shard, a fairly low write volume.

18.3 Concurrency considerations

Multiple price updates for the same product can arrive concurrently from different sellers (not conflicting — they’re independent offers) but a single seller’s own repricing bot might also send rapid successive updates for one product. Per-product-per-seller sequencing (via Kafka partition key = productId+sellerId) plus optimistic concurrency control (a version check on write) prevents a race where an older update overwrites a newer one.

18.4 Failure recovery

If the Normalization Service crashes mid-processing, Kafka’s consumer offset mechanism ensures no event is lost — processing resumes from the last committed offset on restart. Combined with idempotent writes (keyed by event version), this guarantees at-least-once delivery without risking duplicate or out-of-order application of price changes.

18.5 Product matching at scale

Matching offers from different sellers to one canonical product typically combines deterministic matching (shared identifiers like UPC/GTIN/ISBN when available) with probabilistic/ML matching (title similarity, image embeddings, attribute comparison) for the large fraction of listings that lack a clean shared identifier — with a human review queue for low-confidence matches above a certain sales volume.

18.6 Consistent hashing, worked through

Recall from Core Concepts that consistent hashing places both cache nodes and data keys on a conceptual ring using a hash function. To look up which node owns a given product ID, hash the product ID to get a position on the ring, then walk clockwise until the first node is found — that node owns the key. When a node is added, it only takes ownership of the small arc of the ring between itself and the next node clockwise, meaning only keys in that narrow arc need to move. Many production systems also add “virtual nodes” — each physical cache server is represented by many points on the ring rather than just one — which spreads load more evenly and avoids any single physical node ending up responsible for a disproportionately large arc purely by hash chance.

18.7 Distributed locking for concurrent repricing

While most price updates are independent writes that don’t conflict, occasionally two processes might need to update the same offer record concurrently — for example, an automated fraud-review process flagging an offer at the same moment the Normalization Service is writing a legitimate update to it. A distributed lock (commonly implemented using a data store like Redis with a time-bound lock key, or via a coordination service like ZooKeeper/etcd) ensures only one writer touches that specific record at a time, with the lock automatically expiring even if the lock holder crashes, so the system doesn’t deadlock waiting for a lock that will never be released.

18.8 Exactly-once vs. at-least-once, revisited

It’s worth stating plainly why this system explicitly chooses at-least-once delivery combined with idempotent processing over pursuing true exactly-once semantics end-to-end. Exactly-once processing across a distributed pipeline generally requires either distributed transactions (which add coordination latency at every hop) or careful two-phase commit-style protocols — both of which meaningfully reduce throughput and increase complexity. Since every write in this pipeline is naturally idempotent (a later price event with a higher version number simply overwrites an earlier one; applying the same event twice produces the same final state), the weaker, cheaper at-least-once guarantee is fully sufficient. This is a recurring theme in distributed systems design: understand what guarantee the problem actually requires, and resist paying for a stronger, more expensive one than necessary.

A real-time, multi-seller price comparison system is fundamentally two pipelines sharing one data layer: a durable, ordered, event-driven write path that absorbs constant seller price changes, and a fast, cache-first, horizontally-scaled read path that serves ranked comparisons to millions of shoppers. The two are deliberately decoupled by a message queue so that neither one’s failure modes or scaling needs constrain the other.

20.1 Key takeaways

• Separate the seller-facing write path from the shopper-facing read path; connect them with a durable, ordered queue.
• Normalize every offer to one comparable basis (currency, tax, shipping) before ranking.
• Cache aggressively for reads; accept eventual consistency as a deliberate, well-understood trade-off.
• Isolate failure per seller feed so one bad integration never degrades the whole platform.
• Monitor freshness and correctness explicitly — not just uptime — since stale or wrong prices fail silently.
• Prefer idempotent, at-least-once processing over expensive exactly-once guarantees the problem doesn’t actually require.
• Design for failure from the start — circuit breakers, bulkheads, and graceful degradation are not optional extras but core requirements.
• Treat security (signed feeds, anomaly detection, least privilege) as inseparable from correctness, since a manipulated price is as harmful to trust as a slow one.

20.2 A closing narrative

If you take one narrative away from this entire design: the system starts with an independent, unreliable set of sellers each changing prices whenever they like, and ends with a single, fast, trustworthy number a shopper can act on in under a second. Every layer discussed above — ingestion, queueing, normalization, caching, ranking, push delivery, and the observability wrapped around all of it — exists purely to make that unreliable-to-trustworthy transformation happen reliably, at massive scale, without ever letting one misbehaving seller or one traffic spike take down the experience for everyone else.

20.3 Interview preparation guidance

For anyone studying this as interview preparation, it’s worth practicing explaining this design at three different depths: a thirty-second elevator summary (event-driven ingestion feeding a cache-first read path, connected by a durable queue), a five-minute whiteboard walkthrough (the architecture diagram in Section 4 plus the lifecycle in Section 5), and a deep dive into any single component an interviewer chooses to probe further — whether that’s the sharding strategy, the consistency model, or the failure-handling approach. Being able to zoom fluidly between these three levels of detail, rather than only having one fixed depth of explanation prepared, is usually what distinguishes a strong system design interview performance from an average one. Practicing all three levels ahead of time, on a real whiteboard or blank document rather than only reading passively, is the single highest-leverage preparation step for turning this material into confident, fluent interview delivery.