Imagine you run a large online store. A rival just dropped the price of a popular headphone by 15%. Your pricing team reacts instantly and cuts your price too. But here is the catch: your product page for that headphone has already been rendered and cached millions of times — on your CDN, in your application cache, on a customer’s phone from an hour ago, in search engine snippets, and inside a comparison-shopping widget. If even a fraction of those cached copies keep showing the old, higher price, you lose sales, confuse customers, and in some jurisdictions you can even run into legal trouble for advertising an incorrect price.

This is not a hypothetical problem. It is one of the most common “hard” system design problems asked at senior and staff engineering interviews, precisely because it forces you to reason about caching, consistency, event-driven architecture, and scale all at once.

Historically, this problem did not exist in this form. In the early days of the web (1990s), most pages were rendered fresh on every request — a customer asked for a page, the server read the price from the database, and sent back HTML. There was no caching layer to worry about, but there was also no way to serve millions of users cheaply, because every single request hit the database.

As traffic exploded in the 2000s, engineers introduced caching everywhere — in-memory caches like Memcached and later Redis, full-page HTML caching, and Content Delivery Networks (CDNs) that serve pages from servers physically close to the user. Caching solved the scale problem beautifully, but it created a new one: staleness. The moment you cache something, you have made a promise to serve it fast — and a competing promise to eventually make it correct becomes much harder to keep.

Modern platforms like Amazon, Flipkart, Uber, and Airbnb solved this by building dedicated price propagation or cache invalidation pipelines: event-driven systems whose entire job is to detect a price change the instant it happens and chase down every single copy of the old price across the system, replacing or invalidating it before a customer can see stale data.

Let us define the problem precisely, the way an interviewer would expect you to restate it before designing anything.

2.1 Why this is genuinely hard

2.2 Goals

2.3 Non-goals

• We are not designing the pricing algorithm itself (dynamic pricing, demand-based pricing) — only how a decided price value gets propagated.
• We are not covering full order/payment/checkout consistency, only enough to explain why checkout always re-validates price server-side.

2.4 Framing the problem as a CAP-theorem trade-off

It helps to explicitly place this problem on the consistency-availability spectrum before designing anything. The CAP theorem tells us that under a network partition, a distributed system must choose between consistency (every reader sees the latest write) and availability (every request gets a response, even if possibly stale). For product page rendering, we deliberately choose availability with bounded, eventually-consistent staleness — a page is always allowed to respond, even if the price it shows is a few seconds old — while for the checkout and payment path we deliberately choose strong consistency, always reading the true, current price directly from the primary database before money changes hands. This split is the single most important design decision in the whole system: it lets us cache aggressively everywhere it is safe to do so, while drawing a hard, non-negotiable line at the one place where staleness would actually cost money.

2.5 Quantifying “millions of pages”

To make the scale concrete: a mid-size marketplace might have 50 million active product listings. A single price change on one popular item might be represented in 20-plus different cached artifacts once you count CDN edge nodes (often 200-plus points of presence worldwide, each potentially holding its own cached copy), application-level Redis clusters in multiple regions, a pre-rendered HTML snapshot, a search index document, and any client-side caches on mobile apps. Multiply this by thousands of price changes happening every minute across the catalog during a sale event, and you can see why “just purge everything” quickly becomes an engineering problem in its own right, not an afterthought.

At a high level, the system has six logical layers: clients, edge/CDN, gateway, application services, an event backbone, and storage. Every component in the diagram below is labeled with exactly what role it plays, because in an interview you should never draw an unlabeled box.

3.1 Component-by-component explanation

Load Balancer

The Load Balancer is the front door for all traffic that is not already served by the CDN edge. It performs TLS termination, distributes requests across many API Gateway instances using algorithms like round-robin or least-connections, and continuously health-checks backend nodes so a failed instance is pulled out of rotation within seconds.

API Gateway

The API Gateway sits behind the load balancer and is responsible for authentication, authorization, rate limiting per client, request validation, and routing each request to the correct microservice — for example, price-update requests go to the Price Service, while page-view requests go to the Page Render Service. It also often does response caching for very short TTLs (1–2 seconds) as a first line of defense against sudden traffic spikes.

Price Service

This is the single, authoritative owner of price data. Every write to a product’s price — whether from an internal pricing tool, a seller dashboard, or an automated repricing algorithm — must go through this service. Centralizing writes here means we always have one clear point where a “price changed” event can be generated, instead of trying to detect price changes from many different systems.

Change Data Capture (CDC)

Rather than asking the Price Service to remember to publish an event every time it writes (which is fragile — a bug or a crash after the DB write but before the event publish would silently lose the update), we attach a CDC connector like Debezium directly to the database’s write-ahead log (WAL). Every committed row change is captured automatically and turned into an event, guaranteeing we never miss a price change even if application code has bugs.

Kafka Cluster (Event Backbone)

Kafka acts as a durable, ordered, replayable log of every price change. Multiple independent consumers — cache invalidation, page rendering, search indexing, and real-time notifications — can all read the same stream of events independently, at their own pace, without competing with each other or with the Price Service.

Cache Invalidation Service

This is the fanout coordinator. For every price-change event, it decides which caches need to be touched: Redis keys to delete or update, CDN paths to purge, and search documents to patch. It batches and rate-limits these calls so a burst of price changes does not overwhelm the CDN’s purge API.

CDN Edge Nodes and Purge Agent

CDN edge nodes hold cached HTML/JSON close to the end user, drastically reducing latency and origin load. The Purge Agent is the interface that lets our backend tell hundreds of geographically distributed edge nodes “this URL is now stale” — most modern CDNs support both hard purges (delete immediately) and soft purges (serve stale while revalidating in background).

Redis Cache

Redis stores rendered page fragments and price lookup keys with short TTLs. It is the first layer application servers check before hitting the primary database, absorbing the vast majority of read traffic.

Page Render Service

Responsible for server-side rendering (SSR) or incremental static regeneration (ISR) of product pages. On receiving a price-change event, it can proactively re-render the hot pages (top N products by traffic) so the very next cache-miss request is served fresh HTML instantly rather than waiting for a slow render.

Notification Gateway (WebSocket/SSE)

For customers who already have the page open in their browser, we do not want to wait for them to refresh. This gateway maintains persistent WebSocket or Server-Sent-Events connections and pushes a small “price updated” message directly to the open tab, which then updates the DOM in place.

Search Index

Product search and category listing pages also show price, and they are backed by a separate search engine like Elasticsearch. The invalidation service patches the relevant document’s price field so search results stay consistent with the product page itself.

Object Store (Static Snapshots)

For extremely high-traffic, mostly-static product pages, some platforms pre-render full HTML and store it in an object store like Amazon S3, served through the CDN. When price changes, the Page Render Service regenerates and overwrites this snapshot.

3.2 Component responsibility and latency budget

A useful exercise when presenting this architecture in an interview is to assign each hop in the pipeline a rough latency budget, so the overall “seconds, not minutes” promise is grounded in concrete numbers rather than a vague aspiration.

Adding these up, a realistic end-to-end propagation time for the majority of the pipeline lands well under two seconds, with the CDN’s own internal global propagation typically being the single largest and least controllable contributor — which is exactly why the WebSocket push path exists as a parallel, faster channel for users who are already actively viewing the page.

3.3 Read-your-writes consistency for the price administrator

There is a subtle user-experience requirement worth calling out: the person who just changed the price should never see the old price reflected back to them, even for a brief moment, or they will reasonably assume the update failed. This is called read-your-writes consistency, and it is solved simply by having the Price Service’s own confirmation response to the admin dashboard include the new price and version directly from the write it just performed, rather than the dashboard making a separate read call that might hit a not-yet-invalidated cache.

3.4 What happens if each component fails, in isolation

A genuinely useful way to stress-test any architecture diagram is to mentally remove one box at a time and ask what actually breaks. If the Load Balancer fails, traffic simply cannot reach the platform at all — this is why load balancers themselves are deployed in redundant pairs or as a managed, highly available cloud service rather than as a single instance. If the API Gateway fails, all backend services become unreachable even though they themselves are healthy, which is why gateway instances are also horizontally scaled and load-balanced rather than run as a single point of failure. If the Price Service fails, new price changes simply cannot be accepted, but critically, already-committed prices remain fully correct and servable — reads are entirely unaffected, only writes pause until the service recovers. If Kafka becomes unavailable, no new events flow to any consumer, so cache invalidation, re-rendering, and notifications all stall, but because the Price Service already committed the price to the database before publishing depended on Kafka being healthy, no data is lost — it simply queues up and flows once Kafka recovers, assuming the Price Service itself buffers or retries publishing rather than dropping events on a Kafka outage. If a single Redis node fails, requests fall through to the origin and repopulate a healthy replica, at the cost of a temporary increase in database load proportional to how much traffic that node was serving. If the CDN itself has an outage, this is the most severe failure mode in the entire system, since it is the layer absorbing the vast majority of read traffic — this is precisely why serious platforms consider a multi-CDN strategy, so a single provider’s outage degrades rather than eliminates edge caching capacity.

Let us walk through what actually happens, step by step, from the moment someone clicks “Save” on a new price to the moment a shopper across the world sees it.

1. An authorized user or automated system sends a price update request through the API Gateway to the Price Service.
2. The Price Service validates the request (permissions, price sanity bounds, currency) and writes the new price to the primary database inside a transaction, along with a monotonically increasing version number or timestamp.
3. The database commits the transaction and appends the change to its write-ahead log.
4. The CDC connector picks up this WAL entry within milliseconds and publishes a structured PriceChanged event to Kafka, partitioned by product ID so that all changes for the same product are strictly ordered.
5. Three consumer groups read this event independently and in parallel:
   • The Cache Invalidation Service deletes or updates the Redis key for that product and calls the CDN purge API for that product’s URL.
   • The Page Render Service re-renders the product page HTML in the background and writes the fresh snapshot to the object store, so the next request is served instantly rather than triggering a slow render.
   • The Notification Gateway looks up which open WebSocket connections are currently viewing that product and pushes a tiny “price updated” payload.
6. The CDN edge nodes, upon receiving the purge signal, drop their cached copy. The very next request for that page is a cache miss, which is routed to the origin, which now returns freshly rendered HTML with the correct price — and the CDN re-caches this new version.
7. Any shopper who already had the page open sees the price update via the WebSocket push without needing to refresh at all.
8. Regardless of any of the above succeeding perfectly, every cached copy still carries a short TTL (for example, 30–60 seconds) as a safety net, guaranteeing an absolute upper bound on staleness even if a purge call is lost.

4.1 Idempotency and exactly-once effects

Kafka, like almost every distributed messaging system, offers “at-least-once” delivery by default — meaning a consumer might occasionally see the same PriceChanged event twice, for example after a consumer restart replays a few unacknowledged messages. Our downstream consumers must therefore be idempotent: processing the same event twice must produce exactly the same end state as processing it once. This is why every consumer checks the event’s version number before acting — deleting an already-deleted cache key, or purging an already-purged CDN path, is harmless and safe to repeat, but only if the consumer first confirms the event is not older than what it has already applied.

4.2 Ordering guarantees in practice

Kafka guarantees ordering only within a single partition, not across the whole topic. By partitioning the price-change-events topic on product ID, we ensure all changes for the same product always arrive at any given consumer in the exact order they were committed, while changes for different products can be processed fully in parallel across many partitions and consumer instances — giving us both correctness and horizontal scalability at the same time.

The sequence diagram below traces a single price-change request end to end, showing exactly which component talks to which, and in what order.

5.1 Lifecycle of a single price fact

It helps to think of a price value as having a lifecycle with distinct states: Committed (written to the database, the only place it is guaranteed correct), In-flight (the change event is traveling through Kafka to consumers), Propagating (invalidation and re-render calls are being made across caches and CDN), and finally Converged (every layer of the system now agrees on the new price). The entire goal of this architecture is to shrink the time between Committed and Converged as much as possible, while guaranteeing Converged always eventually happens even if individual steps fail.

It is worth dwelling on the most philosophically important trade-off in this table: choosing eventual consistency for browsing and strong consistency for checkout. Many engineers new to this problem instinctively want the entire system to be strongly consistent everywhere, reasoning that “showing a wrong price is bad, so let us never show a possibly-wrong price.” In practice this instinct, if followed literally, means routing every single page view through the primary database, which simply cannot survive the read volume of a popular e-commerce platform. The insight that unlocks a workable design is realizing that browsing and buying have fundamentally different consistency requirements: a shopper glancing at a product page can tolerate a price that is a second or two out of date, because nothing irreversible happens from that glance. The moment they commit to buying, however, staleness becomes unacceptable, because now real money and a real contractual obligation are on the line. Recognizing this asymmetry — and deliberately applying different consistency guarantees to different parts of the same user journey — is the single most transferable lesson from this entire system design problem, and it applies far beyond pricing to almost any read-heavy, write-light system with a “point of no return” step somewhere in its flow.

6.1 Multi-currency and tax complications

Real-world pricing rarely stops at a single number. Platforms operating across countries must propagate not just a price but also the correct currency, applicable tax treatment, and any region-specific promotional adjustments — all of which can change independently of the base price itself. A robust design treats “the displayed price” as a small, versioned computation over several independent inputs (base price, currency conversion rate, tax rules, active promotions) rather than a single flat field, and the propagation pipeline must be able to invalidate a cached page when any one of those inputs changes, not only when the base price itself changes. This adds real complexity, but the same event-driven, versioned architecture described throughout this tutorial extends naturally to cover it: each input simply becomes its own event type feeding into the same Cache Invalidation Service.

Since this entire design hinges on choosing the right consistency model for the right part of the system, it is worth spelling out the main models an interviewer expects you to know and where each one shows up in this architecture.

Notice that “eventual consistency” on its own is actually a fairly weak guarantee — it promises convergence but not a deadline. This is exactly why this design never relies on eventual consistency alone; every cache in the system is eventual-consistency-plus-a-TTL, which upgrades the weak guarantee into the much more useful bounded-staleness guarantee, giving both engineers and the business a concrete number they can reason about and put in a service-level objective.

The core scalability principle here is the classic read-heavy, write-light pattern: price reads (page views) vastly outnumber price writes (actual price changes) — often by a ratio of 100,000:1 or more. This tells us exactly where to invest: reads must be served almost entirely from cache and CDN, and only writes need to touch the primary database directly.

8.1 Sharding and partitioning

Kafka partitions by product ID so that changes for a single product stay ordered while different products can be processed in full parallel across many consumer instances. The primary database itself is typically sharded by product category or seller ID, so no single database node bears the entire write load.

8.2 Hot key problem

A flash-sale product can become a “hot key” — a single Redis key or CDN URL receiving disproportionate traffic. Mitigations include: replicating the hot key across multiple Redis nodes (read replicas), adding a short random jitter to TTLs so caches do not all expire at the exact same millisecond (“thundering herd”), and using request coalescing at the origin so that if 10,000 requests miss cache for the same key simultaneously, only one actually queries the database while the other 9,999 wait for that single result.

8.3 Batching invalidations

During a large sale event where thousands of prices change within seconds, the Cache Invalidation Service batches purge calls (for example, one CDN API call listing 500 URLs instead of 500 separate calls) to respect CDN provider rate limits and reduce network overhead.

8.4 Back-of-the-envelope capacity estimation

It is good interview practice to size the system with rough numbers. Assume 50 million product page views per day platform-wide, roughly 600 requests per second on average but bursting to 50,000 requests per second for a single flash-sale product. If Redis and CDN together absorb 99.9% of reads (a realistic hit ratio for a well-tuned cache), the primary database only needs to handle roughly 0.1% of that traffic directly — around 50 requests per second even at peak — which a modestly sized database cluster handles comfortably. This single calculation is usually enough to justify the entire caching architecture to an interviewer: without caching, that same flash-sale burst would require the database to survive 50,000 requests per second, which is a completely different, far more expensive scaling problem.

8.5 Read replicas and cache warm-up

Just before a scheduled sale event goes live, mature platforms proactively “warm” the cache and CDN for the products going on sale, pre-populating Redis and pre-rendering hot product pages a few minutes before the price change is publicly visible, so the very first wave of traffic never has to suffer a cold cache-miss storm at the worst possible moment.

8.6 Request coalescing in detail

Request coalescing deserves a closer look because it is one of the more elegant techniques in this design. When a cache entry expires or is invalidated for an extremely popular product, it is common for thousands of nearly simultaneous requests to all miss the cache at once and attempt to fetch the fresh value from the origin, a pattern often called a “thundering herd.” Without coalescing, all of those requests independently hit the database or the render service, potentially overwhelming it with duplicate work for what is ultimately the exact same piece of data. A coalescing layer sitting in front of the origin recognizes that multiple in-flight requests are asking for the same key, lets exactly one of them proceed to actually do the work, and has all the others simply wait for and share that single result once it completes. This turns a potential spike of thousands of redundant origin calls into just one, at the cost of a small amount of added latency for the requests that had to wait.

8.7 Horizontal scaling of stateless services

Because the Page Render Service, Cache Invalidation Service, and Notification Gateway are all designed to be stateless — holding no request-specific data in local memory between calls — they scale horizontally simply by adding more instances behind the load balancer, with no special coordination required between instances. This is a deliberate architectural choice: any state that does need to persist, like which WebSocket connections are subscribed to which product, is kept in a shared, externally accessible store (such as Redis) rather than in an individual instance’s local memory, so that any instance can handle any request and instances can be added or removed freely as load changes throughout the day.

Every component in this pipeline must assume its downstream neighbor can be temporarily unavailable, and must not lose data as a result.

9.1 Backpressure handling

During extreme bursts, if the Cache Invalidation Service starts falling behind Kafka’s incoming event rate, consumer lag rises. Rather than trying to process every single event with equal priority, well-designed consumers apply backpressure intelligently: they keep up with high-traffic product events in near real time while deliberately allowing lower-traffic product events to queue slightly longer, trusting the TTL safety net to cover the gap. This graceful degradation under load is far preferable to the service falling over entirely or processing everything so slowly that even hot products go stale.

9.2 Multi-region failover

If an entire cloud region hosting the Price Service and primary database becomes unavailable, traffic fails over to a standby region with a synchronously or near-synchronously replicated database. Because CDN edge nodes are already globally distributed and independent of any single region, read traffic for already-cached pages continues to be served with minimal disruption even during a regional outage — only new price writes are blocked until failover completes, which is an acceptable trade-off given how rare full regional outages are.

9.3 Graceful degradation from the customer’s point of view

It is worth explicitly designing what a customer experiences when parts of this pipeline are degraded, rather than leaving it as an accident of implementation. If the Notification Gateway is down, a customer simply does not get instant live updates on an open tab — a minor, invisible degradation, since the page will still show the correct price on their next natural refresh or navigation, well within the TTL window. If the CDN purge path is degraded, pages fall back to the TTL for freshness, meaning a slightly longer but still bounded window of possible staleness while browsing. The one thing that must never degrade, under any combination of failures, is the checkout price check itself — so that endpoint is deliberately built as the simplest, most direct path in the entire system, reading straight from the primary database with no caching layer in between at all, precisely so it has the fewest possible failure modes of anything in this architecture.

9.4 Health checks and automatic recovery

Every service in this pipeline exposes a liveness check (is the process still running and able to respond at all) and a separate readiness check (is the process not just alive, but actually able to do useful work right now — for example, does it have a healthy connection to Kafka and Redis). The load balancer and orchestration layer use these to automatically remove an unhealthy instance from rotation and replace it, without requiring a human to notice and intervene for the common case of a single instance becoming unhealthy.

Price manipulation is a high-value attack target — an attacker who can forge a price-change event could sell a product at ₹1 or make prices swing unpredictably.

10.1 Rate limiting price-write endpoints

The API Gateway enforces per-seller and per-service rate limits on price-update calls, preventing a misbehaving automated repricing script from flooding the Price Service and Kafka with an unreasonable volume of changes for a single product in a short window.

10.2 Anomaly detection on price changes

Beyond simple bounds checking, mature platforms run lightweight anomaly detection on the stream of price-change events itself — flagging, for example, a product whose price has changed more than a handful of times in a minute, or a seller account whose price changes deviate sharply from historical patterns, routing these to a human review queue before they take effect for very high-traffic items.

10.3 Encryption and secrets management

All traffic between the client and the edge, and between every internal service in this pipeline, is encrypted in transit using TLS, including the connection between Kafka brokers and consumers, which is easy to overlook since internal traffic can feel “safe” simply because it is inside a private network boundary. Database credentials, CDN API keys, and any signing keys used to authenticate internal service-to-service calls are stored in a dedicated secrets manager (such as HashiCorp Vault or a cloud provider’s native secrets service) rather than in configuration files or environment variables checked into source control, and are rotated on a regular schedule so a leaked credential has a limited window of usefulness to an attacker. Sensitive fields at rest, such as the audit log’s record of who changed a price and when, are protected with encryption at rest on the underlying storage volumes as a standard defense-in-depth measure.

10.4 Least-privilege access to the write path

Only a small, well-defined set of internal services and authenticated automated systems should ever be authorized to call the Price Service’s write endpoints. This is enforced through role-based access control at the API Gateway, combined with mutual TLS between internal services so that even a compromised network segment cannot impersonate an authorized caller. Human access for manual price corrections goes through a separate, more heavily audited administrative interface with its own approval workflow for unusually large changes, rather than sharing the same automated write path used by repricing algorithms.

Because this system’s entire purpose is speed and correctness of propagation, its most important metric is propagation latency: the time between a price commit in the database and the moment every cache layer reflects it.

Distributed tracing (for example with OpenTelemetry) should tag every price-change event with a single trace ID at the moment it is created, so engineers can follow that one event across CDC, Kafka, invalidation, CDN, and notification in one unified trace view, rather than piecing logs together manually across five different systems.

11.1 Alerting thresholds

Good alerting here is tiered rather than binary. A soft warning fires when p95 propagation latency crosses, say, 5 seconds, giving on-call engineers time to investigate before customers notice. A hard page fires if p99 latency crosses the TTL ceiling itself (meaning the safety net is now the only thing protecting correctness), or if Kafka consumer lag grows unboundedly, since either condition means stale prices could realistically reach real customers.

11.2 Synthetic monitoring

Beyond passively watching real traffic, teams run synthetic “canary” price changes on a small set of test products at regular intervals, then automatically measure how long it actually takes for that change to appear on the live product page, CDN, and search index — turning “we think propagation is fast” into a continuously measured, verified fact rather than an assumption.

11.3 What a good operational dashboard looks like

An on-call engineer should be able to answer “is the pricing pipeline healthy right now?” within seconds of glancing at a single dashboard, without digging through logs. A well-designed dashboard for this system typically groups panels into three tiers: a top row showing the headline propagation-latency percentiles and current Kafka consumer lag across all consumer groups, so the overall health of the pipeline is visible at a glance; a middle row breaking down cache hit ratios and CDN purge success rates per region, since regional CDN issues are common and easy to miss if everything is aggregated globally; and a bottom row surfacing business-facing signals like the count of confirmed stale-price incidents and checkout price-mismatch events, which is ultimately the metric that matters most to the business even though it sits furthest downstream from the technical pipeline itself.

11.4 Search index and cross-system consistency checks

Because the search index is updated by its own independent consumer, it is possible in rare failure scenarios for the product page and the search result snippet for the same product to briefly disagree on price. A periodic reconciliation job — comparing a sample of product prices in the primary database against what the search index currently holds — catches this class of drift proactively, well before a customer notices the same product showing two different prices in two different parts of the site, which is a particularly damaging kind of inconsistency because it is so visibly obvious to end users.

This architecture is naturally suited to a multi-region cloud deployment. The Price Service and primary database typically live in one or two regions (to keep writes strongly consistent), while CDN edge nodes and read replicas of the cache are deployed globally to keep read latency low everywhere.

12.1 Disaster recovery and backups

The primary database is backed up continuously using write-ahead-log shipping to a secondary storage location, allowing point-in-time recovery to any second within the retention window if a bad deploy or human error corrupts price data. Kafka topics themselves also act as a natural, replayable backup of every price change: if a downstream cache or search index is ever found to be inconsistent, operators can simply rewind the consumer group’s offset and replay events from an earlier point to rebuild correct state, rather than needing a separate backup mechanism for derived data.

12.2 Cost optimization

Running a globally distributed CDN, a large Kafka cluster, and a fleet of stateless application services is not free, so cost-conscious teams apply a few concrete levers. First, TTL tuning directly trades cost against freshness: a slightly longer TTL on low-traffic products meaningfully reduces CDN purge API calls and re-render compute without materially harming the customer experience, since few people are viewing those pages anyway. Second, tiered compute — reserved or committed-use instances for steady baseline load, combined with autoscaled spot or burstable instances for the Page Render Service during traffic spikes — keeps the expensive rendering workload from being provisioned at peak capacity around the clock. Third, right-sizing Kafka partition counts and broker instance types to actual sustained throughput (rather than over-provisioning for a rare worst case) is one of the most common places teams overspend without realizing it.

12.3 Infrastructure-as-code and environment parity

Every environment — local development, staging, and production — should be defined through infrastructure-as-code (Terraform, Pulumi, or similar) so that the Kafka topic configuration, cache TTL defaults, and service scaling rules are identical in shape across environments and differ only in scale. This matters specifically for this system because a subtle bug in event ordering or idempotency handling is often invisible at the small scale of a local development environment and only surfaces under real production concurrency, so staging environments should be provisioned with realistic (if smaller) partition counts and consumer group configurations rather than a single simplified instance of everything.

12.4 A simple capacity planning worksheet

When sizing this system for a new deployment, work through these questions in order: What is the peak page-view rate for the single hottest product you need to support (this drives cache and CDN capacity, not the average across the catalog)? What is the peak rate of price changes per second across the whole catalog during your busiest known event (this drives Kafka partition count and consumer throughput)? What staleness window is acceptable to the business, in seconds (this drives your TTL, and indirectly, how aggressively you need active invalidation to actually work)? And finally, what is an acceptable cost per additional second of staleness reduction (this is what ultimately decides how much infrastructure investment — more CDN purge capacity, more pre-rendering compute — is actually justified, since driving propagation latency from two seconds to two hundred milliseconds has a real, quantifiable cost that should be weighed against the real, but often smaller than assumed, business benefit).

A frequently underestimated part of this system is what happens when multiple writers try to change the same product’s price at nearly the same instant — for example, an automated repricing engine and a human merchandiser both updating the same product within milliseconds of each other.

13.1 Optimistic concurrency control

Rather than locking the price row for the duration of every write (which would serialize all writes to a hot product and hurt throughput), the Price Service uses optimistic concurrency control: each write includes the version number it expects to be overwriting, and the database update is conditional on that version still matching. If two writers race, the second writer’s conditional update fails, and that writer’s request is retried against the now-current version. This keeps the common case (no contention) fast while still preventing a rare race condition from silently corrupting the price.

13.2 Why we do not need distributed consensus here

It is worth explicitly noting in an interview that this system does not need a distributed consensus protocol like Raft or Paxos for price writes themselves, because all writes for a given product are funneled through a single logical owner — the Price Service backed by one primary database. Consensus protocols matter when multiple independent nodes must agree on a single value without a designated leader; here, the primary database already plays that leader role, and its own internal replication (leader-follower with synchronous acknowledgment from at least one replica) is sufficient to guarantee no committed write is ever lost.

13.3 Failure recovery walkthrough

Consider a concrete failure: the Cache Invalidation Service crashes halfway through processing a batch of 200 price-change events. Because Kafka only advances a consumer’s committed offset after successful processing, every event in that batch that had not yet been fully processed is redelivered to a newly started instance of the service after it recovers. The idempotency guarantees discussed earlier mean this redelivery is completely safe — some events might be processed twice, but never incorrectly, and no event is ever silently dropped. This replay-on-restart behavior is precisely why we invest in making every consumer idempotent up front, rather than treating it as an edge case to handle later.

13.4 Networking considerations

The CDN purge API calls, WebSocket pushes, and cross-region database replication all depend on network reliability that engineers cannot fully control. To stay resilient, every outbound network call in this pipeline uses sensible timeouts (so a slow CDN API does not block the invalidation consumer indefinitely), exponential backoff with jitter on retries (so a brief network blip does not turn into a synchronized retry storm across thousands of parallel workers), and connection pooling to avoid the overhead of establishing a fresh TCP and TLS handshake for every single outbound request.

14.1 Primary database choice

A relational database like PostgreSQL is a strong default for the price-of-record table because price changes need ACID transactional guarantees (you never want a half-applied price update) and because CDC tooling like Debezium has first-class, mature support for Postgres’s write-ahead log.

14.2 Caching layers, and why we need more than one

It is tempting to think “just use Redis” and be done, but real systems layer multiple caches because each one optimizes for a different distance from the user:

The general rule: the closer a cache is to the user, the shorter its TTL should be, because it is the hardest one to actively purge quickly and reliably.

14.3 Load balancing strategy

The Load Balancer uses Layer 7 (application-aware) routing so it can make smarter decisions than pure round-robin — for example, routing all requests for a given product ID consistently to the same Page Render Service instance to maximize local render-cache hit rates, a technique called consistent hashing.

14.4 Database schema design

A minimal but production-honest schema for the price-of-record table looks like this:

CREATE TABLE product_price (
    product_id      BIGINT PRIMARY KEY,
    price_amount    NUMERIC(12, 2) NOT NULL,
    currency        CHAR(3) NOT NULL,
    version         BIGINT NOT NULL DEFAULT 1,
    updated_by      VARCHAR(64) NOT NULL,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE product_price_history (
    id              BIGSERIAL PRIMARY KEY,
    product_id      BIGINT NOT NULL,
    price_amount    NUMERIC(12, 2) NOT NULL,
    currency        CHAR(3) NOT NULL,
    version         BIGINT NOT NULL,
    updated_by      VARCHAR(64) NOT NULL,
    updated_at      TIMESTAMPTZ NOT NULL
);

Separating a small, hot product_price table from an append-only product_price_history table matters for two reasons: it keeps the row that every read query touches small and cache-friendly, and it gives us a complete, queryable audit trail for compliance and dispute resolution without bloating the hot path. The version column is what powers both the optimistic concurrency control discussed earlier and the staleness checks every downstream consumer performs.

14.5 Consistent hashing, explained simply

Imagine arranging all your backend servers around a circular dial, and also placing every product ID on that same dial using a hash function. Each request for a product is routed to the next server clockwise from its position on the dial. The benefit is that when you add or remove a server, only a small slice of the dial’s traffic needs to move to a new server — unlike naive hashing, where changing the server count reshuffles almost all keys at once. This keeps render-cache locality stable even as the Page Render Service fleet scales up and down with traffic.

14.6 Cache invalidation strategies compared

Each service exposes a narrow, well-defined API. Here is a simplified Price Service write endpoint in Java (Spring Boot style):

@RestController
@RequestMapping("/api/v1/products")
public class PriceController {

    private final PriceService priceService;

    public PriceController(PriceService priceService) {
        this.priceService = priceService;
    }

    @PutMapping("/{productId}/price")
    public ResponseEntity<PriceUpdateResponse> updatePrice(
            @PathVariable String productId,
            @RequestBody @Valid PriceUpdateRequest request) {

        PriceUpdateResult result = priceService.updatePrice(
                productId,
                request.getNewPrice(),
                request.getCurrency(),
                request.getUpdatedBy()
        );

        return ResponseEntity.ok(new PriceUpdateResponse(
                result.getProductId(),
                result.getNewPrice(),
                result.getVersion(),
                result.getCommittedAt()
        ));
    }
}

And the consumer side, listening for price-change events and invalidating the Redis cache:

@Component
public class PriceChangeConsumer {

    private final RedisTemplate<String, String> redisTemplate;
    private final CdnPurgeClient cdnPurgeClient;

    @KafkaListener(topics = "price-change-events", groupId = "cache-invalidation-service")
    public void handlePriceChange(PriceChangedEvent event) {

        String cacheKey = "product:price:" + event.getProductId();
        Long cachedVersion = getCachedVersion(cacheKey);

        if (cachedVersion != null && cachedVersion >= event.getVersion()) {
            return; // stale or duplicate event, skip
        }

        redisTemplate.delete(cacheKey);
        cdnPurgeClient.purge("/products/" + event.getProductId());
    }
}

Notice the version check before invalidating — this is the guard that prevents an out-of-order or duplicate Kafka delivery from undoing a more recent update, which is a critical correctness detail interviewers specifically look for.

Here is a simplified batching CDN purge client that groups individual purge requests together to respect the CDN provider’s rate limits:

@Component
public class CdnPurgeClient {

    private final Queue<String> pendingPaths = new ConcurrentLinkedQueue<>();
    private final CdnApiHttpClient httpClient;

    public void purge(String path) {
        pendingPaths.add(path);
    }

    @Scheduled(fixedDelay = 500)
    public void flushBatch() {
        List<String> batch = new ArrayList<>();
        String path;
        while ((path = pendingPaths.poll()) != null && batch.size() < 500) {
            batch.add(path);
        }
        if (!batch.isEmpty()) {
            httpClient.purgeUrls(batch);
        }
    }
}

And the Notification Gateway, pushing a live update to any browser tab currently subscribed to that product:

@Component
public class PriceNotificationConsumer {

    private final WebSocketSessionRegistry sessionRegistry;

    @KafkaListener(topics = "price-change-events", groupId = "notification-gateway")
    public void handlePriceChange(PriceChangedEvent event) {

        List<WebSocketSession> subscribers =
                sessionRegistry.getSubscribers(event.getProductId());

        PricePushMessage message = new PricePushMessage(
                event.getProductId(), event.getNewPrice(), event.getVersion());

        for (WebSocketSession session : subscribers) {
            session.sendAsync(message);
        }
    }
}

15.1 Why microservices here, specifically

Splitting Price Service, Cache Invalidation Service, Page Render Service, and Notification Gateway into separate deployable units means each can be scaled, deployed, and even written in a different language, independently — a traffic spike in page rendering does not force the invalidation service to also scale, and a bug in the notification path can be rolled back without touching price writes at all.

15.2 API contract details

A well-designed price-update endpoint does not just accept a new number; it needs enough context to be safe and auditable. The request payload typically includes the expected current version (for optimistic concurrency), the new price and currency, an identifier for who or what initiated the change, and an optional reason code (manual correction, automated repricing, promotional event) that feeds directly into the audit log and anomaly detection discussed earlier. The response, in turn, always echoes back the authoritative committed version and timestamp, which is what powers the read-your-writes guarantee for the admin dashboard.

15.3 Testing strategy

Because correctness here spans multiple services and an asynchronous event pipeline, unit tests alone are not enough. A thorough strategy layers four kinds of tests: unit tests for pure logic like version comparison and idempotency checks; integration tests that spin up a real (or embedded) Kafka broker and database to verify a price write genuinely produces the expected sequence of downstream events; contract tests that ensure the event schema published by the Price Service and the schema expected by each consumer never silently drift apart as services evolve independently; and end-to-end synthetic tests, as described earlier, that continuously measure real propagation latency in production rather than only in a staging environment.

16.1 Compensating actions for partial failures

Occasionally a price change needs to be reversed shortly after it was applied — for example, a merchandiser fat-fingers a decimal point and a ₹4,999 item briefly shows as ₹49.99. Rather than building special-case “undo” logic, the correct fix is simply to issue a new, corrected price change through the exact same pipeline, with a higher version number, exactly as described in the FAQ. This is a lightweight relative of the saga pattern used in distributed transactions: instead of trying to roll back a distributed operation that has already partially propagated, you move forward with a new compensating operation that supersedes it. It is a simpler and more robust mental model than trying to “undo” propagation that may already be partially complete across dozens of CDN edge nodes, some of which may not even be reachable for a true rollback at that exact moment.

16.2 Anti-patterns to avoid

17.1 Best practices

17.2 Common mistakes

To tie every component together, let us walk through a single realistic scenario end to end, the way you might narrate it on a whiteboard in an interview.

Walking through a concrete timeline like this in an interview does two things: it proves you understand how the individual components actually interact under real conditions, and it naturally surfaces the reasoning behind design decisions — like pre-warming ahead of a known event, or prioritizing purge calls by traffic — that might otherwise sound arbitrary if listed as disconnected bullet points.

19.1 Alternative Designs Considered, and Why They Were Rejected

A strong system design answer does not just present one solution — it shows you evaluated and consciously rejected reasonable alternatives. Here are three that commonly come up.

19.1.1 Polling-based invalidation

Instead of an event-driven pipeline, each cache layer could periodically poll the database for products whose updated_at timestamp is newer than the last time it checked. This is simpler to build initially, since it needs no Kafka cluster or CDC connector. It was rejected here because its best-case latency is bounded by the poll interval (polling every 5 seconds means up to 5 seconds of guaranteed staleness even in the best case, not the worst case), and because it wastes significant database load continuously scanning for changes across millions of products, the overwhelming majority of which have not changed since the last poll.

19.1.2 Client-side short polling for price refresh

Another option is having the browser itself poll a lightweight price-check endpoint every few seconds while a product page is open, rather than using a WebSocket push. This avoids the complexity of maintaining persistent connections, but was rejected as the primary mechanism because it multiplies request volume by every open tab times the poll frequency, most of which return “no change,” making it far less efficient than a push-based model where the server only sends data when something actually changed. Some platforms do use light client-side polling as a fallback for clients that cannot maintain a WebSocket connection, but it is deliberately not the primary channel.

19.1.3 No caching, database-backed pages only

The simplest possible design serves every page view with a direct database read, guaranteeing perfect consistency with zero propagation delay at all. This was rejected purely on scalability grounds: a popular product page can receive tens of thousands of requests per second during a sale, and routing all of that directly at the primary database would require a database fleet sized for worst-case peak load around the clock, which is both technically risky and financially wasteful compared to a caching architecture sized for typical load with headroom for bursts.

19.2 Internationalization and Regional Considerations

Global platforms add another dimension of complexity worth mentioning briefly: the same product might have genuinely different prices, currencies, and even different active promotions in different countries, driven by local market conditions, import duties, and regional pricing strategy. This means the cache key for a rendered price fragment typically needs to include not just the product ID but also the customer’s detected region or storefront, and a single “price change” event from the pricing team might actually fan out into several region-specific sub-events, each independently propagated through the same pipeline described throughout this tutorial. The architecture does not need to fundamentally change to support this — it simply means the granularity of what counts as “a price” is a composite of product and region, rather than product alone.

20.1 Glossary of key terms

20.2 Frequently asked questions

20.3 Putting it all together

Stepping back from the individual components, this design ultimately rests on a small number of ideas working together rather than any single clever trick. A single authoritative write path through the Price Service, backed by Change Data Capture, guarantees that no price change is ever silently lost, regardless of what happens downstream. An event backbone decouples that single source of truth from an arbitrary number of independent consumers — cache, CDN, search, real-time notifications — each of which can evolve, fail, and recover on its own schedule without taking the others down with it. Layered caching, from Redis close to the application through to CDN edges close to the user, absorbs the overwhelming majority of read traffic so that the primary database only ever has to handle a small, steady trickle of requests even during the largest sales events. And a deliberate, explicit split between eventually-consistent browsing and strongly-consistent checkout means the system can be fast and cheap where staleness is harmless, while remaining strict and safe at the one point where it genuinely matters. None of these ideas is unique to pricing — the same pattern of CDC plus event backbone plus layered caching plus a strongly consistent point-of-no-return step shows up repeatedly across large-scale systems handling inventory counts, seat availability, and account balances, which is exactly why this particular problem is such a popular and durable interview question.