Every online marketplace is, underneath its polished storefront, a giant distributed database of prices that anyone with seller access can change at will. That flexibility is a feature — it lets a small business run a flash sale in minutes — but it is also a loaded gun pointed at the company’s own bank account. A single misplaced decimal point, a broken pricing script, or a hijacked seller account can turn a $499 laptop into a $4.99 laptop, and the internet moves faster than any human review process ever could.

This class of problem is not new. It predates e-commerce itself: department stores have always had to train cashiers to double-check a tag that read “$5” when the intended price was “$50.” What changed with the internet is blast radius and speed. A mispriced tag in a physical store might fool a few dozen shoppers before an employee notices. A mispriced listing on a global marketplace can be discovered by deal-hunting communities, shared across Reddit, Telegram, and deal-aggregator sites, and exhausted in inventory within minutes — often before the pricing team has finished their morning coffee.

The earliest large marketplaces treated this purely as a customer service and legal problem: a mispriced item would sell, orders would go out, and the company would either honor the price (as a goodwill and reputational move) or cancel the orders (risking backlash and, in some jurisdictions, legal exposure for “bait and switch” claims). As transaction volumes grew into the millions per day, this reactive approach became financially untenable. A single well-publicized pricing error on a popular electronics item has, in real incidents across the industry, cost sellers and platforms losses ranging from thousands to millions of dollars in a matter of hours.

This gave rise to a dedicated engineering discipline sometimes called pricing integrity or catalog trust and safety. Rather than treating anomalous prices as an unavoidable cost of doing business, large marketplaces (Amazon, Walmart, eBay, Flipkart, Shopify-powered stores, and airline/travel booking systems) built dedicated systems that watch every price change in near real time, compare it against statistical and historical baselines, and — when something looks wrong — intervene before the price becomes visible to buyers or before checkout is allowed to complete.

What makes this system design problem particularly interesting is that it sits at the intersection of several classic system design domains: it is a real-time streaming and event-processing problem (prices change continuously and must be scored within milliseconds), a machine learning and statistics problem (what counts as “anomalous” depends on product category, seasonality, and historical distribution), a distributed systems consistency problem (you must freeze a price update across many read replicas and caches, fast, without breaking the shopping experience for everyone else), and a human-in-the-loop workflow problem (someone eventually has to look at the flagged case and decide: genuine error, or attack, or actually fine).

1.1 How the industry’s thinking evolved

It is worth tracing the evolution of thinking here, because the same progression tends to repeat inside every company that eventually builds a mature version of this system. The first generation of protection was almost always a spreadsheet: a pricing analyst manually scanning a list of the day’s biggest price drops, usually the morning after the damage was already done. This “generation zero” approach is cheap to build and genuinely fine for a marketplace with a few thousand listings and modest traffic, but it does not survive contact with real internet-scale demand, where a viral mispricing can be fully exhausted from inventory within single-digit minutes of being discovered by a deal community.

The second generation introduced simple, hard-coded rules directly in the pricing pipeline — usually something like “reject any price update more than 70% below the previous price.” This buys real protection against the most obvious accidents, but it is brittle in both directions: it blocks legitimate steep discounts on clearance and seasonal inventory (a false positive that costs real revenue and annoys honest sellers), and it is trivially easy for a deliberate attacker to reverse-engineer and stay just underneath the threshold (a false negative that defeats the entire point of the control).

The third and current generation, which is the subject of this tutorial, treats pricing integrity as a full-fledged, standing engineering discipline with its own dedicated team, its own service-level objectives, and its own continuously retrained models — the same organizational maturity level that most companies eventually give to payment fraud, account security, and content moderation. This tutorial builds that third-generation system from first principles, one architectural decision at a time, so that by the end you can both defend every design choice in an interview and reason about the trade-offs if you were building this for real.

Let’s define the problem precisely, the way you would in the first five minutes of a system design interview or a real design document.

2.1 The core scenario

A marketplace has millions of active product listings. Sellers (which can be individual merchants, large brands, or the marketplace’s own first-party retail arm) update prices constantly — for promotions, competitive repricing, inventory clearance, or simple corrections. We need a system that:

• Observes every price change as it happens, across every seller and every listing.
• Decides, within a very tight time budget, whether the new price is plausible given everything the system knows about that product, that seller, and the broader category.
• When a price looks implausible, prevents customer-facing harm — either by holding the price change before it goes live, or by freezing checkout on an already-live listing — without meaningfully slowing down the 99.9% of price changes that are completely legitimate.
• Routes suspicious cases to a human reviewer (or an automated secondary check) with enough context to make a fast, confident decision.
• Learns from every past decision to get better at telling genuine errors and attacks apart from legitimate steep discounts (which are common and should never be blocked).

2.2 Two very different root causes, one symptom

It is worth being explicit that “anomalously low price” has (at least) two very different underlying causes, and a good system needs to reason about both:

The detection signals for these two categories overlap heavily (both produce a price that deviates strongly from history), but the appropriate response can differ: a genuine error from a trusted, long-tenured seller might get a softer, faster self-service correction flow, while a suspicious pattern combined with recent account changes (new login device, password reset, sudden bulk price drops across unrelated SKUs) should escalate straight to a security/fraud review path, not just a pricing-ops queue.

2.3 Why naive solutions fail

2.4 Quantifying the cost of doing nothing

It helps to put real numbers on the problem, even rough ones, because that’s exactly what you’d be expected to reason through out loud in an interview or a design review. Imagine a mid-sized marketplace with a catalog average order rate of a few hundred orders per minute across all listings, and imagine one popular SKU normally priced at $600 gets mistakenly listed at $6. If that mispricing stays visible for just ten minutes before detection and enforcement, and word spreads quickly through a deal-alert community or a browser extension that scans for price drops, it is entirely plausible for several hundred to several thousand orders to be placed against that single listing before anyone intervenes. At even a conservative one thousand orders, the gap between the intended and actual price alone represents roughly $594,000 of exposure on that one SKU, before accounting for the operational cost of processing refunds or honoring the erroneous price, and before accounting for the reputational cost of either angering customers with a bulk order cancellation or overpaying for a viral mistake. This single scenario is the entire economic justification for the system this tutorial designs: shrinking the detection-to-enforcement window from double-digit minutes down to a few hundred milliseconds is not a nice-to-have performance target, it is the single biggest lever available for reducing this category of loss.

2.5 Who is harmed, and how, when this system does not exist

It is easy to think of pricing errors as a victimless mistake that mostly costs the company money, but the harm is broader than that. Genuine customers who happen to purchase during the exposure window are frequently caught in the middle: some marketplaces honor the erroneous price as a goodwill gesture and absorb the loss, while others cancel the orders, which understandably frustrates buyers who did nothing wrong and simply purchased an item at the price the platform itself displayed. Honest sellers are harmed too, since a viral mispricing on one listing can flood a fulfillment center’s outbound capacity, delaying shipments for unrelated orders. And the marketplace’s own trust and safety reputation takes a hit each time an incident becomes a public story, which is precisely the kind of headline risk that makes leadership willing to fund a dedicated pricing-integrity engineering team in the first place.

Before we draw any boxes and arrows, let’s build a shared vocabulary. Every term below is explained the way you would explain it to someone who has never designed a detection system before.

3.1 Anomaly detection

What: The general practice of identifying data points that differ significantly from the pattern established by the rest of the data. Why: Because most fraud, errors, and attacks look statistically different from “business as usual,” even if we’ve never seen that exact bad pattern before.

3.2 Baseline (or reference price)

What: A statistically derived “normal” price range for a product, built from its own price history, category peers, and possibly external signals like competitor pricing. Why: You cannot judge “anomalous” without something to compare against.

3.3 Z-score / standard-deviation-based scoring

What: A statistical technique that measures how many standard deviations a data point is from the mean of a distribution. Why: It gives you a single normalized number that says “how weird is this,” regardless of the product’s absolute price scale — a $2 drop on a $10 item and a $200 drop on a $1,000 item can both normalize to the same z-score. Example: If the mean price is $500 with a standard deviation of $40, a new price of $100 has a z-score of (100−500)/40 = −10, an extremely strong outlier signal.

3.4 Feature store

What: A centralized system that computes, stores, and serves the numerical “features” (inputs) used by a scoring model or rules engine — things like “30-day median price,” “seller’s historical error rate,” “category volatility index.” Why: Because computing these features fresh for every single price check would be far too slow; a feature store pre-computes and caches them so scoring can happen in milliseconds.

3.5 Rules engine vs. ML model

What: A rules engine applies explicit, human-written conditional logic (“if price drops more than 80% AND seller account age < 30 days, flag it”). A machine learning model instead learns statistical patterns from historical labeled data (past confirmed errors, attacks, and legitimate discounts) and outputs a continuous risk score. Why both: Rules are transparent, auditable, and fast to deploy for known bad patterns; ML models catch subtler, previously-unseen patterns but need training data and are harder to explain to a human reviewer. Production systems almost always use both together.

3.6 Circuit breaker / kill switch

What: A mechanism that instantly halts a specific action across the system when a dangerous condition is detected — in this context, freezing checkout for a specific listing or even an entire seller’s catalog. Why: Detection is only useful if paired with an equally fast enforcement mechanism.

3.7 Human-in-the-loop (HITL)

What: A workflow design where automated systems handle the fast, high-confidence decisions, but route ambiguous or high-stakes cases to a trained human for final judgment. Why: No automated model achieves perfect accuracy, and the cost of a wrong automated decision here (blocking a legitimate flash sale, or letting a real attack through) is high enough to justify human review for the gray-zone cases.

3.8 Idempotency

What: A property of an operation such that performing it multiple times has the same effect as performing it once. Why it matters here: Price update events can be retried or duplicated by distributed messaging systems; the pricing pipeline must be built so that reprocessing the same price-update event twice does not double-apply a freeze or double-publish a price.

3.9 CAP theorem, applied to this specific system

What: The CAP theorem states that a distributed data system can provide at most two of three guarantees simultaneously during a network partition: consistency (every read sees the latest write), availability (every request gets a response), and partition tolerance (the system keeps working despite network failures between nodes). Why it matters here: This system actually needs different CAP trade-offs in different places, which is a genuinely interesting design nuance. The “frozen listing” flag, checked at checkout, needs to lean toward consistency — it is far better for a checkout request to fail or retry than to succeed against a price that should have been frozen, so that specific check should be backed by a strongly consistent store even at some availability cost. The storefront’s general catalog browsing experience, by contrast, should lean toward availability — it is perfectly acceptable for a shopper browsing a product page to see a price that is a few seconds stale, since browsing a stale-but-plausible price causes far less harm than the storefront going down entirely.

3.10 Consensus and coordination

What: Consensus algorithms (such as Raft or Paxos, or systems built on top of them like a coordination service) allow a cluster of machines to agree on a single, authoritative value even when some nodes fail or messages are delayed. Why it matters here: The catalog database’s primary-replica setup and the coordination service backing the strongly-consistent “frozen” flag store both rely on consensus internally to guarantee that all healthy nodes agree on which node is currently the write leader, preventing a dangerous split-brain scenario where two nodes both believe they are authoritative and accept conflicting writes.

3.11 Partitioning and sharding strategy

What: Partitioning (or sharding) splits a large dataset or event stream across multiple independent nodes, each responsible for a subset of the keyspace, so that no single node needs to hold or process the entire dataset. Why it matters here: Both the event bus and the catalog database are partitioned by a consistent key — typically SKU or seller ID — which guarantees that all events and rows for a given product land on the same partition and are therefore processed in a strict, predictable order relative to each other, which is essential since the scoring logic needs to know the immediately preceding price to compute a meaningful deviation.

3.12 Concurrency control

What: Concurrency control covers the techniques used to let multiple operations happen at once on shared data without corrupting it — optimistic concurrency (check a version number before committing, retry on conflict) and pessimistic concurrency (lock the row before modifying it) are the two dominant strategies. Why it matters here: Two systems can legitimately race to modify the same listing’s state at nearly the same instant — a seller submitting a new price at the exact moment the Anomaly Detection Service is applying a freeze from a slightly earlier event. An optimistic concurrency approach, where every price row carries a version number and a write is rejected and retried if the version has moved since it was read, avoids the throughput penalty of row-level locking while still preventing the two writers from silently overwriting each other’s intent.

3.13 Networking considerations

What: The choice of network protocol and connection pattern between services directly shapes the system’s latency budget. Why it matters here: Internal service-to-service calls on the hot scoring path (Anomaly Detection Service to Feature Store, Feature Store to its cache layer) typically use a binary RPC protocol such as gRPC over persistent, pooled connections rather than opening a fresh HTTP connection per request, since connection setup overhead alone can consume a meaningful fraction of the entire sub-500-millisecond scoring budget if done naively.

Now let’s assemble the full system. Every box below is explicitly labeled with the infrastructure role it plays, exactly as you’d want to narrate it on a whiteboard.

4.1 Component walkthrough

Let’s go one level deeper into how the Anomaly Detection Service and Decision Engine actually compute a verdict, since that’s the intellectual core of this system.

5.1 Two-tier scoring: fast path and deep path

A production-grade system almost never relies on a single scoring pass. Instead, it typically layers two tiers:

1. Fast path (synchronous, sub-50ms): A lightweight, rule-based check embedded directly in the Price Service’s write path. This catches only the most blatant, unambiguous anomalies (e.g., price is literally zero or negative, price dropped more than 95% with no active promotion flag) and can immediately reject or soft-hold the update before it is even persisted as “live.”
2. Deep path (asynchronous, sub-second to a few seconds): The full pipeline described in Section 04 — feature lookup, statistical scoring, ML model inference, and policy-based decisioning. This runs on every price change, including ones the fast path already approved, because some attacks only become visible when combined with contextual signals (seller account age, recent password reset, unusual bulk-update pattern) that are too slow to check synchronously.

This two-tier design is the same pattern used by payment fraud systems: an extremely fast rule layer blocks the obvious cases immediately, while a slower, richer model catches everything else shortly after, with a hold mechanism bridging the gap.

5.2 Anatomy of the anomaly score

The Decision Engine typically combines several independently computed signals into one composite risk score:

5.3 Combining rules and ML

A common production pattern is a score-then-gate architecture: the ML model outputs a continuous probability (0 to 1) that this price change is anomalous, and a rules layer on top applies hard business overrides that no model score can bypass — for example, “always hold if price is exactly $0.00, regardless of model confidence” or “never auto-hold if the seller is on the verified first-party retail allowlist and the SKU has an active, pre-approved campaign ID.” This gives you the flexibility of ML with the auditability and safety guarantees of explicit rules.

5.4 A minimal fast-path rule check in Java

public class FastPathPriceGuard {

    private final BaselineCache baselineCache;

    public FastPathPriceGuard(BaselineCache baselineCache) {
        this.baselineCache = baselineCache;
    }

    // Returns blocked/deferred without any network call -- pure cache + arithmetic.
    public FastPathResult evaluate(PriceChangeRequest request) {
        if (request.getNewPrice().compareTo(BigDecimal.ZERO) <= 0) {
            return FastPathResult.blocked("Price must be positive");
        }

        PriceBaseline baseline = baselineCache.get(request.getSku());
        if (baseline == null || baseline.getSampleCount() < 5) {
            // Not enough history to judge -- defer to the deep path.
            return FastPathResult.deferToDeepPath();
        }

        BigDecimal dropRatio = request.getNewPrice()
                .divide(baseline.getMedianPrice(), 6, RoundingMode.HALF_UP);

        boolean extremeDrop = dropRatio.compareTo(BigDecimal.valueOf(0.05)) < 0;
        boolean hasActivePromotion = request.isPartOfApprovedCampaign();

        if (extremeDrop && !hasActivePromotion) {
            return FastPathResult.blocked(
                "Price dropped below 5% of 90-day median with no approved campaign");
        }

        return FastPathResult.deferToDeepPath();
    }
}

Notice this fast-path check deliberately does very little: no network calls to the feature store, no ML inference — just a cache lookup and arithmetic. This is intentional; it must run inline on the write path in well under 50 milliseconds so it never becomes the bottleneck for the 99.9% of legitimate price updates.

5.5 Deep-path composite scoring in Java

public class AnomalyScorer {

    private final FeatureStoreClient featureStore;
    private final MlModelClient mlModel;

    public AnomalyScorer(FeatureStoreClient featureStore, MlModelClient mlModel) {
        this.featureStore = featureStore;
        this.mlModel = mlModel;
    }

    public RiskScore score(PriceChangedEvent event) {
        FeatureVector features = featureStore.fetch(
            event.getSku(), event.getSellerId(), event.getCategoryId());

        double zScore = computeZScore(event.getNewPrice(), features.getBaseline());
        double reputationPenalty = 1.0 - features.getSellerTrustScore();
        double velocityFactor = features.getRecentChangeVelocity() > 20 ? 1.3 : 1.0;

        double mlProbability = mlModel.predictAnomalyProbability(features, zScore);

        double compositeScore = (0.5 * mlProbability
                + 0.3 * normalize(zScore)
                + 0.2 * reputationPenalty) * velocityFactor;

        if (features.hasApprovedCampaign()) {
            compositeScore *= 0.2; // heavily discount score for pre-approved promotions
        }

        return new RiskScore(compositeScore, buildExplanation(zScore, mlProbability, features));
    }

    private double computeZScore(BigDecimal price, PriceBaseline baseline) {
        double mean = baseline.getMean();
        double stdDev = Math.max(baseline.getStdDev(), 0.01);
        return (price.doubleValue() - mean) / stdDev;
    }

    private double normalize(double z) {
        return 1.0 / (1.0 + Math.exp(z + 4)); // sigmoid centered near z = -4
    }

    private String buildExplanation(double z, double mlProb, FeatureVector f) {
        return String.format(
            "z=%.2f, mlProb=%.2f, sellerTrust=%.2f, velocity=%d",
            z, mlProb, f.getSellerTrustScore(), f.getRecentChangeVelocity());
    }
}

Two things are worth calling out in this snippet. First, every score comes with an explanation string — this is not decoration; a human reviewer in the Review Queue Service needs to understand why the system flagged a listing, and regulators or internal auditors may later require it. Second, the approved-campaign discount is applied as a late multiplicative adjustment rather than baked into the model itself, which keeps the override auditable and easy to reason about independently of the ML model’s internals.

5.6 Algorithms and data structures worth knowing for this system

A handful of classic algorithms and data structures do most of the heavy lifting inside a production-grade version of this pipeline, and being able to name them and explain why they fit is exactly the kind of depth that separates a good system design answer from a great one.

• Sliding window counters for velocity features — tracking “how many price changes has this seller made in the last five minutes” efficiently, without re-scanning full history on every event.
• HyperLogLog or Count-Min Sketch for approximate, memory-efficient counting at very high cardinality — useful when tracking distinct buyers attempting checkout on a flagged listing across a fleet of stateless workers, where an exact count would require expensive coordination.
• Priority queues / heaps for ordering the human Review Queue by urgency (a composite of risk score and potential financial exposure), so the highest-stakes cases surface to analysts first rather than being processed strictly first-in-first-out.
• Bloom filters as a fast, space-efficient pre-check for “has this SKU ever had a confirmed incident before,” avoiding an expensive database lookup for the common case of a clean product history.
• Exponentially weighted moving averages (EWMA) for keeping rolling baselines fresh without storing and reprocessing the entire raw price history on every update — recent prices are weighted more heavily than older ones, letting the baseline adapt gradually to genuine, sustained shifts (like a permanent cost-of-goods change) while still resisting a single-event spike.

5.7 A sliding-window velocity tracker in Java

public class SlidingWindowVelocityTracker {

    private final Duration windowSize;
    private final Deque<Instant> timestamps = new ArrayDeque<>();

    public SlidingWindowVelocityTracker(Duration windowSize) {
        this.windowSize = windowSize;
    }

    // Records a new price-change event and returns the current
    // number of changes still inside the trailing window.
    public synchronized int recordAndCount(Instant eventTime) {
        timestamps.addLast(eventTime);
        Instant cutoff = eventTime.minus(windowSize);

        // Evict anything that has aged out of the window --
        // amortized O(1) per call since each timestamp is removed exactly once.
        while (!timestamps.isEmpty() && timestamps.peekFirst().isBefore(cutoff)) {
            timestamps.pollFirst();
        }

        return timestamps.size();
    }
}

This tracker underpins the “velocity factor” used in the composite scoring example above. Its key property is amortized constant-time cost per event: each timestamp is inserted once and evicted once, so even under sustained high throughput the tracker never needs to rescan its full history to answer “how many events happened recently.” In a real deployment, one instance of this tracker exists per seller, typically backed by a distributed cache rather than in-process memory, so the count stays accurate across a horizontally scaled fleet of stateless scoring workers rather than being fragmented per instance.

Let’s trace a single listing through its full lifecycle, from a healthy price to a flagged, held, and resolved anomaly.

6.1 Why the “re-evaluated” loop matters

A subtle but important design detail: some anomalies are not statistically obvious from the price alone. A price drop of 40% might sit comfortably inside normal variance for a given category, pass all scoring, and go live — and only reveal itself as a problem once checkout traffic on that SKU spikes 50x above its normal rate within minutes. This is why the Decision Engine should not be a one-shot gate; it should continue consuming a lightweight stream of post-publish signals (order velocity, cart-add velocity) and be able to re-trigger a freeze even on an already-published listing.

6.2 Step-by-step walkthrough

1. Submission: A seller (or an internal repricing bot) submits a new price through the API Gateway.
2. Fast-path check: The Price Service applies the inline rule check described in Section 05.
3. Persistence and event emission: If it passes, the price is written to the catalog database in a “pending” state and an event is published to the event bus — this is the moment the write becomes durable and visible to downstream consumers, but not yet to buyers.
4. Feature enrichment: The Anomaly Detection Service consumes the event and enriches it with features pulled from the feature store (which itself is kept fresh by a separate streaming aggregation job continuously recomputing rolling statistics).
5. Scoring: The composite risk score is computed, combining statistical, reputational, and ML-derived signals.
6. Decision: The Decision Engine converts the score into an action: publish, hold, or escalate.
7. Enforcement: If held, the Listing Hold Service invalidates relevant caches and sets a “frozen” flag checked at both the catalog-read path and the checkout path.
8. Human review (if applicable): A trained analyst reviews the case in the Review Queue Service, sees the explanation string, price history chart, and seller context, and makes the final call.
9. Resolution and feedback: The human decision is logged and fed back as a labeled training example, continuously improving the ML model’s future accuracy.

7.1 Key trade-off: precision vs. recall vs. seller friction

This is the central tension of the entire system. Set thresholds too aggressively (favoring recall — catching every possible anomaly) and you generate excessive false positives, frustrating honest sellers running legitimate sales and creating an unmanageable human review backlog. Set thresholds too conservatively (favoring precision — only flagging near-certain anomalies) and genuine attacks slip through, causing real financial loss. There is no threshold that eliminates both error types simultaneously; the right operating point is a business decision informed by the relative cost of each error type, and it should differ by category (a $50 accessory tolerates more false positives than a $3,000 appliance).

7.2 Key trade-off: synchronous blocking vs. asynchronous scoring

Fully synchronous scoring (block the price write until the deep model finishes) guarantees no anomalous price is ever visible, but adds unacceptable latency to every single price update and creates a single point of failure — if the scoring service is slow or down, all pricing grinds to a halt. Fully asynchronous scoring (publish first, score after) keeps the write path fast and resilient, but opens a small window where an anomalous price could theoretically be visible or even purchasable before the hold kicks in. The two-tier fast-path/deep-path design in Section 05 is precisely the engineering answer to this trade-off — accept a small, well-bounded exposure window in exchange for both speed and thoroughness.

7.3 Key trade-off: centralized decision policy vs. per-category autonomy

A related tension worth naming explicitly is how much decision-making authority to centralize in one global Decision Engine versus delegating category-specific policy to the teams that actually understand each category best. A fully centralized policy is simpler to reason about, easier to audit, and avoids duplicated logic, but it inevitably becomes a bottleneck as different categories (fashion, electronics, groceries, big-ticket appliances) develop genuinely different pricing dynamics and risk tolerances that a single global rule set struggles to capture well. A fully decentralized model, where each category team owns its own thresholds and even its own scoring logic, adapts better to local nuance but risks inconsistent standards, duplicated engineering effort, and a much harder-to-audit overall system. The practical middle ground most mature systems land on is a shared, centrally-owned scoring engine and infrastructure, with category-specific configuration (thresholds, feature weights, campaign allowlists) exposed as data rather than code, so category teams get the autonomy they need without forking the underlying pipeline.

At the scale of a major marketplace, this system needs to comfortably absorb tens of millions of price events per day, with sharp bursts during major sale events (Black Friday, festival sales, flash promotions) where price-update volume can spike 10–50x above baseline in minutes.

8.1 Horizontal scaling strategy

• Event bus partitioning: Partition the Kafka topic by SKU or seller ID, so all events for a given product are processed in order by the same consumer instance — critical, because scoring a price change needs to know the immediately preceding price for that same SKU.
• Stateless scoring workers: The Anomaly Detection Service consumers should be entirely stateless, pulling all context from the feature store and cache layer, which allows adding more consumer instances during traffic spikes without any coordination overhead.
• Feature store read replicas: Since feature lookups happen on every single scoring call, the feature store needs to scale reads independently of writes, typically via a Redis cluster with read replicas or a purpose-built low-latency store like a wide-column database with in-memory caching in front.
• Backpressure and autoscaling: Consumer lag on the event bus is the primary scaling signal — when lag grows past a threshold, autoscaling adds more scoring worker instances; if scoring latency itself grows (not just queue depth), that signals a need to scale the feature store or ML inference tier instead.

8.2 Handling burst traffic during major sale events

Sale events are exactly when both event volume and legitimate steep discounts spike simultaneously — the worst possible combination, since the system must scale up while also being more permissive about large price drops for pre-approved campaigns. The standard approach is a campaign allowlist: marketing/pricing teams register upcoming sale campaigns in advance with approved SKUs and price floors, and the Decision Engine checks this allowlist before applying its normal statistical thresholds, dramatically reducing false positives during expected high-volume events while keeping full scrutiny on unplanned drops.

8.3 Latency budget breakdown

8.4 Capacity planning

Capacity planning for this system should be driven by peak, not average, load, since the whole point of the system is to remain fully effective during exactly the highest-traffic, highest-risk moments — a major sale event is simultaneously the highest-volume period and the period when a missed anomaly is most costly, since it coincides with the largest pool of buyers actively hunting for deals. A reasonable planning approach is to size the steady-state scoring worker fleet for typical daily peak load, then rely on autoscaling with a meaningful headroom buffer and pre-warmed capacity ahead of known, calendared sale events, rather than depending purely on reactive autoscaling to catch up during a traffic spike that can develop within seconds of a popular deal going viral. Load testing should explicitly simulate the worst historically observed burst pattern, not just a smooth ramp, since real burst traffic during a flash sale tends to arrive in a sudden step change rather than a gradual increase.

The single most important reliability principle in this system is: a scoring failure must never silently become a scoring skip. If the Anomaly Detection Service is down, degraded, or too slow, the system must fail toward safety (blocking or delaying the price change) rather than failing open (letting an unscored price go live).

9.1 Fail-safe design patterns

• Fail closed on scoring timeout: If deep-path scoring does not complete within its budget, the default action is to hold the listing pending manual review rather than auto-publish — this trades a small amount of seller friction for a much larger guarantee against catastrophic financial exposure.
• Dead letter queue with automated replay: Any event that fails processing (malformed schema, downstream dependency failure) goes to a DLQ rather than being dropped, with an automated replay worker retrying with backoff, and an alert firing if the DLQ depth exceeds a threshold.
• Multi-region redundancy: The event bus, feature store, and scoring service are deployed across multiple availability zones (and, for global marketplaces, multiple regions), so a single data center outage does not halt pricing integrity checks entirely.
• Graceful degradation tiers: If the ML model service is unavailable, the system falls back to pure statistical (z-score) rules rather than failing entirely — a degraded but still functional detection layer beats no detection layer.
• Circuit breakers on downstream calls: Calls from the Anomaly Detection Service to the feature store or ML model use circuit breakers, so a slow dependency doesn’t cascade into a full pipeline stall.

9.2 Consistency considerations

Enforcing a freeze consistently across a globally distributed, heavily cached storefront is genuinely hard. A “frozen” flag written to a primary database is not instantly visible to every edge cache and read replica worldwide. The practical answer is a layered defense:

1. Push the freeze flag proactively to the cache layer (rather than waiting for cache expiry) using a fast pub/sub invalidation mechanism.
2. Enforce a final, authoritative check at the moment of checkout/order placement against a low-latency, strongly-consistent source (even if the catalog browsing experience itself remains eventually consistent) — this is the true last line of defense, and it must never be skipped even if it adds a few milliseconds to checkout.
3. Accept that a small number of orders may occasionally slip through during the propagation window, and have an automated post-hoc order-cancellation workflow for the rare cases where enforcement lags behind detection.

9.3 Disaster recovery and backup strategy

Beyond day-to-day fault tolerance, the system needs a genuine disaster recovery plan for larger-scale failures — a full regional outage, a corrupted feature store, or a catastrophic bug that poisons the baseline data itself. The catalog database and audit log should have point-in-time recovery capability with a clearly defined recovery point objective and recovery time objective, agreed with the business based on how much data loss and downtime is tolerable for pricing data specifically, which is typically far less tolerant than, say, a marketing analytics dataset given the direct financial stakes involved. Baselines and feature store state, since they are derived data rather than a primary source of truth, can generally be fully rebuilt from the underlying event log given enough processing time, which is itself a strong argument for retaining that raw event history durably even after it has been consumed once — the ability to replay history from scratch is a powerful disaster recovery tool that a purely stateful, non-event-sourced design would not have.

Because this system exists partly to catch deliberate attacks, it must itself be resistant to being gamed, and it must integrate tightly with the broader account-security ecosystem.

10.1 Threats specific to this system

• Account takeover pricing attacks: An attacker who compromises a seller’s credentials can crash prices intentionally. The Decision Engine should weight recent security events (new device login, password reset, MFA disabled) heavily, since a price anomaly combined with a recent security event is a much stronger attack signal than either alone.
• Model probing / adversarial exploration: A sophisticated attacker could make small, incremental price changes to map out the exact threshold at which the system triggers a hold, then set a price just below that threshold. Mitigations include randomized jitter in thresholds, rate-limiting the number of price changes a single seller can make in a short window, and monitoring for the specific pattern of “many small experimental changes” as its own anomaly signal.
• Coupon/loyalty-point stacking attacks: Even a “normal-looking” listed price can be driven to near-zero at actual checkout by stacking abusable discount codes. The detection scope should include effective checkout price, not just listed catalog price.
• Insider threat: Someone with internal access to the pricing tools, review queue, or campaign-allowlist system could approve fraudulent prices. All approvals must be logged immutably, and campaign-allowlist entries should require dual approval for high-value SKUs.

10.2 Standard security practices applied here

• Least privilege: The Review Queue Service should grant analysts the minimum permissions needed — view context and approve/reject — never direct database write access to catalog prices.
• Immutable audit log: Every price change, score, decision, and human override is written to an append-only audit log, both for regulatory compliance and for post-incident forensics.
• API authentication and rate limiting: Enforced at the API Gateway, preventing brute-force probing of pricing endpoints and ensuring every actor is authenticated and authorized for the specific seller account they’re modifying.
• Encryption in transit and at rest: Standard TLS for all service-to-service and client-to-service communication; encryption at rest for the catalog database and audit logs.
• Secrets management: Credentials for the feature store, ML model service, and database connections are stored in a dedicated secrets manager, never in code or configuration files.

10.3 Compliance and regulatory considerations

Depending on the jurisdictions a marketplace operates in, consumer protection law may place real constraints on how pricing errors can be handled after the fact — some regions have “bait and switch” or false advertising statutes that limit a seller’s ability to simply cancel an order once a customer has paid a confirmed, checked-out price, regardless of whether the underlying listing was a genuine mistake. This means the enforcement policy layer discussed in Section 15 needs input from legal and compliance stakeholders, not just engineering and pricing operations, and the audit log described earlier in this section needs to be complete and reliable enough to serve as evidence in a regulatory inquiry or a customer dispute, showing exactly when a price was detected as anomalous, what action was taken, and by whom or what system the final decision was made.

10.4 Data privacy in the feature store

While most of the signals feeding this system are about products and seller accounts rather than individual buyers, the demand-shock and order-velocity signals do touch buyer behavior data, which means the feature store needs to respect the same data retention, minimization, and access-control policies that govern any other system touching customer purchase activity. A practical approach is to aggregate buyer-side signals (checkout attempt counts, unique buyer counts via an approximate structure like a Bloom filter or HyperLogLog) rather than retaining individually identifiable buyer records in the anomaly detection feature store at all, since the system generally does not need buyer-level identity to do its job — only aggregate behavioral shape.

A detection system that nobody is watching is just another silent point of failure. Observability here needs to answer two very different kinds of questions: “is the system technically healthy?” and “is the system doing its job well?”

11.1 Operational health metrics

• Event bus consumer lag per partition (early warning of scoring falling behind ingestion).
• p50/p95/p99 scoring latency, broken down by fast-path vs. deep-path.
• Feature store cache hit rate and read latency.
• Dead letter queue depth and replay success rate.
• ML model inference error rate and timeout rate.

11.2 Detection quality metrics

• False positive rate: Percentage of held listings later confirmed by a human reviewer to have been legitimate — the primary metric for seller-friction cost.
• False negative rate (via retrospective audit): Sampled review of published prices to catch anomalies the system missed, since these never generate a natural “held” signal to measure directly.
• Mean time to detection: Time from price submission to a decision being reached — critical for understanding the actual exposure window.
• Mean time to human resolution: How long flagged cases sit in the review queue before a human acts — long queue times mean anomalous listings stay frozen (hurting good sellers) or, worse, get auto-released by a queue-overflow policy.
• Financial exposure avoided (estimated): Modeled loss prevented, calculated as the price gap multiplied by likely order volume had the anomaly gone unchecked — a key metric for justifying the system’s ROI to leadership.

11.3 Alerting and dashboards

On-call engineers need real-time dashboards (built on the metrics service, typically Prometheus with Grafana, or an equivalent managed observability stack) showing consumer lag, scoring latency, and DLQ depth, with paging alerts on SLA breaches. Separately, pricing-ops leadership needs a daily/weekly dashboard on detection quality metrics — false positive rate trending up is just as important a signal as a latency spike, since it means the model or thresholds have drifted and need retuning.

11.4 Incident response playbook

Even a well-tuned detection system will occasionally miss something, and having a rehearsed incident response playbook matters as much as the detection pipeline itself. A mature playbook typically defines a small number of clearly named severity tiers — for instance, a single-listing miss affecting a handful of orders versus a systemic miss affecting many listings simultaneously, which is a much stronger signal of either a widespread seller-side tooling bug or a coordinated attack. The playbook should specify, in advance, who has authority to trigger an emergency catalog-wide freeze on a category or seller, what the rollback procedure looks like for a bad model deployment, and how the post-incident review feeds back into both the rules layer and the training data for the next model iteration. Treating this as a rehearsed, tested process — not an improvised scramble the first time it happens for real — is what separates teams that recover from an incident in minutes from teams that take hours.

This system is a natural fit for a cloud-native, containerized microservices deployment, given its need for independent scaling of very different workload shapes (stateless request handling vs. stream processing vs. ML inference).

12.1 Deployment topology

• Container orchestration: Each service (API Gateway, Price Service, Anomaly Detection Service, Listing Hold Service, Review Queue Service) runs as an independently deployable, horizontally scalable containerized workload, typically on a managed Kubernetes cluster, with separate node pools for CPU-bound scoring workers versus lightweight API services.
• Managed event streaming: The event bus is typically a managed Kafka service (or a cloud-native equivalent) rather than self-hosted, reducing operational burden for partition rebalancing, broker patching, and replication.
• Managed cache and feature store: A managed in-memory data store cluster (with cross-AZ replication) backs the cache layer and feature store’s low-latency tier.
• Blue-green or canary deployments for the ML model: New model versions are deployed to a small percentage of scoring traffic first, with automated comparison of decision distributions against the previous model version before a full rollout — a model that suddenly flags 5x more listings than its predecessor should block its own promotion to 100% traffic.
• Infrastructure as code: The entire topology (clusters, topics, cache clusters, IAM policies, autoscaling rules) is defined declaratively, enabling reproducible environments and safe, reviewable infrastructure changes.

12.2 Multi-region strategy

For a global marketplace, price scoring should happen in the same region as the originating write to minimize latency, with baselines and models replicated across regions asynchronously. A regional outage in the scoring tier should trigger the fail-closed behavior described in Section 09 for that region’s traffic, rather than attempting a cross-region synchronous failover that would add unacceptable latency.

12.3 Cost optimization

The most expensive components in this architecture, in rough order, are typically the always-on scoring worker fleet, the managed streaming cluster, and the in-memory cache/feature-store layer — none of which can simply be shut down outside business hours, since a marketplace never truly closes. Sensible cost levers include right-sizing the scoring worker fleet against actual measured throughput rather than provisioning for a theoretical worst case, using spot or preemptible compute capacity for the asynchronous batch jobs that recompute long-window baselines (since those jobs can tolerate occasional interruption and retry, unlike the latency-sensitive scoring path), and tiering the feature store so that only the hottest, most frequently accessed baselines live in the most expensive in-memory tier while colder, less-frequently-queried historical data sits in cheaper storage. It is also worth periodically auditing the ML model’s own inference cost against its marginal accuracy contribution — a smaller, cheaper model that captures most of the detection value at a fraction of the inference cost is often the right trade for the majority of traffic, reserving a heavier model only for cases the lightweight model itself flags as uncertain.

12.4 Rollback strategy

Because this system directly gates revenue-generating price changes, any deployment must have a fast, well-tested rollback path. Configuration changes (thresholds, campaign allowlist entries) should be versioned and revertible within seconds through the config service described in Best Practices. Code deployments to the scoring service should follow the canary pattern described above, with automated rollback triggered the moment the new version’s decision distribution or error rate deviates meaningfully from its predecessor, without waiting for a human to notice a dashboard anomaly first.

13.1 Catalog database

The system of record for current live prices should be a horizontally sharded relational or distributed database (sharded by product/seller ID), with primary-replica replication for read scaling on the storefront browsing path. Price writes are relatively low-volume compared to catalog reads (browsing traffic vastly exceeds price-change traffic), so read replicas are essential to keep the storefront fast without contending with the write-heavy pricing pipeline.

13.2 Feature store storage choice

Feature lookups need single-digit-millisecond latency at high read throughput, which points toward an in-memory key-value store (such as a Redis cluster) as the serving layer, backed by a separate durable store (a wide-column or time-series database) that holds the longer history from which rolling statistics are computed by a background streaming job. This separation — fast serving layer plus durable computation layer — is the standard feature store pattern used across recommendation and fraud systems generally, not just this one.

13.3 Caching strategy

13.4 Load balancing

A Layer 7 load balancer in front of the API Gateway distributes seller and internal traffic across gateway instances using health-checked round-robin or least-connections routing. Within the streaming tier, “load balancing” takes the form of Kafka partition assignment across consumer group members — as consumer instances scale up or down, partitions automatically rebalance across the available workers, which functionally serves the same purpose as a load balancer for the stream-processing workload.

13.5 Replication and read scaling in practice

The catalog database’s primary-replica topology deserves a concrete walkthrough, since it is a recurring building block across nearly every large-scale system, not just this one. Writes — price submissions and publish confirmations — always go to the primary node, which then asynchronously streams its write-ahead log to one or more replica nodes spread across availability zones. The storefront’s read-heavy browsing traffic is served almost entirely from replicas, which can be scaled out horizontally far more cheaply than the primary, since replicas do not need to coordinate with each other to serve a read. The trade-off, inherent to asynchronous replication, is replication lag — a replica might serve a price that is a few hundred milliseconds behind the primary’s true current value. For catalog browsing this lag is entirely acceptable, but it is precisely why the checkout-time enforcement check discussed in the High Availability section must read from the primary or from a strongly consistent store, never from an asynchronous replica, since that is the one read in the entire system where a stale answer could allow an anomalous price to be purchased.

This system is naturally decomposed into single-responsibility microservices communicating over well-defined APIs, with the event bus as the primary integration backbone between the write path and the scoring pipeline.

14.1 Representative API surface

14.2 Synchronous vs. asynchronous communication choices

The Price Service exposes a synchronous REST/gRPC API to sellers (they need an immediate response confirming their price update was accepted, even if final scoring is still pending). Internally, the Price Service to Anomaly Detection Service handoff is asynchronous via the event bus, for the decoupling and resilience reasons discussed earlier. The Listing Hold Service’s cache-invalidation call, by contrast, must be synchronous and fast, since checkout enforcement depends on it propagating before any order can complete.

14.3 Idempotency and retry semantics

Because the event bus can redeliver messages (at-least-once delivery semantics are standard for systems like Kafka), every consumer in this pipeline must be idempotent. The Anomaly Detection Service should key its processing on an event ID, so reprocessing the same price-change event twice does not double-count it in velocity features or re-trigger duplicate holds. Similarly, the Price Service’s price-submission endpoint should accept an idempotency key from the client, so a network retry from a seller’s system doesn’t create two conflicting price-change records.

14.4 API versioning and backward compatibility

Because thousands of independent third-party seller systems integrate against the Price Service’s public API, breaking changes are extremely costly to roll out — a seller’s automated repricing bot that suddenly starts receiving unexpected response fields or status codes can itself become a source of pricing errors. The standard approach is explicit API versioning in the URL path (as shown in the endpoint table above), additive-only changes within a version wherever possible, and a long, clearly communicated deprecation window before retiring an old version, with usage metrics tracked per seller so the platform team knows exactly who still depends on a version before turning it off.

14.5 Internal service contracts

Internally, the event schema published to the event bus (the price.updated event) is effectively a contract between the Price Service and every downstream consumer, including the Anomaly Detection Service, analytics pipelines, and audit logging. Changes to this schema should go through the same rigor as a public API change — using a schema registry with compatibility checks (ensuring new fields are optional and old fields are never repurposed) prevents a well-intentioned schema change in one team from silently breaking another team’s consumer.

15.1 Patterns applied in this design

15.2 Anti-patterns to avoid

16.1 Best practices

• Start in shadow mode. Before enforcing any holds, run the full pipeline logging what it would have done, and compare against actual historical incidents to calibrate thresholds without risking false positives on real sellers.
• Make thresholds configurable, not hardcoded. Pricing ops teams need to tune sensitivity per category without a code deployment — expose thresholds through a config service with an audit trail of changes.
• Design the review console around speed, not just correctness. A reviewer who has to click through five screens to see price history will process fewer cases per hour, leaving more listings frozen for longer — surface the price chart, seller context, and explanation on one screen.
• Close the feedback loop deliberately. Every human decision should automatically become a labeled training example; without this, the ML model’s accuracy plateaus and the team keeps solving the same false positives manually forever.
• Version and A/B test model changes. Never fully cut over a new model version without comparing its decision distribution against the previous version on live shadow traffic first.
• Build a synthetic anomaly test suite. Continuously inject known-anomalous synthetic events into a staging environment to verify the pipeline still catches them after every deployment — this is your regression test for detection quality, not just code correctness.

16.2 Common mistakes

• Building the ML model before building solid observability — teams often cannot debug why the model made a bad call because they never instrumented the pipeline to log the full feature vector alongside each decision.
• Ignoring seasonality — a baseline built from a 30-day rolling window will misfire spectacularly right after Black Friday or a major seasonal sale event, flagging the return-to-normal price as anomalous.
• Under-investing in the human review tooling because it “isn’t the interesting engineering problem” — in practice, review queue throughput is often the actual bottleneck limiting how aggressively the automated layer can be tuned.
• Not testing the checkout-time last-line-of-defense check under load — teams sometimes verify the catalog-level freeze works but never load-test the final guard check at high checkout concurrency, missing race conditions exactly when they matter most (a flash-sale event).
• Treating the system as “done” after the initial launch — thresholds, feature weights, and category groupings all drift out of calibration over months as the catalog mix, seller base, and buyer behavior evolve, and a system left untouched for too long quietly degrades into either an annoying false-positive machine or a leaky detector that misses real incidents.
• Optimizing purely for detection accuracy while ignoring reviewer experience — a model that is 2% more accurate but produces explanations a human cannot parse quickly in practice slows down the entire review pipeline and can net out as a worse system overall.
• Forgetting to plan for the cold-start problem when launching in a brand-new product category or geographic market with no historical pricing data at all — falling back to only the loosest possible rules in this situation, rather than borrowing baselines from the most similar existing category, leaves a genuine detection gap exactly when a new market is most vulnerable to bad actors testing its defenses.

16.3 A short checklist before going to production

Before flipping this system from shadow mode into live enforcement, it is worth working through a short readiness checklist, the same way you would before any other high-stakes production launch. Confirm that every category has a baseline built from a statistically sufficient sample size, not just an average across the whole catalog. Confirm that the campaign allowlist workflow is fully wired up and tested, since the very first major promotional event after launch is exactly when a missing allowlist entry will generate a flood of false positives and erode trust in the system among both sellers and internal stakeholders. Confirm that the fail-closed behavior has actually been chaos-tested — deliberately kill the scoring service in a staging environment and verify that price publishing correctly halts or falls back to the statistical-rules-only mode rather than silently failing open. And confirm that the review queue has enough staffed capacity to handle the expected volume of flagged cases at launch, with a clear escalation path if that volume runs higher than anticipated during the first few weeks.

Designing a system to catch anomalously low prices before they cause real damage is fundamentally an exercise in balancing speed, statistical rigor, and human judgment. The winning architecture is never a single clever algorithm — it is a layered pipeline: a synchronous fast-path guard for the obvious cases, an asynchronous deep-path combining statistics and machine learning for everything else, a fast and reliable enforcement mechanism that reaches all the way to the checkout path, and a well-designed human review loop that both resolves ambiguous cases and continuously improves the automated layer.

Whether you’re asked this exact problem in a system design interview or you’re building it for a real marketplace, the discipline is the same: define the baseline, score the deviation, act fast but reversibly, and always leave a human a clear, well-explained path to the final call.