From printed flyers at a store counter to automated pipelines that watch millions of prices a day.

Imagine you buy a blender from an online store for $80. Three days later, you notice the exact same blender selling for $65 on a different website. Annoying, right? A “price-match guarantee” is a promise retailers make to their customers: “If you find the same product cheaper somewhere else, we’ll match that price — or even beat it — automatically or on request.” It is a trust-building feature. It tells the customer, “You don’t need to shop around. We already made sure you’re getting the best deal.”

Price matching is not a new idea. Brick-and-mortar retailers like electronics and home-improvement stores have offered manual price matching for decades — a customer would walk in with a printed flyer or a competitor’s ad, and a cashier would honor the lower price after a manual check. That process was slow, relied on human judgment, and only worked because a human employee compared two physical pieces of paper.

Online marketplaces changed the scale of this problem entirely. A large e-commerce platform can have tens of millions of products, sold by thousands of competing sellers and websites, with prices that can change dozens of times a day (a practice called dynamic pricing). Doing this manually is now completely impossible. This is why marketplaces began building automated price-match guarantee systems — software that continuously watches competitor prices, decides whether two products from two different stores are really “the same,” compares prices, and automatically triggers a refund, discount, or price adjustment when a lower price is found.

This tutorial walks through, from the ground up, how such a system is designed: how it discovers competitor prices, how it decides two listings are the same product (this is surprisingly the hardest part), how it stores and compares billions of price points a day, and how it stays reliable, fast, and fair even under massive scale.

1.1 A Short Timeline

Let’s break down why this is genuinely difficult, one layer at a time.

2.1 The Data Collection Problem

To know a competitor’s price, you first need to see it. Competitors do not usually hand you a clean feed of their prices. You either (a) have a data-sharing partnership, (b) subscribe to a third-party price-intelligence data provider, or (c) collect the data yourself through web scraping or APIs. Each competitor site has its own HTML structure, its own bot-detection defenses, its own rate limits, and its own habit of changing its page layout without warning. Multiply this by thousands of competitor domains and millions of product pages, and you have an enormous, constantly-shifting data pipeline problem.

2.2 The “Is This the Same Product?” Problem

This is the single hardest part of the whole system, and it’s worth pausing on. Suppose your marketplace sells “Sony WH-1000XM5 Wireless Noise-Canceling Headphones, Black.” A competitor’s site lists “Sony WH1000XM5 Headphones – Black (Renewed).” Are these the same product? The renewed (refurbished) unit is not the same as a new one, and matching them would mean giving customers a discount they should not get. Now imagine doing this comparison across millions of listings with typos, different bundle sizes (“2-pack” vs “1-pack”), regional variants, and marketing filler text. This is a classic entity resolution (also called record linkage or deduplication) problem, and it needs a mix of structured identifiers (like UPC/GTIN barcodes), machine learning text similarity, and image similarity to solve reliably.

2.3 The Business-Rules Problem

Even after you know Product A equals Product B, and Store X sells it cheaper, you still need to apply business logic: Does the price-match guarantee exclude marketplace/third-party sellers? Does it exclude flash sales or clearance items? Is there a minimum price difference threshold (e.g., match only if the difference is more than $1, to avoid matching on rounding noise)? Does the customer need to submit a claim, or is it automatic? These rules change often and must be configurable without a code deployment.

2.4 The Scale and Freshness Problem

Prices change constantly. A system that checks competitor prices once a week is nearly useless for high-velocity categories like electronics. But re-crawling every competitor page every minute for millions of SKUs is prohibitively expensive and will get your IP addresses blocked. The system has to be smart about which products to check more frequently (high-traffic, price-volatile products) versus less frequently (long-tail, stable-price products).

2.5 Why This Cannot Just Be a SQL Join

A tempting first instinct is to imagine that if two retailers just published price feeds into a shared table, mutual matching would collapse into a straightforward join on a product identifier. This intuition fails on almost every axis at real scale. Competitor prices are not offered as clean feeds — they must be inferred from HTML pages that change layout without warning. Even when identifiers are available, they are frequently missing, wrong, or reused for different regional variants of the same product name. And even when identifiers agree, the comparison itself has to respect business rules, currency conversion, tax treatment, freshness windows, and per-domain politeness policies that a plain relational join has no vocabulary for. Recognizing that this is a distributed data-collection, entity-resolution, and rule-evaluation problem — not a lookup — is exactly the shift in framing the rest of this tutorial depends on.

Before we look at the architecture, let’s define the vocabulary we’ll use throughout this tutorial.

3.1 A Closer Look: Why Entity Resolution Is Its Own Discipline

It’s worth slowing down here because this single concept quietly determines whether the entire feature is trustworthy or embarrassing. Think about how a ten-year-old would compare two toys in two different store catalogs. They’d look at the picture first (“does it look the same?”), then read the name (“does it say the same thing?”), and finally check the box for a serial number if there is one. That is, in essence, exactly the three-signal approach production entity-resolution systems use, just automated and made mathematically precise:

• Structured identifiers (strongest signal): A GTIN/UPC/EAN barcode, when present and correctly captured, is close to a guarantee that two listings are the same manufactured item. The catch is that a large fraction of listings, especially from smaller sellers, either omit this field or enter it incorrectly, so it can never be the only signal the system relies on.
• Textual similarity (medium-confidence signal): Product titles, brand names, and attribute fields (color, size, model number) are converted into numeric vector embeddings using a language model, and compared using cosine similarity. Two titles that use different wording for the same product (“Sony 65-inch 4K Smart TV” vs “Sony BRAVIA 65in Television, Smart, 4K”) will land close together in this embedding space even though they share almost no exact words.
• Image similarity (supporting signal, good tiebreaker): Product photos are embedded using a vision model and compared the same way. This catches cases where the text is ambiguous or sparse but the product photo is nearly identical, and also helps rule out false matches — for example, catching that a “renewed” or “refurbished” badge visible in one image but not the other likely means these are different SKUs even if the titles are very close.

None of these signals is perfect alone, which is exactly why the system combines them into a single confidence score, and why anything below a safe threshold gets routed to a human reviewer rather than auto-approved. This layered approach — cheap, precise checks first, expensive, fuzzy checks only when needed, and a human safety net for anything uncertain — is a recurring pattern you’ll see throughout large-scale matching and deduplication systems generally, not just in price matching.

Now let’s design the system end-to-end — a set of independent microservices, each owning one responsibility, connected through an API Gateway at the edge and a Load Balancer in front of every scalable service tier.

Every box below explicitly states which infrastructure component (API Gateway, Load Balancer, etc.) sits in front of or manages it, so you can see exactly how traffic and requests physically travel through the system.

4.1 API Gateway

The API Gateway is the single front door for all external traffic — mobile apps, web frontend, and partner integrations. It handles authentication (verifying who is calling), authorization (verifying what they’re allowed to do), request routing to the right internal service, and coarse-grained rate limiting to protect backend services from being overwhelmed. Every write path in this system (for example, a customer manually submitting a “I found it cheaper” claim) enters through this gateway.

4.2 Load Balancer

Every stateless service tier in this design — Catalog Service, Entity Resolution Service, Rule Engine, Trigger Service, Order Service, Notification Service — sits behind its own Load Balancer. This is deliberate: a Load Balancer distributes incoming requests evenly across many identical pods/instances of a service, so that if one instance is slow or crashes, traffic simply flows to the healthy ones. We use Layer 7 (application-aware) load balancing so that routing decisions can consider things like URL path and headers, not just IP and port.

4.3 Crawl Scheduler and Crawler Worker Pool

The Crawl Scheduler decides when to check each competitor product page next, based on a priority score (high-traffic SKUs, historically volatile prices, and categories close to a big sale event get checked more often). It pushes jobs into a priority queue, which is consumed by an autoscaled pool of Crawler Workers. These workers fetch pages (or call competitor/partner price-feed APIs where available), extract price and availability, and pass raw data downstream.

4.4 Proxy / IP Rotation Layer

To avoid overloading (and getting blocked by) competitor sites, requests go through a proxy pool that rotates IP addresses and respects each site’s rate limits — a “politeness policy.” This layer is itself load balanced across many proxy nodes.

4.5 Price Normalization Service

Raw scraped data is messy: “$1,299.00”, “1299”, “USD 1299”, “1299 kr” all mean different things. The Normalization Service converts every price into a single canonical format (currency, decimal precision, tax-inclusive vs exclusive) so downstream comparison is apples-to-apples.

4.6 Entity Resolution Service (Product Matching)

This service decides whether a competitor’s listing refers to the same product as one in our catalog. It uses a layered strategy: exact barcode match first, then a machine-learning text/image similarity model, then a confidence threshold that routes uncertain matches to a human review queue. Confirmed matches are written into the Canonical Product Graph.

4.7 Price Match Rule Engine

Given a confirmed match and a competitor price lower than ours, the Rule Engine applies business rules: minimum discount threshold, excluded categories, excluded sellers, geographic eligibility, and promotional exclusions. This is intentionally a separate, configurable service so that business/legal teams can update rules without redeploying code.

4.8 Price Match Trigger Service

Once a match passes all rules, this service orchestrates the actual outcome: adjusting the displayed price, issuing a refund on an existing order, or generating a coupon — depending on business configuration — and records everything to an immutable audit log for compliance.

Let’s trace exactly what happens, step by step, inside the system when a competitor drops a price.

1. Discovery: The Crawl Scheduler pulls the next batch of due jobs from its priority queue (backed by a min-heap keyed on “next check time”).
2. Fetch: A Crawler Worker requests the competitor’s product page through the rotating proxy layer, respecting a per-domain rate limit (e.g., no more than 2 requests/second to a given competitor domain).
3. Extraction: The worker parses the page (using CSS selectors or a trained extraction model for pages without clean structure) to pull out price, currency, availability, and product identifiers.
4. Publish raw event: The worker publishes a RawCompetitorPrice event onto a Kafka topic. Publishing to a queue (rather than calling the next service directly) decouples the crawler’s throughput from downstream processing speed.
5. Normalize: A consumer service normalizes currency, computes a per-unit price (important for multi-packs), and forwards a clean NormalizedPrice event.
6. Match: The Entity Resolution Service looks up whether this competitor listing is already linked to a canonical product. If yes, skip straight to comparison. If no, run the matching pipeline (barcode → text embedding similarity → image similarity → confidence score).
7. Compare: If matched with confidence above the auto-accept threshold, compare the normalized competitor price against our current price for that canonical product (fetched from a Redis cache of hot prices, falling back to the Catalog DB).
8. Evaluate rules: If competitor price is lower by more than the configured threshold, the Rule Engine checks eligibility rules.
9. Trigger: If all rules pass, the Trigger Service either updates the live price, generates a discount code, or issues an automatic refund against a recent order, and writes an audit record.
10. Notify: The customer (if they had a pending claim, or if the marketplace auto-applies price protection to prior purchases) is notified.

5.1 What Happens When Confidence Is Uncertain

Not every case is as clean as the example above. Suppose the crawler finds a listing titled “Sony Wireless Headphones XM5 – Refurb, Black” at a much lower price. The text embedding similarity to our catalog’s “Sony WH-1000XM5 Wireless Noise-Canceling Headphones, Black” might score around 0.80 — high enough to suggest a possible match, but below the 0.92 auto-accept threshold, and the word “Refurb” is exactly the kind of detail that changes the correct answer. In this situation the event is routed to a human review queue instead of being auto-approved. A trained reviewer (or a specialized secondary, more expensive verification model) looks at both listings side by side, including the product images, and makes the final call. Their decision is recorded and fed back as labeled training data, so the primary matching model gradually gets better at recognizing this exact pattern — refurbished-vs-new language — without needing every future case to go through a human again. This human-in-the-loop feedback cycle is what allows the auto-accept threshold to be raised safely over time as the model’s real-world accuracy on hard cases improves.

The lifecycle of a single price data point, from being scraped off a competitor’s page to potentially becoming a refund in the customer’s bank account.

Notice the feedback loop at the very end: the Trigger Service’s outcome feeds back into the Crawl Scheduler’s prioritization logic. If a SKU’s price frequently triggers a match, the scheduler increases its crawl frequency; if a SKU never changes, the scheduler backs off to save cost. This is a form of adaptive, self-tuning scheduling.

6.1 Java Code Example: Publishing a Normalized Price Event

public class PriceNormalizer {

    private final KafkaProducer<String, NormalizedPriceEvent> producer;

    public PriceNormalizer(KafkaProducer<String, NormalizedPriceEvent> producer) {
        this.producer = producer;
    }

    // Converts raw scraped price data into a canonical form and publishes it
    public void normalizeAndPublish(RawCompetitorPrice raw) {
        BigDecimal amount = CurrencyUtil.toUsd(raw.getPrice(), raw.getCurrencyCode());
        BigDecimal perUnitPrice = amount.divide(
                BigDecimal.valueOf(raw.getPackQuantity()), 4, RoundingMode.HALF_UP);

        NormalizedPriceEvent event = NormalizedPriceEvent.builder()
                .competitorDomain(raw.getSourceDomain())
                .externalProductId(raw.getExternalProductId())
                .rawTitle(raw.getTitle())
                .normalizedPriceUsd(amount)
                .perUnitPriceUsd(perUnitPrice)
                .capturedAt(Instant.now())
                .build();

        // Keyed by external product id so all events for one listing land on the
        // same partition, preserving per-listing ordering
        producer.send(new ProducerRecord<>("normalized-competitor-prices",
                raw.getExternalProductId(), event));
    }
}

6.2 Java Code Example: Confidence-Based Entity Matching

public class EntityResolutionService {

    private final ProductGraphRepository graphRepository;
    private final EmbeddingClient embeddingClient;

    private static final double AUTO_ACCEPT_THRESHOLD = 0.92;
    private static final double REVIEW_THRESHOLD      = 0.70;

    public MatchResult resolve(NormalizedPriceEvent event) {
        // Step 1: strongest signal - exact barcode match
        Optional<CanonicalProduct> byBarcode =
                graphRepository.findByGtin(event.getGtin());
        if (byBarcode.isPresent()) {
            return MatchResult.confirmed(byBarcode.get(), 1.0);
        }

        // Step 2: text + image embedding similarity against candidate products
        float[] titleEmbedding = embeddingClient.embedText(event.getRawTitle());
        List<CandidateMatch> candidates =
                graphRepository.findNearestByEmbedding(titleEmbedding, 5);

        if (candidates.isEmpty()) {
            return MatchResult.noMatch();
        }

        CandidateMatch best = candidates.get(0);
        if (best.getSimilarityScore() >= AUTO_ACCEPT_THRESHOLD) {
            return MatchResult.confirmed(best.getProduct(), best.getSimilarityScore());
        } else if (best.getSimilarityScore() >= REVIEW_THRESHOLD) {
            return MatchResult.needsHumanReview(best.getProduct(), best.getSimilarityScore());
        }
        return MatchResult.noMatch();
    }
}

6.3 Java Code Example: Rule Engine Evaluation

public class PriceMatchRuleEngine {

    private final RuleConfigRepository ruleConfig;

    public RuleDecision evaluate(MatchedPriceEvent event, CatalogItem ourItem) {
        BigDecimal difference = ourItem.getCurrentPrice()
                .subtract(event.getCompetitorPriceUsd());

        RuleSet rules = ruleConfig.getActiveRuleSetFor(ourItem.getCategoryId());

        if (difference.compareTo(rules.getMinimumDifference()) <= 0) {
            return RuleDecision.reject("Difference below minimum threshold");
        }
        if (rules.getExcludedSellerIds().contains(event.getCompetitorSellerId())) {
            return RuleDecision.reject("Competitor seller excluded from guarantee");
        }
        if (!rules.isCategoryEligible(ourItem.getCategoryId())) {
            return RuleDecision.reject("Category not eligible for price match");
        }
        if (ourItem.isOnActivePromotion() && !rules.allowsDuringPromotion()) {
            return RuleDecision.reject("Item currently on promotion, excluded by policy");
        }
        return RuleDecision.approve(difference);
    }
}

Every design choice above trades something away — here they are, side by side.

At marketplace scale, the system may need to track tens of millions of SKUs against thousands of competitor domains, generating hundreds of millions of price observations per day. Here’s how each layer scales.

8.0 Back-of-the-Envelope Capacity Estimation

Before designing for scale, it helps to put rough numbers on the problem — this is also a very common thing interviewers want to see you do out loud. Let’s walk through a simplified estimate for a large marketplace.

• Catalog size: Assume 50 million actively-sold SKUs.
• Tiering: Using the tiered crawl strategy from section 8.3 — Tier 1 (5%) checked every 20 minutes, Tier 2 (25%) checked every 3 hours, Tier 3 (70%) checked once daily.
• Tier 1 crawl volume: 2.5 million SKUs times roughly 72 checks/day (every 20 minutes) is about 180 million crawl requests per day from this tier alone.
• Tier 2 crawl volume: 12.5 million SKUs times 8 checks/day (every 3 hours) is about 100 million requests per day.
• Tier 3 crawl volume: 35 million SKUs times 1 check/day is 35 million requests per day.
• Total: Roughly 315 million crawl requests per day, or about 3,650 requests/second sustained on average — though real traffic is bursty around sale events, so the system should be provisioned for several times that average as a peak, not just the average itself.
• Storage estimate: If each raw price observation is roughly 1 KB once serialized (price, currency, timestamp, source, listing id, raw title snippet), 315 million observations/day is about 315 GB/day of raw event data before compression, which is very manageable for a Kafka cluster with a short retention window (a few days) feeding into cheaper long-term columnar storage for analytics.

This kind of estimate immediately tells you two important things: the crawler tier needs to comfortably sustain several thousand requests per second at peak (driving the decision to horizontally autoscale the worker pool and spread load across a large proxy pool), and the event pipeline’s storage footprint, while large in absolute terms, is entirely tractable with standard stream-processing infrastructure as long as raw events are aged out of “hot” storage after a short window.

8.1 Horizontal Scaling of Crawlers

Crawler Workers are stateless and horizontally autoscaled based on queue depth (how many crawl jobs are waiting). If the queue backs up, Kubernetes’ Horizontal Pod Autoscaler spins up more worker pods, each pulling jobs independently.

8.2 Partitioning the Message Broker

Kafka topics are partitioned by product/SKU id, so all events for a given product land on the same partition and preserve order, while different products are processed fully in parallel across partitions.

8.3 Priority-Based Crawl Scheduling

Not all SKUs deserve equal crawl frequency. A tiered approach works well in practice:

8.4 Caching Hot Prices

The current “our price” lookup, done on every single comparison, is one of the hottest read paths in the system. Rather than hitting the Catalog Database each time, the Rule Engine reads from a Redis cache populated by Change Data Capture from the Catalog DB, giving sub-millisecond lookups for millions of comparisons per hour.

8.5 Batching ML Inference

The Entity Resolution Service batches embedding-generation requests (grouping many normalized events together before calling the model) to make efficient use of GPU/accelerator hardware, trading a small amount of added latency (tens of milliseconds) for a large throughput gain.

A price-match system touches money directly (refunds, discounts), so reliability and correctness matter enormously — a bug here has real financial consequences.

9.1 Idempotency

Every trigger event carries a unique idempotency key (e.g., a hash of order id + competitor listing id + captured timestamp bucket). The Trigger Service checks this key before acting, so retries from a crashed consumer never cause a duplicate refund.

9.2 At-Least-Once Delivery with Deduplication

Kafka consumers are configured for at-least-once delivery (safer to reprocess than to silently drop a price event), paired with a deduplication store (keyed on idempotency key, short TTL) downstream to guarantee exactly-once effects even though delivery itself may occasionally repeat.

9.3 Circuit Breakers on External Dependencies

Calls to competitor sites, third-party price-feed APIs, and the ML model-serving layer are all wrapped in circuit breakers. If a competitor site starts timing out heavily, the breaker trips, stops sending requests for a cool-down period, and the scheduler reroutes crawl budget elsewhere instead of piling up stuck threads.

9.4 Multi-Region Replication

The Canonical Product Graph and Catalog databases are replicated across regions. In case of a regional outage, traffic fails over to a secondary region using a Load Balancer configured with health-check-based failover.

9.5 Graceful Degradation

If the Entity Resolution Service is unavailable, the system does not crash the customer-facing price display. Instead it falls back to only using previously-confirmed canonical matches, pausing new match discovery until the service recovers — the guarantee still works for known products, just doesn’t grow with new ones temporarily.

9.6 Advanced: Consistency, CAP Theorem & Consensus

It’s worth pausing on some deeper distributed-systems theory that directly shapes several decisions we’ve already made in this design.

9.6.1 CAP Theorem in This System

The CAP theorem says that in the presence of a network partition, a distributed system must choose between Consistency (every read sees the latest write) and Availability (every request gets a response, even if it might be slightly stale). This system deliberately makes different choices for different data:

• Current price cache (Redis): We choose Availability over strict Consistency. If the cache is a few seconds stale during a network blip, that’s an acceptable trade — the Rule Engine simply compares against a slightly outdated price, and the next comparison cycle will catch up.
• Order/Refund system: Here we lean toward Consistency for the specific write path that issues a refund — we would rather have that request fail and retry than risk issuing a refund based on stale, possibly-already-changed order state.
• Canonical Product Graph: Reads can tolerate eventual consistency (a newly confirmed match might take a few seconds to propagate to all replicas), but writes to the “confirmed match” relationship are funneled through a single writer per canonical product to avoid conflicting concurrent updates.

9.6.2 Consensus and Leader Election

Certain coordination tasks in this system need a single source of truth even across multiple replicas — for example, deciding which Crawl Scheduler instance is currently “active” so that duplicate schedulers don’t double-dispatch the same crawl jobs. This is solved using a consensus-based leader election mechanism (commonly implemented via Zookeeper, etcd, or a similar strongly-consistent coordination service using the Raft consensus algorithm). Only the elected leader dispatches jobs; if it crashes, the remaining nodes detect the missing heartbeat and elect a new leader within a bounded time window.

9.6.3 Data Structures & Algorithms Under the Hood

A few classic data structures and algorithms show up repeatedly in this design:

• Min-heap priority queue: The Crawl Scheduler’s “what to check next” logic is a textbook min-heap, keyed by next-due-time, giving O(log n) insert and O(log n) extract-min even with millions of scheduled SKUs.
• Token bucket algorithm: Per-domain rate limiting uses a token bucket (a counter that refills at a fixed rate and is decremented per request), which smooths out bursts while still allowing short spikes up to the bucket’s capacity.
• Approximate nearest-neighbor search (HNSW): Product embedding similarity search uses a Hierarchical Navigable Small World graph index, giving fast approximate search instead of a slow brute-force scan across hundreds of millions of vectors.
• Consistent hashing: Used to assign competitor domains to specific crawler worker groups, so that adding or removing workers only reshuffles a small fraction of domain assignments instead of all of them.
• Bloom filters: A lightweight, space-efficient probabilistic structure used by the Normalizer to quickly check “have I likely seen this exact raw price payload before?” before doing a full, expensive database dedup lookup — trading a small false-positive rate for huge memory savings.

9.6.4 Disaster Recovery, Backup & Cost Optimization

Because the audit log and order/refund data are financially and legally sensitive, they follow a stricter backup regime than, say, cached price data: point-in-time recovery snapshots, cross-region backup replication, and periodic restore drills to confirm backups actually work — a backup nobody has ever restored is not a real backup. For cost optimization, the tiered crawl-frequency strategy described earlier is itself the single biggest lever. The majority of SKUs are long-tail and don’t need minute-level freshness, so spending compute and proxy budget proportionally to a SKU’s revenue and volatility, rather than uniformly, can cut crawling costs dramatically while barely affecting the guarantee’s effectiveness where it matters most.

Because this system moves money, security is not optional polish — it’s core.

• Authentication & Authorization: The API Gateway enforces OAuth2/JWT-based authentication for all customer-facing claim submissions and mTLS between internal services.
• Rate Limiting and Abuse Detection: Customers submitting manual “I found it cheaper” claims are rate-limited and screened for fraud patterns (e.g., submitting fabricated competitor URLs, or repeatedly claiming against the same order).
• Input Validation on Scraped Data: Never trust data scraped from an external site. Prices, URLs, and titles are sanitized and validated before entering the pipeline, to prevent injection attacks or malformed data from corrupting the Product Graph.
• Least Privilege for Refund Triggering: The Trigger Service is granted narrow, specific permissions to the Order/Payment system (issue-refund-up-to-X-amount) rather than broad admin access, limiting blast radius if it is ever compromised.
• Audit Logging: Every triggered match is written to an append-only, tamper-evident audit log for compliance and dispute resolution.
• Encryption: All data in transit uses TLS; sensitive data (order/payment references) is encrypted at rest.
• Secrets management: Credentials for proxy providers, third-party price-feed partners, and internal service-to-service authentication tokens are stored in a dedicated secrets manager rather than embedded in configuration files or environment variables checked into source control, and are rotated on a regular schedule.
• Web Application Firewall (WAF): Sits in front of the API Gateway to filter out common attack patterns (SQL injection attempts, malformed payloads, known bad IP ranges) before they ever reach application code, adding a defense layer above and beyond application-level input validation.

Observability lets you know the system is behaving correctly, not just that it’s “up.”

Distributed tracing (e.g., using OpenTelemetry with a trace id propagated from the Crawl Scheduler all the way through to the Trigger Service) lets engineers follow a single price event’s entire journey across services when debugging a specific customer complaint. Structured logs are shipped to a centralized log store, and dashboards visualize the metrics above with alerting thresholds (e.g., page on-call if triggered-match volume in any category exceeds 3x its 7-day average).

11.1 Designing Good Alerts, Not Just Dashboards

It’s a common mistake to build beautiful dashboards that nobody looks at until something has already gone wrong. A better approach is to invest specifically in a small number of high-signal, low-noise alerts tied directly to customer and business impact, rather than alerting on every possible metric:

• Financial guardrail alert: Total approved refund amount in any rolling 15-minute window exceeds a hard dollar ceiling — pages on-call immediately, since this is the single most direct signal that something is badly wrong (a bug, a bad rule deploy, or an attack).
• Pipeline health alert: Kafka consumer lag on any critical topic grows unbounded for more than a few minutes, indicating a downstream service is stuck or under-provisioned.
• Data quality alert: The proportion of crawl jobs failing to extract a valid price for a given competitor domain jumps sharply, indicating that domain’s page layout likely changed and the extraction logic needs updating.
• Model drift alert: The human-review override rate (how often a human reviewer disagrees with the model’s proposed match) crosses a threshold, indicating the entity-resolution model may need retraining sooner than its scheduled cadence.

Each of these alerts should route to the team that can actually act on it, and each should link directly to a relevant dashboard and a runbook describing the first diagnostic steps — an alert without a clear next action just trains people to ignore alerts.

This system is well suited to a containerized, cloud-native deployment.

• Kubernetes orchestrates all stateless service tiers (Catalog Service, Entity Resolution, Rule Engine, Trigger Service), each with its own Horizontal Pod Autoscaler tied to CPU, queue depth, or custom metrics.
• Managed Kafka (or an equivalent managed streaming service) handles the event backbone, avoiding the operational burden of running brokers manually.
• Managed relational/NoSQL databases with automated backups and cross-region read replicas back the Catalog and Product Graph stores.
• Serverless functions are a good fit for spiky, low-latency tasks like the Notification Service, since notification volume correlates directly with matched-event volume and can spike unpredictably.
• CI/CD pipelines deploy each microservice independently, using canary releases for the Rule Engine specifically (since a bad rule change has direct financial impact) — a small percentage of traffic sees the new rule set first, monitored closely before full rollout.
• Infrastructure as Code (e.g., Terraform) defines the entire environment, so proxy pool scaling, Kafka topic configuration, and autoscaling policies are version-controlled and reproducible across environments.

Storage decisions for three very different data shapes: OLTP catalog, a canonical product graph with vector search, and a hot price cache.

13.1 Catalog Database

Stores our own product catalog: SKU, price, category, promotions. This is a classic OLTP workload — sharded by product/category id for write scalability, with read replicas fronted by a Load Balancer to absorb the heavy read traffic from the Rule Engine and customer-facing pages.

13.2 Canonical Product Graph Database

This is the trickiest data store in the system. It needs to support: fast exact lookup by barcode, nearest-neighbor vector search for embedding similarity, and graph-like relationships (one canonical product can link to many competitor listings). In practice, this is often implemented as a combination of a relational/document store for the canonical records plus a specialized vector database (or a vector index like HNSW built into the primary store) for similarity search.

13.3 Distributed Cache

A Redis cluster caches (a) our current price per SKU for fast Rule Engine lookups, and (b) recently resolved competitor-to-canonical-product mappings, so repeat sightings of the same competitor listing skip the expensive matching pipeline entirely.

13.3.1 Sharding and Replication Strategy in Detail

Let’s go one level deeper on how the Catalog Database and Canonical Product Graph are actually partitioned across machines, since this is a favorite interview follow-up.

• Catalog Database sharding key: Sharded by category id combined with a hash of the SKU id. Category-based sharding keeps related products physically close together (useful for category-wide analytics and bulk rule application), while the SKU hash component prevents any single mega-popular category from creating a “hot shard” that gets disproportionately more traffic than its neighbors.
• Canonical Product Graph sharding key: Sharded by canonical product id itself, generated as a new identifier the first time a product is confirmed. This keeps all of one product’s linked competitor listings and match history together on a single shard, so a full lookup (“give me every known competitor price for this product”) never has to fan out across multiple shards.
• Replication factor: Both stores run with at least three replicas per shard, spread across separate availability zones, using leader-follower replication. Writes go to the leader; reads can be served from followers for the majority of traffic (like the Rule Engine’s price lookups), reserving leader reads only for paths that need the absolute latest write (like right after a new match is confirmed).
• Rebalancing: As shards grow unevenly over time (some categories simply have far more SKUs and price history than others), an online resharding process — one that moves data between shards without taking the system offline — periodically rebalances hot shards, similar in spirit to how consistent hashing minimizes data movement when the crawler worker pool changes size.

13.4 Load Balancing Strategy

Every internal service tier is fronted by its own Load Balancer using health-check-based routing (unhealthy pods are automatically removed from rotation) and, where relevant, consistent hashing (for example, routing all requests for a given competitor domain to the same crawler worker group, to make local rate-limiting state easier to manage without needing a distributed lock).

Representative API contracts for two key services on either side of the trust boundary.

14.1 Customer-Facing Claim Submission API

POST /v1/price-match/claims
Headers: Authorization: Bearer <jwt>
Body:
{
  "orderId": "ORD-88213",
  "competitorUrl": "https://competitor.example.com/product/9981",
  "claimedPriceUsd": 259.00
}

Response 202 Accepted:
{
  "claimId": "CLAIM-771102",
  "status": "PENDING_VERIFICATION",
  "estimatedResponseTime": "PT2H"
}

14.2 Internal Rule Engine Evaluation API

POST /internal/v1/rule-engine/evaluate
Body:
{
  "canonicalProductId": "CPG-554210",
  "ourCurrentPriceUsd": 299.00,
  "competitorPriceUsd": 259.00,
  "competitorSellerId": "SELLER-COMPETITOR-A",
  "matchConfidence": 0.97
}

Response 200 OK:
{
  "decision": "APPROVE",
  "priceDifferenceUsd": 40.00,
  "reasonCodes": []
}

Each microservice in this design owns its own data store (Catalog Service owns the Catalog DB, Entity Resolution owns the Product Graph) and communicates with others exclusively through well-defined APIs or asynchronous events — never by directly querying another service’s database. This keeps services independently deployable and lets teams evolve their internal schemas freely.

Patterns worth reusing, and mistakes worth naming so you can avoid them.

15.1 Patterns Used

15.2 Anti-Patterns to Avoid

15.3 Testing Strategy for a Money-Moving System

Because this system’s end effect can be an actual customer refund, testing needs to go beyond typical unit and integration tests. A few practices matter especially here:

• Shadow mode before launch: When first turning on auto-matching for a new category, run the full pipeline end-to-end but stop just short of the Trigger Service actually acting — instead log what would have happened. This lets the team compare the model’s proposed matches and rule outcomes against expected behavior over days or weeks of real traffic, with zero financial risk, before flipping it to fully live.
• Golden-set regression tests: A curated set of known tricky product pairs (same product different packaging, similar-but-different models, refurbished-vs-new) is re-run against the entity-resolution pipeline on every model or code change, catching regressions before they reach production.
• Chaos testing on the crawl and messaging layers: Deliberately killing crawler worker pods, injecting Kafka broker failures, and simulating competitor sites timing out in a staging environment verifies that circuit breakers, retries, and the scheduler’s rebalancing logic behave as designed under real failure conditions, not just in theory.
• Financial reconciliation jobs: A separate, independent batch job periodically re-checks that every entry in the audit log corresponds to an actual matching order/refund record, catching any silent inconsistency between the Trigger Service’s actions and the ground-truth financial systems.

Each row below pairs a habit worth building in with the failure mode it prevents.

The same skeleton appears in retail, marketplaces, and travel — a good sign it captures a genuinely reusable pattern.

Large electronics and general-merchandise retailers have historically offered formal price-match guarantees, evolving from purely manual, in-store processes toward automated online systems as e-commerce volume grew. Modern price-intelligence vendors now offer competitor price-tracking as a service, which many mid-size retailers integrate rather than building their own crawling infrastructure from scratch — illustrating the “buy vs. build” trade-off for the data-collection layer specifically. Large online marketplaces with massive third-party seller ecosystems face an even harder version of this problem internally: deciding which of many seller offers for the “same” product should win the featured placement (sometimes called a “buy box”), which relies on very similar entity-resolution and price-comparison technology to what’s described in this tutorial, just applied to their own internal sellers rather than external competitors.

Streaming and subscription businesses have also adopted lightweight versions of this idea (price-drop notifications and rebooking guarantees in travel booking are a close cousin), reusing the same core building blocks: continuous external monitoring, entity/itinerary matching, and a rules engine deciding when to notify or auto-refund a customer.

Travel booking platforms offer a particularly instructive variant of this same architecture: instead of matching physical products, they match itineraries (same flight number, same dates, same cabin class) and instead of comparing against an external competitor, they often compare a customer’s already-booked price against that same platform’s own price for the identical itinerary later in the booking window — effectively price-matching against their own future price drops. The underlying building blocks are nearly identical: a scheduler that decides which bookings to re-check and how often, an entity-matching layer that confirms “same flight, same fare class, same dates,” a rule engine that applies eligibility windows and blackout dates, and a trigger service that issues a travel credit or refund. Seeing the same architectural skeleton reappear across such different domains — physical retail and travel — is a good sign that the design captured here reflects genuinely reusable, fundamental patterns rather than being narrowly specific to one industry.

Grocery delivery and meal-kit companies have also experimented with automated price matching for staple goods, though they typically restrict it to a narrower set of well-identified, barcoded products (where the “same product” question is much easier to answer confidently), illustrating a practical lesson: many real production systems deliberately scope the entity-resolution problem down to the cases they can solve with very high confidence, and grow the scope over time as their matching technology and confidence calibration improve, rather than attempting perfect coverage from day one.

Most competitors don’t expose a public pricing API to outside parties, since it would let others (including their own competitors) track their pricing strategy in real time. When such feeds do exist — often through paid data partnerships — they’re strongly preferred over scraping because they’re more reliable and don’t carry the legal/ethical questions of automated scraping.

Tier-1 SKUs are crawled frequently enough (every 15–30 minutes) to catch most flash sale windows, and the Rule Engine can apply a rule that ignores extremely short-lived price drops (below a configurable minimum duration) to avoid matching momentary pricing glitches or flash-sale-only pricing that doesn’t reflect the competitor’s real standing price.

The Rule Engine always compares against the single lowest verified competitor price for a canonical product at evaluation time, computed from all currently known, confidence-verified listings linked to that product.

It’s a hybrid: crawling and event processing are streaming/real-time (seconds to minutes of latency end-to-end), but analytics, model retraining, and long-tail SKU crawling run on batch/scheduled cadences to control cost.

By only trusting price observations from a curated list of vetted, well-known competitor domains (not arbitrary user-submitted URLs) for the automated path, and independently re-verifying any customer-submitted claim through the system’s own crawler before approving it.

Exact string matching fails constantly in practice because sellers word titles differently, include marketing filler, use different unit orderings, and sometimes contain typos. A small pilot might get away with rule-based fuzzy string matching (edit distance, token overlap) for a narrow catalog, but any marketplace with a broad catalog and many competitor sources will need embedding-based similarity to get acceptable precision and recall — the false-positive cost (wrongly matching two different products) is high enough that investing in a proper ML approach pays for itself quickly.

This depends on how much your catalog and competitor landscape shift, but a common cadence is a scheduled retrain (for example, monthly) combined with an event-driven retrain trigger whenever the human-review override rate crosses an alert threshold, since that’s a strong signal the model’s assumptions no longer match current data.

Dynamic repricing is when a seller automatically adjusts their own price continuously based on market conditions to stay competitive, purely as a pricing strategy — the customer is never directly compensated for a specific missed price drop. A price-match guarantee is customer-facing: it’s a promise that ties a concrete, verifiable competitor price observation to a concrete customer benefit (a refund, discount, or price adjustment), and it typically comes with published eligibility rules the customer can rely on. The two can share underlying infrastructure (both need competitor price monitoring) but serve different business purposes.

Both patterns exist in practice and solve different problems. A pre-checkout check (comparing live prices while the customer is still browsing) is typically a much simpler, synchronous, real-time price comparison against a small number of already-confirmed canonical matches, and is closer to a “price display” feature than the full guarantee. The retroactive, post-purchase guarantee described throughout this tutorial is the harder and more valuable version, because it protects the customer for a window of time after they’ve already committed to the purchase, which is exactly when new, previously-undiscovered competitor listings and price drops are most likely to surface through the full crawling and matching pipeline. Many production systems offer both: a lightweight real-time comparison at checkout, and the full asynchronous guarantee pipeline running continuously afterward for the eligibility window.

Ultimately, this system is a useful case study precisely because it looks like a simple lookup on the surface and reveals, on closer inspection, nearly every major theme in distributed systems design at once: partitioning data across machines, choosing between consistency and availability under real failure conditions, picking the right algorithm and data structure for a genuinely fuzzy input (product identity), caching aggressively without sacrificing correctness on financially-sensitive edges, and building operational practices — monitoring, SLOs, chaos testing, financial reconciliation — that keep the whole thing healthy long after the initial launch. A candidate who can design this system well, end to end, has effectively demonstrated fluency in the core skills that large-scale backend engineering demands day to day.