Every time you tap your card at a coffee shop, a decision has to be made almost instantly: is this really you, or is it someone who stole your card number five minutes ago in a different country? That decision is not made by a human — it is made by a fraud detection system that must respond before the payment terminal times out, before the cashier gets impatient, and before you start wondering why your coffee is taking so long. The industry-standard target for this decision is under 100 milliseconds — faster than the blink of an eye (a blink takes roughly 100-150ms) — end to end, including network round trips.

This tutorial designs exactly that system: one that ingests a live stream of credit card authorization requests at marketplace-payment-processor scale (millions of transactions per minute across global peak periods like Black Friday or Singles’ Day), extracts and looks up features about the cardholder’s recent behavior, runs one or more machine learning models and rule engines, and returns a binary (or graded) decision — all inside a strict, non-negotiable latency budget, because a payment network will simply time out and treat a non-response as a failure if the deadline is missed.

1.1 A short history of the problem

This tutorial designs an Era-4 system, with an explicit latency budget breakdown (Section 8) because, unlike most system designs where “make it faster” is a nice-to-have, here missing the 100ms deadline is a hard functional failure — the transaction gets an implicit decline or timeout regardless of how good the fraud model would have said “approve.”

By the end of this tutorial you will be able to answer, in interview-level depth, questions like: how do you keep every dependency in a decision pipeline fast enough that their combined cost still fits inside 100 milliseconds; how do you combine a machine learning model’s probabilistic judgment with deterministic, instantly explainable business rules; how do you decide, region by region and merchant by merchant, whether the system should fail open or fail closed when something goes wrong; and how do you scale all of this to millions of transactions per minute without a single global bottleneck. Every architectural box in the diagrams below is explicitly labeled — API Gateway, Load Balancer, Decision Orchestrator, Model Serving Cluster, and so on — exactly as you would draw them on a whiteboard in a real system design interview.

2.1 Problem statement

Design a system that, for every incoming card-authorization request, makes a fraud decision (APPROVE / DECLINE / STEP-UP-CHALLENGE) within a strict 100 millisecond end-to-end budget, at a throughput of millions of transactions per minute globally, using real-time and historical features about the card, the merchant, the device, and the transaction context, combined with machine learning models and deterministic rules, while maintaining extremely high availability (a fraud system that’s down cannot simply “fail open” without cost, nor “fail closed” without blocking legitimate commerce).

2.2 Why this is hard

2.3 Goals

• Functional: real-time, per-transaction fraud decisioning (APPROVE/DECLINE/CHALLENGE) with explainable contributing factors, at P99 latency under 100ms end to end.
• Non-functional: horizontal scalability to millions of transactions/minute, 99.99%+ availability (with an explicit, safe fail-open/fail-closed policy per merchant risk tier), strict data security/compliance (PCI-DSS), and continuous model/rule evolution without downtime.

2.4 Non-goals

This system is not a post-transaction dispute/chargeback investigation tool (that’s a separate, offline workflow), not a general anti-money-laundering (AML) case-management system (though it may share signals with one), and not responsible for merchant-side fraud (fake merchants, refund fraud) — this tutorial is scoped specifically to the real-time authorization-time decision for a single card transaction.

Every box below is deliberately labeled with its concrete role, exactly as it would appear on a whiteboard in a system design interview.

4.1 High-level architecture diagram

4.2 Component responsibilities

4.3 Component deep-dive

API Gateway

What: The single entry point that all authorization requests pass through before reaching the Decision Orchestrator. Why here: Centralizes authentication of the network/acquirer connection, request-schema validation, and per-source rate limiting, so the orchestrator’s hot-path code stays entirely focused on fraud logic, not connection/protocol concerns.

Load Balancer

What: Distributes each request across many stateless Decision Orchestrator replicas. Why here: At millions of transactions per minute, no single instance can handle the load; the load balancer’s health checks are also what removes a slow or degraded instance from rotation within milliseconds, which matters enormously under a 100ms budget where one bad instance could otherwise blow the deadline for many requests.

Online Feature Store

What: A low-latency key-value store (often an in-memory store like Redis, or a purpose-built low-latency feature-serving system) holding the latest computed features per card/merchant/device. Why here: Feature lookups must return in low single-digit milliseconds; a system designed for analytical queries (like a data warehouse) simply cannot meet this latency at this request volume.

Streaming Feature Pipeline

What: A stateful stream-processing job (Flink is the industry-standard choice) that consumes the transaction event stream and incrementally updates rolling counters/aggregates. Why here: Fraud’s strongest signals are velocity-based and time-sensitive; if features were only refreshed by a nightly batch job, they would be useless for catching a fast-moving fraud burst happening right now.

Model Serving Cluster

What: A dedicated, horizontally scaled inference service hosting the trained fraud model, optimized specifically for latency (not just throughput). Why here: Separating model serving from the orchestrator allows independent scaling, independent deployment of new model versions (via canary), and hardware specialization (e.g., optimized numeric libraries, sometimes GPU acceleration for deep models) without touching orchestration logic.

Rule Engine Service

What: A fast, deterministic evaluator of business/compliance rules, typically compiled or JIT-evaluated for speed rather than interpreted line-by-line at request time. Why here: Rules must both run inside the same tight budget as the model and be independently auditable/explainable — qualities a black-box ML model alone cannot provide for regulators or dispute resolution.

Decision Cache (Redis)

What: A short-TTL cache mapping a transaction’s idempotency key to its already-computed decision. Why here: Payment networks retry authorization messages under ambiguous network conditions; without this cache, a retried message could be independently re-scored, potentially producing an inconsistent decision or double-counting the transaction in velocity features.

1. A cardholder taps their card. The merchant terminal sends an authorization message through the acquirer to the card network, which routes it to the issuer’s infrastructure — arriving at the Edge Ingress / API Gateway.
2. The API Gateway authenticates the connection, validates the message schema, and forwards it through the Load Balancer to an available Decision Orchestrator instance.
3. The Decision Orchestrator immediately checks the Decision Cache using the transaction’s idempotency key — if this exact transaction was already scored (a network retry), the cached decision is returned instantly, skipping the rest of the pipeline entirely.
4. On a cache miss (the normal case), the Orchestrator fires off several lookups in parallel, not sequentially: a feature lookup to the Online Feature Store (card velocity, merchant risk tier, recent geography), a device/identity risk lookup, and a call to the Rule Engine Service.
5. Once features are assembled, the Orchestrator calls the Model Serving Cluster with the feature vector, which returns a fraud probability score in a few milliseconds.
6. The Orchestrator combines the rule engine’s verdict (which can hard-override to DECLINE regardless of the model) with the model’s probability score against a merchant/region-specific threshold, producing a final decision: APPROVE, DECLINE, or CHALLENGE (step-up authentication).
7. The decision is written to the Decision Cache (for retry-safety) and returned through the Load Balancer and API Gateway back through the network to the merchant — ideally with tens of milliseconds still to spare inside the 100ms budget.
8. Asynchronously, and entirely off this critical path, the full transaction plus its decision and contributing factors is published to Kafka for the streaming feature pipeline (to update velocity counters for the next transaction) and to the Async Event Sink / Data Lake for audit, dispute handling, and future model retraining.

As with the earlier tutorial’s category-anomaly design, this system has two parallel lifecycles: the short, strict synchronous lifecycle of a single authorization request (must complete in under 100ms), and the long-running, continuously updating lifecycle of a card’s or merchant’s rolling risk features (never “finishes,” constantly refreshed by the streaming pipeline). The sequence diagram below focuses on the synchronous request lifecycle, since that is where the hard latency constraint lives.

Let’s look at how the actual decision is computed once features, a model score, and rule verdicts are all in hand.

7.1 Hybrid rules + ML scoring

What: Combining fast, explainable, deterministic rules with a probabilistic ML model rather than relying on either alone. Why: Rules catch known, hard-and-fast patterns instantly and are trivially explainable to regulators and dispute teams; ML catches subtle, evolving patterns humans haven’t explicitly encoded yet. How they combine: Rules can hard-override to DECLINE (e.g., card reported stolen) regardless of model score; otherwise, the model’s fraud probability is compared against a threshold to decide APPROVE / CHALLENGE / DECLINE.

7.2 Gradient-boosted trees for low-latency inference

What: An ensemble machine learning model (like XGBoost or LightGBM) that is fast to evaluate at inference time (often sub-millisecond to a few milliseconds for a single prediction) and works well on the kind of structured, tabular features (amounts, counts, categorical merchant codes) that describe a transaction. Why: Deep neural networks can be more powerful for some fraud patterns but are typically slower and heavier to serve at this latency budget unless carefully optimized; gradient-boosted trees remain the industry default for the primary real-time score precisely because of their excellent latency/accuracy trade-off.

7.3 Threshold calibration per risk tier

What: Instead of one global fraud-probability threshold, the system uses different thresholds per merchant category, region, and transaction amount band. Why: The cost of a false decline versus a false approval isn’t uniform — a $5 coffee purchase can tolerate a much more lenient threshold than a $5,000 electronics purchase, because the financial exposure of a wrong decision differs by orders of magnitude.

7.4 Step-up challenge as a third option

What: Instead of a strict binary APPROVE/DECLINE, transactions in an ambiguous risk band (moderate fraud probability) can be routed to a step-up authentication challenge (e.g., a one-time code or biometric confirmation) rather than an outright decline. Why: This recovers legitimate transactions that would otherwise be falsely declined, at the cost of some added friction — the “smart middle ground” required by regulations like PSD2’s Strong Customer Authentication in the EU.

7.5 Cost-sensitive threshold selection

What: Choosing the decision threshold not by pure statistical accuracy but by explicitly weighing the dollar cost of a false decline (lost sale, customer churn risk) against the dollar cost of a false approval (chargeback liability, fees, reputational cost). Why: Because fraud is rare, a model optimized purely for accuracy would learn to nearly always predict “not fraud” and still score well on paper while missing almost every real case — the threshold has to be chosen against the business’s actual cost structure. Example: If a false decline costs the business an estimated $8 in lost margin/goodwill on average, and a false approval costs $45 in chargeback and fee liability on average, the threshold is tuned so that the expected combined cost is minimized:

$$ text{ExpectedCost} = C_{text{FD}} cdot P(text{FD} mid text{approve}) + C_{text{FA}} cdot P(text{FA} mid text{approve}) $$

where $C_{text{FD}}$ and $C_{text{FA}}$ are the per-error costs and the threshold $tau$ is chosen to minimize the total expected cost across the traffic mix.

7.6 Model explainability for regulatory and dispute needs

What: Techniques (such as SHAP values or simpler feature-contribution breakdowns) that attribute a model’s fraud-probability output to specific input features, producing a human-readable explanation alongside the raw score. Why: Regulators, dispute-resolution teams, and sometimes customers themselves are entitled to understand why a transaction was declined; a bare probability number with no explanation is both an operational liability and, in some jurisdictions, a compliance gap. Example: A declined transaction’s response carries not just fraudProbability: 0.87 but also its top three contributing factors — e.g., “unusual merchant category for this card,” “high transaction velocity in the last 10 minutes,” “new/unrecognized device.”

7.7 A simplified decision combiner (Java)

// Combines rule-engine verdicts and an ML fraud-probability score
// into a final APPROVE / CHALLENGE / DECLINE decision.
public class FraudDecisionCombiner {

    private final double declineThreshold;
    private final double challengeThreshold;

    public FraudDecisionCombiner(double declineThreshold, double challengeThreshold) {
        this.declineThreshold = declineThreshold;
        this.challengeThreshold = challengeThreshold;
    }

    public Decision decide(RuleVerdict ruleVerdict, double modelFraudProbability,
                            RiskTier riskTier) {

        // Hard rule overrides always win, regardless of model score.
        if (ruleVerdict == RuleVerdict.HARD_DECLINE) {
            return new Decision(Outcome.DECLINE, "rule_hard_override", modelFraudProbability);
        }
        if (ruleVerdict == RuleVerdict.HARD_APPROVE) {
            return new Decision(Outcome.APPROVE, "rule_trusted_override", modelFraudProbability);
        }

        // Adjust thresholds by risk tier (e.g., high-value electronics
        // gets stricter thresholds than a low-value grocery purchase).
        double effectiveDeclineThreshold = declineThreshold - riskTier.thresholdAdjustment();
        double effectiveChallengeThreshold = challengeThreshold - riskTier.thresholdAdjustment();

        if (modelFraudProbability >= effectiveDeclineThreshold) {
            return new Decision(Outcome.DECLINE, "model_high_risk", modelFraudProbability);
        } else if (modelFraudProbability >= effectiveChallengeThreshold) {
            return new Decision(Outcome.CHALLENGE, "model_medium_risk", modelFraudProbability);
        }
        return new Decision(Outcome.APPROVE, "model_low_risk", modelFraudProbability);
    }

    public enum Outcome { APPROVE, CHALLENGE, DECLINE }
    public enum RuleVerdict { HARD_DECLINE, HARD_APPROVE, NO_OVERRIDE }

    public record Decision(Outcome outcome, String reasonCode, double fraudProbability) {}

    public interface RiskTier {
        double thresholdAdjustment(); // positive value = stricter effective threshold
    }
}

This is the section that makes this design fundamentally different from most system designs: latency isn’t a “nice to have” metric to optimize after the fact — it’s a hard constraint that shapes every architectural decision from the very start. Let’s break the budget down explicitly.

8.1 Illustrative latency budget (P99, single transaction)

8.2 Timeout & fallback strategy

• Feature store timeout (e.g., 8ms): On timeout, proceed with whatever features are available (possibly stale, cached-elsewhere copies) rather than blocking; log the degradation for monitoring.
• Model serving timeout (e.g., 15ms): On timeout or model-service failure, the circuit breaker trips and the Orchestrator falls back to rules-only decisioning for a cooldown period, accepting a temporarily higher false-negative rate in exchange for guaranteed latency.
• Overall Orchestrator deadline (e.g., 45ms internal budget): If the cumulative elapsed time approaches this ceiling, the Orchestrator short-circuits to a safe default decision (typically a conservative APPROVE for very low transaction amounts, or CHALLENGE for higher amounts) rather than risk a network-level timeout that produces an even worse implicit decline with zero visibility.

10.1 Back-of-the-envelope estimation (million-requests-per-minute scenario)

• Assume 2 million authorization requests/minute at global peak, or roughly 33,000 requests/sec sustained, with bursts several times higher during major shopping events.
• Each Decision Orchestrator instance, if it can sustain roughly 2,000-3,000 requests/sec at sub-100ms latency (a realistic figure for a well-tuned, mostly-I/O-bound service with async parallel fan-out), requires on the order of 12-17 instances at sustained peak, with headroom (typically 2-3×) provisioned for burst and failover — so a fleet sized in the several dozens of instances per region is a reasonable planning number.
• Each request triggers roughly 3-4 downstream calls (feature store, device-risk, rule engine, model serving) run in parallel, meaning the Online Feature Store and Model Serving Cluster must each independently sustain the full 33,000+ QPS, not a fraction of it.
• Regional deployment is essential: routing each authorization to the nearest region (geo-routing at the network/DNS or acquirer-routing level) both reduces network latency (a large chunk of the 100ms budget) and naturally partitions load across independent regional fleets.

10.2 Scaling techniques per layer

10.3 Regional sharding & global consistency trade-off

Because network latency itself consumes a large share of the 100ms budget, this system is deployed as independent regional stacks (their own Decision Orchestrator fleet, feature store replica, model serving cluster) rather than one global cluster. This means a card used in two different regions within a short window relies on cross-region feature replication (typically asynchronous, with a small propagation delay) rather than a single global source of truth queried synchronously — an explicit, deliberate trade-off favoring low latency over perfectly fresh global consistency, discussed further in the CAP theorem context below.

10.4 CAP theorem applied to this design

Under the CAP theorem, a distributed system facing a network partition must choose between strict Consistency and Availability. This design deliberately chooses availability with eventual consistency for the feature layer: a card used first in Tokyo and moments later in London will have its London-region feature lookup reflect the Tokyo transaction only after cross-region replication catches up (typically well under a second, but not instantaneous), rather than the London Decision Orchestrator blocking on a synchronous cross-region read to guarantee perfect freshness. The alternative — synchronous cross-region consistency — would add tens to hundreds of milliseconds of network latency to every single transaction, which is simply incompatible with a 100ms global budget. The small residual risk this trade-off accepts (a fraudster rapidly using a card across two regions within the replication-lag window) is explicitly mitigated by velocity rules and step-up challenges rather than by trying to force strict consistency into a latency budget that cannot afford it.

10.5 Storage & state sizing

• Online Feature Store: With hundreds of millions of active cards, each holding a compact rolling-feature record (velocity counters, recent geography, a handful of aggregates — a few hundred bytes each), total feature-store data volume lands in the range of tens to a few hundred gigabytes per region, comfortably served from memory by a modestly sized Redis Cluster.
• Kafka retention: At roughly 33,000 transaction events/sec and a modest per-event payload (a few hundred bytes), 24 hours of retention on the transaction-event topic is on the order of a few terabytes — well within a standard multi-broker cluster’s capacity, especially with compression enabled.
• Model Serving Cluster sizing: If a single inference-serving replica can sustain a few thousand low-latency predictions per second, sustaining 33,000+ requests/sec per region requires a modest fleet (low double digits of replicas) with generous headroom, since this cluster sits directly in the latency-critical path and should never be run near saturation.

10.6 Handling traffic spikes (holiday shopping style)

• Predictive/pre-warmed scaling: Capacity for the Decision Orchestrator and Model Serving Cluster is scaled ahead of known peak shopping periods rather than purely reactively, since reactive autoscaling can lag a sudden multi-fold spike by the very seconds that matter most for a strict latency SLA.
• Load shedding as an absolute last resort: If the ingestion path is truly saturated beyond provisioned headroom, the system sheds the least-critical non-transactional traffic (e.g., analyst dashboard queries against the Serving API) long before it would ever consider degrading the core authorization-scoring path.
• Elastic feature-store and Kafka partition scaling: Additional Redis Cluster shards and Kafka partitions can be added ahead of a known peak event, spreading load more finely across the fleet rather than relying on a fixed partition count sized for average-day traffic.

• Multi-AZ, multi-region deployment: Every component is deployed across multiple availability zones within a region, and multiple regions globally, so a zone or regional failure degrades capacity rather than causing an outage.
• Fail-open/fail-closed policy per merchant tier: When the Model Serving Cluster or Rule Engine is unavailable beyond its circuit-breaker threshold, the Orchestrator falls back to a pre-defined, tier-specific safe default (e.g., approve low-value transactions from established, low-risk merchants; challenge or decline higher-value/higher-risk ones) rather than blocking indefinitely.
• Graceful degradation of the feature store: If a specific feature lookup times out, the Orchestrator proceeds with a reduced feature set (flagging the decision as “degraded” for monitoring) instead of failing the entire request.
• Chaos testing: Regularly killing Decision Orchestrator instances, Model Serving replicas, and feature store nodes in staging to verify failover and circuit-breaker behavior under realistic conditions before a real incident tests them for the first time.
• Disaster recovery: Regional data replicated asynchronously to a standby region; the Async Event Sink’s Data Lake archive is the ultimate source of truth for full historical reconciliation if a region needs to be rebuilt.

11.1 Failure scenarios worked through

• PCI-DSS compliance: Card data (PAN, CVV) is tokenized at the earliest possible point in the pipeline; the fraud system operates on tokens and hashed/derived identifiers wherever possible, never storing raw card numbers in feature stores or logs.
• Encryption everywhere: TLS for all network hops (merchant-to-acquirer, acquirer-to-network, network-to-issuer, and every internal service-to-service call via mTLS); encryption at rest for the feature store, Decision Cache, and Data Lake.
• Least-privilege access: The Decision Orchestrator and Model Serving Cluster only have read access to the specific feature namespaces they need; the Config/Rules Management Service enforces role-based access so only authorized fraud analysts can modify live rules, with every change immutably audit-logged.
• Rate limiting & abuse protection: The API Gateway enforces per-merchant/per-acquirer rate limits to prevent a compromised or misbehaving upstream integration from overwhelming the pipeline.
• Model security: Model artifacts are versioned, signed, and validated before deployment to the Model Serving Cluster, preventing tampering or accidental deployment of an unvalidated/experimental model to the live decision path.
• Adversarial robustness: Because fraudsters actively probe the system, feature and rule definitions are treated as sensitive — a compromise of the Config/Rules Management Service could let an attacker learn exactly which patterns evade detection, so it’s isolated and access-controlled as carefully as the decisioning path itself.

13.1 Key metrics to track

13.2 Logging & tracing

• Structured, correlation-ID-tagged (transaction ID) logs across every hop, with all sensitive fields redacted/tokenized before they ever leave the service boundary.
• Distributed tracing (OpenTelemetry) spanning API Gateway → Orchestrator → parallel feature/rule/model calls → decision, broken down per stage, so a latency regression can be pinpointed to the exact stage responsible.
• Real-time dashboards showing the latency budget breakdown (as in Section 8’s table) live, per region, so an emerging latency regression in one specific stage is visible immediately rather than only as an aggregate SLA breach.
• Sampled, structured logging for high-cardinality debug details versus unsampled logging for anything tied to a decline or challenge decision, balancing storage cost against the forensic completeness needed for dispute resolution and regulatory inquiries.

13.3 Precision, recall, and the business cost function

Because confirmed fraud labels (chargebacks) arrive weeks after the original decision, this system’s most important quality metrics are necessarily lagging indicators, tracked over rolling multi-week windows rather than in real time: precision (of all declined/challenged transactions, what fraction were truly fraudulent) and recall (of all eventually-confirmed fraud, what fraction was caught at authorization time). These are combined into a single cost-weighted metric reflecting the business’s actual false-decline versus false-approval cost trade-off discussed in Section 7, and that combined metric — not raw model accuracy — is what ultimately governs whether a new model version or threshold change is considered an improvement.

• Containerization + orchestration: The Decision Orchestrator, Rule Engine Service, and Model Serving Cluster are containerized and run on Kubernetes (or similar) for self-healing and rolling deployments, with pod anti-affinity rules ensuring replicas spread across failure domains.
• Regional, multi-cluster deployment: Each geographic region runs its own full stack (Orchestrator fleet, feature store replica, model serving cluster) to minimize network latency and provide regional fault isolation.
• Canary deployment for models and rules: A new model version or rule change is first deployed in shadow mode (scoring live traffic without affecting decisions), then rolled out to a small percentage of real traffic, with automated comparison of key metrics (approve/decline rates, latency) against the incumbent before full rollout.
• Blue-green for the Decision Orchestrator: Given its statelessness and criticality, blue-green deployment allows instant rollback if a new version regresses latency or introduces a bug, without any partial-traffic ambiguity.
• Infrastructure as Code: Terraform (or similar) defines the regional Kubernetes clusters, feature store clusters, and networking, enabling reproducible, auditable environments — particularly important given the regulatory scrutiny financial infrastructure receives.

• Online Feature Store: An in-memory, sharded key-value store (Redis Cluster or a purpose-built low-latency feature-serving system) holding only the recent, hot window of features (e.g., trailing 24-72 hours of velocity counters); older/aggregate features live in a separate offline store used only for training, never queried on the hot path.
• Decision Cache: A short-TTL (minutes, not hours) Redis store keyed by transaction/idempotency ID, purely for retry-safety, not a general-purpose data store.
• Offline/Training Feature Store: A larger, slower data warehouse or data lake storing full historical features and labeled outcomes (confirmed fraud/chargeback data), used exclusively for model training and offline analytics — never in the synchronous decision path.
• Load balancing strategy: L4 load balancing for raw connection distribution across API Gateway instances; L7 load balancing at the gateway for health-check-aware routing to Decision Orchestrator replicas; consistent hashing at the feature store client level to route each card/merchant’s lookups to the correct shard with minimal added latency.

16.1 Sample Decision Orchestrator API (internal)

POST /v1/authorize/score
Request:
{
  "transactionId": "txn_9f2a1c",
  "cardToken": "tok_4242_abcd",
  "merchantId": "merch_88213",
  "amount": 87.50,
  "currency": "USD",
  "cardPresent": true,
  "deviceId": "dev_77a1",
  "timestamp": "2026-08-03T14:32:07.113Z"
}

Response 200 (within 100ms):
{
  "transactionId": "txn_9f2a1c",
  "decision": "APPROVE",
  "reasonCode": "model_low_risk",
  "fraudProbability": 0.014,
  "elapsedMs": 62
}

This is a classic microservices architecture built around clear domain boundaries: the Decision Orchestrator owns the synchronous request/response contract, the Model Serving Cluster and Rule Engine Service are called internally as fast, narrowly-scoped dependencies, and the streaming feature pipeline communicates exclusively through Kafka events rather than direct calls — allowing each to be deployed, scaled, and evolved independently.

16.2 Internal event contract (Transaction Event, simplified)

// TransactionScored event published asynchronously after every decision
message TransactionScored {
  string transaction_id = 1;
  string card_token = 2;
  string merchant_id = 3;
  double amount = 4;
  string decision = 5;
  double fraud_probability = 6;
  int64 timestamp_ms = 7;
}

A schema registry enforces backward/forward compatibility on this event as it evolves, since the streaming feature pipeline, the audit/data-lake sink, and any future consumers all depend on a stable contract without needing to coordinate deployments with the Decision Orchestrator team.

17.1 Patterns used

17.2 Anti-patterns to avoid

18.1 Best practices

• Treat the 100ms budget as a hard architectural constraint from day one, not an optimization target applied after the system is built — it should shape which calls are parallel, which have timeouts, and which have fallbacks.
• Always pair an ML model with a fast, explainable rule layer capable of instant hard overrides for known-bad patterns.
• Version and shadow-test every model and rule change against live traffic before it ever affects a real decision.
• Design an explicit, documented fail-open/fail-closed policy per merchant risk tier, rather than leaving it as an implicit accident of how the code happens to handle timeouts.
• Keep the synchronous decision path’s dependencies to the absolute minimum required — every additional dependency is another way to miss the deadline.

18.2 Common mistakes

• Under-provisioning the Model Serving Cluster’s capacity margin, causing latency to creep up under peak load exactly when fraud attempts (and legitimate volume) both spike.
• Forgetting to test the system’s behavior under partial degradation (one dependency slow, not fully down) — full outages are often easier to handle correctly than “slightly too slow,” which is exactly what a strict latency budget is most vulnerable to.
• Not separating the synchronous decision path’s logging/audit trail from the analytical Data Lake pipeline, risking the audit write itself becoming a latency-critical dependency.
• Retraining models without validating against a held-out, recent time period — fraud patterns shift, and a model validated only on older data can look great in offline metrics while already being stale in production.

18.3 Testing strategy

Beyond ordinary unit and integration tests, this system needs: latency load testing that simulates realistic peak traffic (including bursty patterns like a flash sale) while measuring the full p99.9 latency distribution, not just the average; chaos/fault-injection testing that deliberately slows or fails individual dependencies (feature store, model serving) to verify circuit breakers and fallbacks actually engage correctly under real conditions; shadow-mode validation comparing a candidate model/rule change’s decisions against the incumbent on identical live traffic before any real transaction is affected; and replay testing against a labeled historical dataset (including known confirmed fraud cases) to verify a new model version doesn’t regress on previously-caught fraud patterns.