Imagine you are standing at the checkout counter of an online store. You have a phone in your basket that costs $800. You don’t have $800 sitting in your bank account right now, but you get paid every two weeks. A “Buy Now, Pay Later” (BNPL) button appears next to the normal “Pay with card” button. You tap it. Within a second or two, the screen says “Approved — pay $200 today, and $200 every two weeks for the next three payments.” You tap confirm, and the order goes through.

That entire experience — from tapping the button to seeing “Approved” — usually takes less than a second on the checkout page, even though behind the scenes a huge amount of work happened: verifying who you are, pulling your credit history, calculating a risk score, checking if you already have too many open BNPL loans, deciding how much to lend you, and locking in that decision before the merchant’s page times out.

This is exactly the kind of system this tutorial teaches you to design. We will build a real-time credit decisioning system that sits behind a “Pay in 4” or “Pay in 3” button on a partner e-commerce site, and that has to survive traffic spikes like Black Friday, flash sales, and viral product drops — scenarios where the system may need to process millions of decision requests within a single minute.

A short history of BNPL

Installment payment plans are not new — furniture and appliance stores have offered “layaway” and store credit for over a century. What changed in the last decade is speed and integration. Companies like Klarna (Sweden, founded 2005), Afterpay (Australia, 2014), Affirm (USA, 2012), and later PayPal’s “Pay in 4” turned what used to be a multi-day loan-application process (fill a paper form, wait for a bank to call you) into something that happens instantly, embedded directly inside the checkout flow of thousands of online stores. This was made possible by three technology shifts: (1) cheap access to alternative data sources (bank transaction data, telecom data, e-commerce history) beyond traditional credit bureaus, (2) machine learning models that can score risk in milliseconds instead of the days a human underwriter needs, and (3) modern distributed systems (like the ones we design in this tutorial) that can serve those models at massive scale with very low latency.

Why this matters beyond BNPL

Even if you never build a lending product yourself, the patterns in this tutorial generalize to almost any system that must make a high-stakes, low-latency decision under uncertainty at massive scale — fraud detection during card payments, ad-auction bidding within a 100ms window, dynamic pricing at checkout, or real-time insurance quoting. The core tension is always the same: you need enough information to make a good decision, but you don’t have time to gather all of it synchronously, so the entire architecture is organized around answering “what is the minimum information, computed ahead of time and cached, that lets us decide fast — with a safe fallback for everything else?”

Let’s define the problem precisely, the way you would in a system design interview.

The ask

Why this is hard

• Latency budget is brutal. The merchant’s checkout page calls our API and blocks on the response. Shoppers expect an answer in under 1 second end-to-end; anything above ~2 seconds causes measurable cart abandonment.
• Financial correctness is non-negotiable. Every approved loan must be recorded exactly once (no double-lending), every risk decision must be explainable (regulators require this — “why was I declined?”), and money movement must be auditable years later.
• Extreme, spiky traffic. Traffic is not steady. A celebrity product drop, a Black Friday sale, or a flash sale on a large partner site can push demand from a normal 20,000 requests/minute baseline to over a million requests/minute within seconds.
• Fraud and abuse pressure. Because BNPL effectively hands out short-term unsecured credit, it’s a magnet for stolen identities, synthetic identities, and “first payment default” fraud rings.
• Regulatory constraints. Depending on jurisdiction (USA, EU, UK, Australia), BNPL is increasingly regulated like consumer credit — this means mandatory affordability checks, credit bureau “soft pulls,” data retention rules, and adverse-action notices explaining declines.

Functional requirements

1. Given a shopper, a merchant, and a cart amount, return an approve/decline/step-up decision in real time.
2. If approved, return the installment plan (e.g., 4 payments of $200 every 2 weeks) and a one-time checkout token the merchant uses to finalize the order.
3. Support multiple partner merchants, each with different loan products, minimum/maximum amounts, and business rules.
4. Prevent duplicate loans if the merchant retries the same checkout request (idempotency).
5. Support step-up verification (e.g., OTP, ID check) for borderline-risk shoppers instead of an instant decline.
6. Persist an auditable, explainable trail of every decision for compliance and dispute-resolution.

Non-functional requirements

Before drawing boxes and arrows, let’s build a shared vocabulary. Every term below is explained with a real-life analogy, a beginner example, and how it appears in this system.

Credit decisioning / underwriting

What: The process of deciding whether to lend money to someone, and how much.
Analogy: Imagine a librarian deciding whether to lend you a rare book. She checks: have you returned books late before? Do you already have 5 books checked out? Is this book especially valuable? Based on all of that, she decides: lend it fully, lend it with a deposit, or say no.
In our system: A “decision engine” plays the librarian’s role — it looks at the shopper’s history, current exposure, and the specific “book” (loan amount) being requested, and returns approve/decline/step-up.

Risk score

What: A number (say, 0–1000) representing how likely a shopper is to repay this loan.
Beginner example: Think of a school grade — a score of 900 means “very likely to pay back,” a score of 300 means “risky.”
Software example: A machine learning model (e.g., gradient-boosted trees or a logistic regression) trained on millions of past loans outputs this score in a few milliseconds.
Production example: Affirm and Klarna both use proprietary ML models combining bureau data, bank-transaction data, and behavioral signals (like typing speed or device fingerprint) to compute this score.

Idempotency

What: Making sure that performing the same operation multiple times has the same effect as doing it once.
Analogy: Pressing an elevator call button five times doesn’t call five elevators — the button “remembers” it’s already been pressed.
In our system: If the merchant’s checkout page has a network glitch and retries the same “approve this loan” request twice, we must not create two separate loans for the shopper.

Soft pull vs. hard pull (credit bureau)

What: A “hard pull” on your credit report shows up to other lenders and can slightly lower your credit score; a “soft pull” doesn’t. BNPL providers almost always use soft pulls for speed and to avoid discouraging shoppers.

Ledger

What: An append-only, tamper-evident record of every financial event (loan created, payment made, refund issued).
Analogy: A bank’s paper ledger book from the 1800s — you never erase an entry, you only add new ones (like a correction entry) so there’s always a full history.

Feature store

What: A specialized low-latency database that stores pre-computed “features” (inputs) for machine learning models, such as “number of open BNPL loans across all providers” or “average order value in last 90 days,” so the model doesn’t need to compute them from scratch on every request.

Circuit breaker

What: A safety switch that stops calling a failing downstream service for a while, instead of hammering it and making things worse.
Analogy: A home electrical circuit breaker trips and cuts power when there’s a short circuit, protecting the house from fire.

Consistent hashing

What: A technique for distributing data (or requests) across many servers so that when a server is added or removed, only a small fraction of keys need to move — instead of almost all of them, as happens with simple modulo hashing.
Analogy: Imagine a circular clock face with numbers 0–360. Each database shard “owns” an arc on that clock. A shopper’s ID is hashed onto a point on the clock, and whichever shard’s arc contains that point owns their data. Adding a new shard only steals a small arc from its neighbor, instead of reshuffling everyone.
In our system: Used both for sharding the Ledger DB by shopper_id and for distributing Kafka partitions, so scaling out shards or partitions doesn’t require a massive, disruptive data migration.

CAP theorem and where we land

What: The CAP theorem says that during a network partition, a distributed system must choose between Consistency (every read sees the latest write) and Availability (every request gets a response, even if it might be slightly stale). You cannot have perfect versions of both at the same time during a partition.
In our system: We deliberately make different choices for different data. The Ledger DB is CP (consistency-first) — we would rather briefly reject a loan-creation request than risk creating two conflicting loans for the same shopper. The Feature Store is AP (availability-first) — it is fine to serve a feature that is a few seconds stale rather than block the entire decision on a database being perfectly up to date, because the ML model’s risk threshold already has a safety margin built in for exactly this kind of small staleness.

Optimistic concurrency control

What: Instead of locking a row every time it might be updated (pessimistic locking, which hurts throughput under load), the system reads a row along with a version number, and the write only succeeds if the version hasn’t changed since the read — otherwise it retries.
Analogy: Think of editing a shared Google Doc offline — when you reconnect, if someone else already changed that exact sentence, your edit is rejected and you re-apply it against the latest version instead of silently overwriting their work.
In our system: Used when updating a shopper’s “total open exposure” counter in the Feature Store, since many concurrent checkouts across different merchants might try to update it at once.

Token bucket algorithm (rate limiting)

What: A classic algorithm where each merchant has a “bucket” that refills with tokens at a fixed rate; each incoming request consumes one token, and if the bucket is empty, the request is rejected or queued. This allows short bursts (up to the bucket size) while enforcing a steady long-term rate.
In our system: The API Gateway uses per-merchant token buckets, implemented in Redis using atomic INCR/EXPIRE commands, so a single misbehaving merchant integration (e.g., an infinite retry loop) cannot starve capacity from every other merchant sharing the platform.

@Component
public class TokenBucketRateLimiter {

    private final StringRedisTemplate redis;

    public boolean tryAcquire(String merchantId, int capacity, int refillPerSecond) {
        String key = "ratelimit:" + merchantId;
        long now = System.currentTimeMillis() / 1000;

        // Lua script executed atomically in Redis to avoid race conditions
        String script =
            "local tokens = tonumber(redis.call('GET', KEYS[1]) or ARGV[1]) " +
            "if tokens > 0 then " +
            "  redis.call('DECRBY', KEYS[1], 1) " +
            "  redis.call('EXPIRE', KEYS[1], 2) " +
            "  return 1 " +
            "else return 0 end";

        Long allowed = redis.execute(
            RedisScript.of(script, Long.class),
            Collections.singletonList(key),
            String.valueOf(capacity));

        return allowed != null && allowed == 1;
    }
}

Consensus and leader election

What: When multiple nodes need to agree on a single source of truth (e.g., “which node is currently the primary writer for this database shard”), consensus algorithms like Raft or Paxos ensure they agree even if some nodes crash or messages are delayed.
In our system: Each Ledger DB shard’s primary/replica failover is managed by a consensus-based coordination layer (e.g., Patroni running on top of etcd, which itself uses Raft), so that if a shard’s primary node crashes, the remaining replicas agree on a new primary within seconds without any risk of two nodes believing they are both the primary at once (a dangerous “split-brain” scenario that could allow conflicting writes).

Now let’s design the system. We’ll build it up in layers: edge/traffic layer, decisioning layer, data layer, and asynchronous/ledger layer. Every box below explicitly states which architectural layer it belongs to (API Gateway, Load Balancer, Service, Cache, Database, Queue, etc.).

Component responsibilities

Detailed decision-path sequence

The Checkout Orchestrator

The orchestrator is a stateless service — meaning it holds no important data in its own memory between requests, so any instance can handle any request, and instances can be added or removed freely by the load balancer. It receives the incoming decision request, validates it against a JSON schema, generates or looks up an idempotency key, and then fans out calls to the Identity Service and Risk Engine, applying strict per-call timeouts (e.g., 150ms for identity, 300ms for risk scoring) so that a slow downstream call can’t blow the overall latency budget.

@Service
public class CheckoutOrchestratorService {

    private final IdentityClient identityClient;
    private final RiskEngineClient riskEngineClient;
    private final LedgerClient ledgerClient;
    private final KafkaEventPublisher eventPublisher;
    private final RedisIdempotencyStore idempotencyStore;

    private static final Duration IDENTITY_TIMEOUT = Duration.ofMillis(150);
    private static final Duration RISK_TIMEOUT = Duration.ofMillis(300);

    public DecisionResponse decide(DecisionRequest request) {
        // 1. Idempotency check - never process the same checkout twice
        Optional<DecisionResponse> cached =
            idempotencyStore.get(request.getIdempotencyKey());
        if (cached.isPresent()) {
            return cached.get(); // safe to return identical prior result
        }

        // 2. Identity verification with a hard timeout
        IdentityResult identity = identityClient
            .verify(request.getShopperId(), request.getDeviceSignal())
            .orTimeout(IDENTITY_TIMEOUT.toMillis(), TimeUnit.MILLISECONDS)
            .exceptionally(ex -> IdentityResult.degraded())
            .join();

        if (identity.isBlocked()) {
            return DecisionResponse.decline("IDENTITY_BLOCKED");
        }

        // 3. Risk scoring with a hard timeout and graceful fallback
        RiskDecision risk;
        try {
            risk = riskEngineClient
                .score(request, identity)
                .orTimeout(RISK_TIMEOUT.toMillis(), TimeUnit.MILLISECONDS)
                .get();
        } catch (TimeoutException | ExecutionException e) {
            // Fail safe: fall back to a conservative rules-only decision
            risk = riskEngineClient.fallbackRulesOnlyDecision(request);
        }

        if (!risk.isApproved()) {
            DecisionResponse decline = DecisionResponse.decline(risk.getReasonCode());
            idempotencyStore.put(request.getIdempotencyKey(), decline);
            return decline;
        }

        // 4. Atomic loan creation - the only strongly-consistent write on the hot path
        LoanRecord loan = ledgerClient.createLoanIdempotent(
            request.getIdempotencyKey(), request, risk.getApprovedTerms());

        DecisionResponse response = DecisionResponse.approve(loan);
        idempotencyStore.put(request.getIdempotencyKey(), response);

        // 5. Fire-and-forget async event, does not block the response
        eventPublisher.publishAsync(new LoanApprovedEvent(loan));

        return response;
    }
}

The Risk Decision Engine

The risk engine’s job is to answer “approve, decline, or step-up?” plus “how much, and on what schedule?” within roughly 300ms. It combines three signal sources: (1) a pre-trained ML model served from an in-memory model server, (2) hard business rules (“never lend more than $2,000 to a first-time shopper”), and (3) a real-time bureau soft-pull that is called in parallel but has a strict timeout and a safe fallback if it doesn’t return in time.

@Service
public class RiskDecisionEngine {

    private final FeatureStoreClient featureStore;
    private final MLModelServingClient modelClient;
    private final BureauGatewayClient bureauClient;
    private final MerchantRuleEngine ruleEngine;

    public CompletableFuture<RiskDecision> score(DecisionRequest req, IdentityResult id) {
        // Fetch pre-computed features (cache hit ~ 5ms, avoids recomputation)
        CompletableFuture<FeatureVector> features =
            featureStore.getFeatures(req.getShopperId());

        // Call the credit bureau in parallel, not sequentially
        CompletableFuture<BureauSignal> bureau =
            bureauClient.softPullAsync(req.getShopperId())
                .completeOnTimeout(BureauSignal.unavailable(), 250, TimeUnit.MILLISECONDS);

        return features.thenCombine(bureau, (fv, bs) -> {
            fv = fv.withBureauSignal(bs);

            // Hard rules run first - cheap, deterministic, and auditable
            RuleResult ruleResult = ruleEngine.evaluate(req, fv);
            if (ruleResult.isHardDecline()) {
                return RiskDecision.decline(ruleResult.getReasonCode());
            }

            // ML model produces a probability-of-default score
            double pd = modelClient.predictProbabilityOfDefault(fv);

            if (pd < req.getMerchantConfig().getApproveThreshold()) {
                InstallmentPlan plan = buildInstallmentPlan(req.getCartAmount(), pd);
                return RiskDecision.approve(plan, pd);
            } else if (pd < req.getMerchantConfig().getStepUpThreshold()) {
                return RiskDecision.stepUp("ADDITIONAL_VERIFICATION_REQUIRED");
            }
            return RiskDecision.decline("HIGH_RISK_SCORE");
        });
    }
}

Let’s trace one checkout end-to-end, matching it to the sequence diagram in the previous section.

1. Widget render: The merchant’s checkout page loads our BNPL widget via a small JS snippet served from the CDN. This shows the “Pay in 4” option with an estimated plan.
2. Decision request: When the shopper clicks “Continue with Pay in 4,” the merchant’s backend (or our JS widget) calls POST /v1/checkout/decision with an idempotency key generated once per checkout attempt.
3. Gateway validation: The API Gateway authenticates the merchant, checks per-merchant rate limits, and validates the request shape before it ever reaches application logic.
4. Orchestration: The Checkout Orchestrator fans out to Identity and Risk services in parallel where possible, respecting strict timeouts.
5. Feature lookup: The Risk Engine reads pre-computed features (e.g., “shopper’s total open BNPL exposure across merchants”) from the low-latency Feature Store rather than querying multiple databases live.
6. Bureau soft-pull: A parallel, time-boxed call to a credit bureau adds an external signal without becoming a single point of failure — if it’s slow, we fall back to a conservative model-only decision.
7. Decision: The Risk Engine returns approve (with terms), decline (with a reason code, required by law in many regions), or step-up (extra verification).
8. Atomic loan write: On approval, the Ledger Service performs a single atomic, idempotent write creating the loan and its payment schedule — this is the only step requiring strong consistency.
9. Response: The orchestrator returns the decision and a one-time checkout token to the merchant, who finalizes the order.
10. Async fan-out: A LoanApproved event is published to Kafka, triggering notification (SMS/email), deeper async fraud scoring, and a write to the data warehouse — none of which block the shopper.

The installment payment lifecycle (after approval)

Approval is only the beginning of a loan’s life. Once the shopper’s order is placed, the Ledger Service schedules the remaining installment payments (e.g., three future biweekly charges) as rows in a scheduled_payments table, each tied to a payment method on file. A separate, lower-throughput Payment Collection Service — deliberately decoupled from the real-time decision path, since it operates on a schedule rather than in response to shopper action — polls for due payments and initiates charges through a payments processor (e.g., card networks or bank debit rails). If a scheduled charge fails (insufficient funds, expired card), the system does not simply give up: it applies a configurable dunning strategy — a series of retries with increasing delay, paired with shopper notifications — before marking the loan delinquent and, only as a last resort, referring it to collections. This entire lifecycle reuses the same Ledger DB and event-sourcing approach as the initial approval, so a loan’s complete history (approved, first payment collected, second payment retried once, third payment collected) is always fully reconstructable from the append-only event log, which matters enormously when a shopper disputes a charge or a regulator asks for a specific loan’s full history.

This is the heart of the “million requests a minute” requirement. Let’s work through it layer by layer.

Capacity math

1,000,000 requests/minute ≈ 16,667 requests/second average. During a flash-sale spike, instantaneous throughput can be 3–5x that, so we design for roughly 60,000–80,000 requests/second at peak.

Bulkhead pattern

By isolating each downstream dependency into its own bounded thread pool (“bulkhead,” named after the watertight compartments in a ship’s hull), a slow bureau can never starve the threads needed to serve identity or risk calls.

Caching strategy

• Feature Store (Redis/DynamoDB): Pre-computed shopper risk features refreshed by a streaming pipeline (Kafka → feature computation → cache write), so the hot path only ever reads, never computes on the fly.
• Merchant Config Cache: Per-merchant business rules are rarely-changing; cached locally in each service instance with a short TTL and pub/sub invalidation on change.
• Idempotency Cache: Redis with a TTL of ~24 hours stores the result of each idempotency key so retried requests return instantly without redoing work.

Multi-region active-active

The decision path is deployed active-active across at least two regions. The Global Load Balancer routes shoppers to the nearest healthy region, and each region has its own full stack (gateway, orchestrator, risk engine, feature store replica). The Ledger DB uses regional shards with asynchronous cross-region replication for disaster recovery, while a shopper’s “home region” (based on their first interaction) is the source of truth for their loans to avoid conflicting concurrent writes.

Graceful degradation ladder

When things go wrong, the system should degrade in a controlled, well-tested order rather than failing all at once:

1. Bureau unavailable → fall back to model + rules-only decision (slightly more conservative)
2. ML model server unavailable → fall back to a simpler, pre-validated rules engine (“safe mode”)
3. Feature store stale/unavailable → use last-known-good cached features with a wider safety margin
4. Everything unavailable → decline with a clear “temporarily unavailable, please try card payment” message rather than hanging

Disaster recovery & backup

Because the Ledger DB holds legally binding financial records, its backup strategy is stricter than a typical application database. We take continuous write-ahead-log (WAL) shipping to durable object storage in a separate region, plus daily full snapshots retained for at least seven years to satisfy financial record-keeping regulations in most jurisdictions. Recovery Point Objective (RPO) — the maximum acceptable amount of data loss measured in time — is targeted at under 5 seconds thanks to synchronous replication within a region and near-real-time asynchronous replication across regions. Recovery Time Objective (RTO) — how quickly the system must be back up — is targeted at under 5 minutes for a single-region failure, achieved by the Global Load Balancer’s automatic failover plus warm standby capacity already running in the secondary region (not cold-started on demand, since spinning up a fresh regional stack under financial-grade compliance controls can take much longer than 5 minutes).

Failure recovery for in-flight requests

If the Checkout Orchestrator crashes mid-decision, the shopper simply sees a timeout and the merchant’s client retries with the same idempotency key. Because the Ledger write is the only durable state-changing step, and it either fully committed or didn’t (thanks to the database’s atomicity guarantees), a retry safely resumes the flow: the idempotency cache either returns the already-completed result, or the request is re-processed from scratch with no partial, corrupted state left behind. This is a deliberate design choice — we intentionally avoided multi-step distributed transactions across services on the hot path (which are hard to recover from correctly) in favor of a single atomic write plus safe retries.

Cost optimization at scale

Running enough capacity to handle a 60,000+ req/sec peak 24/7 would be extremely wasteful, since real average traffic is a small fraction of that. Cost is controlled through: (1) aggressive autoscaling with fast scale-up (seconds) but slower, dampened scale-down (minutes) to avoid flapping; (2) using cheaper spot/preemptible compute for stateless, easily-restartable services like the orchestrator, while keeping the Ledger DB and Feature Store on stable, reserved capacity; (3) caching aggressively so that repeat or idempotent requests never reach the more expensive ML model-serving tier; and (4) right-sizing the ML model itself — a slightly smaller, faster model that runs cheaply on CPU can sometimes be preferable to a marginally more accurate model that requires expensive GPU serving, if the latency and cost difference outweighs the small accuracy gain.

• mTLS between services: All internal service-to-service traffic uses mutual TLS so a compromised pod can’t impersonate another service.
• Tokenization of PII: Sensitive fields (SSN, bank account numbers) are tokenized at ingestion; only the token flows through most services, and only a narrowly-scoped vault service can de-tokenize.
• Merchant API key + HMAC request signing: Prevents a stolen API key alone from being enough — requests must be signed with a secret only the merchant and gateway know.
• Rate limiting & bot detection: Per-merchant and per-shopper rate limits, plus device fingerprinting, guard against credit-stuffing and synthetic-identity attack rings.
• Encryption at rest: Ledger DB and feature store encrypted at rest with region-specific keys (data residency compliance).
• Adverse action logging: Every decline is logged with the exact reason code and model version used, both for regulatory “right to explanation” requirements and for later audits.

Circuit breaker implementation detail

Since the Bureau Gateway is the least reliable, most latency-variable dependency, its circuit breaker deserves a closer look. It operates in three states — Closed (calls flow normally), Open (calls fail instantly without touching the network), and Half-Open (a small trickle of test calls checks if the dependency has recovered) — a design popularized by Netflix’s Hystrix and now common in libraries like Resilience4j.

@Configuration
public class BureauCircuitBreakerConfig {

    @Bean
    public CircuitBreaker bureauCircuitBreaker() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)                 // trip if 50% of calls fail
            .slowCallRateThreshold(50)                 // also trip on slow calls
            .slowCallDurationThreshold(Duration.ofMillis(250))
            .waitDurationInOpenState(Duration.ofSeconds(10))
            .permittedNumberOfCallsInHalfOpenState(5)  // test with 5 calls
            .slidingWindowSize(100)
            .build();

        return CircuitBreaker.of("bureauGateway", config);
    }
}

With this configuration, if half of the last 100 bureau calls either failed or took longer than 250ms, the breaker trips open for 10 seconds — during which every request instantly falls back to the model-plus-rules-only decision path instead of wasting time waiting on a dependency that’s already known to be struggling. After 10 seconds, a handful of test requests are allowed through to see if the bureau has recovered before fully re-opening the floodgates.

Idempotency store implementation detail

@Component
public class RedisIdempotencyStore {

    private final StringRedisTemplate redis;
    private final ObjectMapper mapper;
    private static final Duration TTL = Duration.ofHours(24);

    public Optional<DecisionResponse> get(String idempotencyKey) {
        String raw = redis.opsForValue().get("idem:" + idempotencyKey);
        if (raw == null) return Optional.empty();
        try {
            return Optional.of(mapper.readValue(raw, DecisionResponse.class));
        } catch (JsonProcessingException e) {
            return Optional.empty(); // fail open to reprocessing, never fail closed
        }
    }

    public void put(String idempotencyKey, DecisionResponse response) {
        try {
            String raw = mapper.writeValueAsString(response);
            // SET NX ensures we never overwrite a concurrently-written result
            redis.opsForValue().setIfAbsent(
                "idem:" + idempotencyKey, raw, TTL);
        } catch (JsonProcessingException ignored) { }
    }
}

Using SET NX (set-if-not-exists) rather than a plain SET matters here: if two requests carrying the same idempotency key somehow race each other (e.g., a merchant’s client double-clicks the buy button and fires two near-simultaneous calls), only the first write wins, and the second request’s result is discarded in favor of the first — guaranteeing a single, consistent answer for that key rather than a “last write wins” scenario that could return a different result to a legitimately identical retry.

Distributed tracing (e.g., OpenTelemetry) tags every request with a trace ID from the API Gateway through every downstream call, so a single slow request can be visually reconstructed across all services — critical when hunting for the source of a latency spike at 60,000 req/sec.

Structured, correlated logging

Every log line emitted anywhere in the decision path includes a shared trace_id, the merchant_id, a tokenized (never raw) shopper reference, and the specific service/component name, all as structured JSON fields rather than free-text — this makes it possible to run precise queries like “show me every log line for this one failed checkout across all six services it touched” in a log aggregation system (e.g., the ELK stack or a managed equivalent) without regex-parsing free text. Decision-relevant logs (which rule fired, which model version scored the request, the final score) are retained separately, for much longer, and with stricter access controls than general operational debug logs, since they form part of the regulatory audit trail.

Alerting philosophy

Alerts are tiered by urgency: page-a-human-immediately alerts are reserved for symptoms that directly affect shoppers right now (P99 latency breach, approval rate cratering, error rate spike), while informational alerts (a single instance’s memory creeping up, a slow but non-breaching trend) route to a dashboard reviewed during business hours. This tiering prevents alert fatigue — a well-known failure mode where too many low-urgency pages train engineers to start ignoring alerts altogether, which is far more dangerous than having too few alerts.

Compliance & regulatory considerations

BNPL sits at the intersection of e-commerce and consumer lending, and regulatory scrutiny has increased significantly as the industry has grown. A production-grade design has to bake compliance in as a first-class architectural concern, not an afterthought:

• Adverse action notices: In many jurisdictions, a shopper who is declined has a legal right to know why in plain language (“insufficient credit history,” “existing balance too high”). The Risk Engine’s reason codes are designed from day one to map to a small, pre-approved, legally-reviewed set of human-readable explanations rather than raw internal model feature names.
• Affordability / responsible lending checks: Regulators increasingly require lenders to make a reasonable assessment that a shopper can actually afford the repayments, not just that they’re statistically likely to eventually pay. This is why total open exposure across all merchants (not just the current one) is a first-class feature in the risk model.
• Data residency: Shopper financial data for EU shoppers, for example, may be legally required to stay within EU data centers. The multi-region active-active design is built with this in mind — a shopper’s “home region” is determined partly by data residency requirements, not purely by network latency.
• Right to erasure / data retention conflicts: Privacy regulations like GDPR grant a “right to be forgotten,” but financial record-keeping laws simultaneously require retaining loan records for years. The system resolves this by tokenizing and eventually anonymizing personally-identifying fields once they’re no longer needed for active servicing, while retaining the financial transaction facts themselves (amounts, dates, outcomes) in a form stripped of directly identifying information.

• Containerized microservices orchestrated by Kubernetes, each with its own horizontal pod autoscaler tuned to the service’s bottleneck resource (CPU for orchestrator, memory for feature caching, GPU/CPU for model serving).
• Canary + blue-green deployments for the Risk Engine specifically — a new model version is rolled out to 1% of traffic, its approval rate and score distribution compared against the baseline, before wider rollout.
• Pre-scaling for known events: Ahead of Black Friday or a partner’s announced flash sale, capacity is pre-warmed rather than relying purely on reactive autoscaling, since ML model servers and DB connection pools take time to spin up.
• Infrastructure as Code (Terraform) ensures every region’s stack is identical and reproducible, which matters enormously for a regulated financial system that must pass audits.

Ledger DB sharding

The Ledger DB is sharded by hash(shopper_id) across many physical shards. This keeps all of one shopper’s loans on a single shard (so a shopper’s own history is always strongly consistent and fast to read) while spreading total load evenly across the cluster. Each shard is a classic relational database (e.g., PostgreSQL) chosen specifically because loan creation needs ACID transactions — you cannot afford a “maybe” when money is involved.

Load balancing algorithm choice

The regional load balancer uses least-outstanding-requests rather than simple round-robin, because decision requests have variable latency (a bureau timeout can make one request take 5x longer than another); routing new requests away from already-busy instances keeps tail latency lower under load.

Concurrency control on the loan write

Loan creation is a classic “read the shopper’s current exposure, decide, then write” sequence — which is vulnerable to a race condition if two checkout requests for the same shopper arrive within milliseconds of each other (e.g., two browser tabs). We guard against this with a short-lived, per-shopper advisory lock (or a unique database constraint on an in-flight “pending decision” row) so that a second concurrent request for the same shopper is queued or rejected rather than allowed to read stale exposure data and over-approve. This is a deliberately narrow, short-duration lock — held only for the milliseconds of the write itself — so it does not become a throughput bottleneck across the wider system, since two different shoppers never contend for the same lock.

Networking considerations

At tens of thousands of requests per second, connection-level overhead matters. All internal service-to-service calls use persistent, pooled HTTP/2 or gRPC connections instead of opening a new TCP+TLS connection per request — TLS handshakes are computationally expensive and, multiplied across millions of requests, would themselves become a bottleneck. DNS lookups for internal services are avoided on the hot path by using a service mesh (e.g., Envoy sidecars) that maintains warm connection pools to healthy instances, refreshed in the background rather than per-request.

Algorithms & data structures used under the hood

A few classic data structures and algorithms quietly do a lot of work in this system, and interviewers often like to probe on them directly:

• Bloom filter: Before hitting the Feature Store at all, a compact, probabilistic Bloom filter can answer “has this shopper ID ever been seen before?” in O(1) time and a tiny memory footprint, letting the system instantly route brand-new shoppers to a “thin file” underwriting path without an expensive cache lookup.
• LRU cache eviction: The local, in-process merchant-config cache inside each service instance uses a Least-Recently-Used eviction policy, so rarely-used merchant configs are dropped first when memory pressure rises, keeping hot merchants’ configs always in memory.
• Sliding window counters: Rather than a simple fixed-window rate limit (which allows a burst of 2x the limit right at a window boundary), a sliding-window-log or sliding-window-counter algorithm is used for the more security-sensitive per-shopper velocity checks (e.g., “how many checkout attempts in the last 60 seconds”), giving smoother, harder-to-game limits.
• Priority queues: The async Kafka consumer for notifications uses a priority mechanism so that time-sensitive events (e.g., a step-up verification code) are processed ahead of lower-priority ones (e.g., a marketing follow-up email) even when the consumer is temporarily backlogged.
• Union-Find (Disjoint Set): Used offline by the Fraud Analytics Service to efficiently cluster related identities (shared devices, shared payment instruments, shared shipping addresses) into fraud rings for investigation, since union-find can merge and query connected groups extremely efficiently across millions of edges.

Testing strategy

Because incorrect behavior here has direct financial and regulatory consequences, testing goes well beyond typical unit tests. Contract tests verify that every internal service honors its API schema so that independently-deployed teams never silently break each other. A dedicated “shadow traffic” pipeline mirrors a sample of real production decision requests to a candidate new version of the Risk Engine without ever returning its answer to a real shopper, letting engineers compare its approval decisions against the live system’s at full production scale before it ever affects a real loan. Deterministic replay tests feed a fixed, versioned set of historical requests through the pipeline on every deploy, asserting that outputs match expected golden results unless a change was explicitly intended — catching accidental regressions in rules or feature computation immediately.

POST /v1/checkout/decision
Headers:
  Authorization: Bearer <merchant_api_key>
  Idempotency-Key: <uuid>
Body:
{
  "merchant_id": "merchant_123",
  "shopper": { "email": "hashed_or_tokenized", "device_signal": "..." },
  "cart_amount": 800.00,
  "currency": "USD",
  "product_category": "electronics"
}

Response (200 OK):
{
  "decision": "APPROVED",
  "loan_id": "loan_9f21ac",
  "installment_plan": {
    "num_installments": 4,
    "amount_per_installment": 200.00,
    "frequency": "BIWEEKLY"
  },
  "checkout_token": "chk_tok_88a1",
  "expires_in_seconds": 600
}

Response (200 OK - Decline):
{
  "decision": "DECLINED",
  "reason_code": "HIGH_RISK_SCORE",
  "adverse_action_notice_url": "https://.../notice/abc123"
}

Each internal microservice exposes a narrow, single-purpose gRPC or REST API (Identity, Risk, Ledger, Bureau Gateway) rather than one large monolithic API, so teams can deploy, scale, and version each independently — the Risk Engine team, for instance, can ship new model versions weekly without touching the Ledger Service at all.

Saga pattern in detail

The Saga pattern deserves a closer look since it appears twice in this design — once for refunds, and conceptually for any multi-step money movement that can’t be wrapped in a single database transaction because it spans multiple services or external systems (like a payments processor). Rather than a distributed transaction (which requires all participants to be locked and available simultaneously — impractical across an external payments network), a saga breaks the operation into a sequence of local transactions, each with a corresponding compensating action that can undo it if a later step fails. For a refund: step one reverses the ledger balance locally (compensable by re-applying the balance), step two calls the payments processor to move money back to the shopper (compensable by re-charging, if ever needed), and step three notifies the shopper (not compensable, but also not something that needs undoing). If step two fails after step one succeeded, the saga’s coordinator automatically triggers the compensating action for step one, keeping the system in a consistent state rather than a stuck, half-completed one.

Across all of these companies, a recurring theme is that the decisioning latency budget shrank over time as competition increased — an approval that took 3–5 seconds a decade ago is now expected to complete in under a second, which has pushed the whole industry toward the caching, pre-computation, and asynchronous-everything-except-the-decision patterns described throughout this tutorial.

Handling Black Friday-scale traffic: a worked example

Consider a large partner retailer running a flash sale where traffic jumps from a 20,000 req/min baseline to 1,200,000 req/min within two minutes as a doorbuster deal goes live. Here is how each layer responds: the CDN absorbs the surge in static widget asset requests without any impact on our backend at all. The API Gateway’s per-merchant token buckets smooth out the burst somewhat, but the majority of the surge is legitimate demand that must be served. Horizontal pod autoscaling on the Checkout Orchestrator and Risk Engine, pre-warmed an hour before the known sale start time (based on the merchant notifying us in advance, a common real-world practice), scales from a baseline pool to several times that size within the first 30–60 seconds. The Feature Store, being read-heavy and already replicated, absorbs the extra read load easily. The single potential bottleneck — the Bureau Gateway’s external dependency — is protected by its bulkhead and circuit breaker; if bureau response times degrade under the industry-wide load of a major shopping event, the circuit breaker trips and the system gracefully shifts to model-plus-rules-only decisions for a period, trading a small amount of approval precision for continued availability, which is the right trade-off during a peak revenue event for the merchant.

Refunds and cancellations

Not every checkout is final — shoppers cancel orders, and merchants issue refunds. Because a loan and its payment schedule are already recorded in the ledger by the time a cancellation happens, a refund cannot simply “undo” the original write; instead, it is handled as a compensating transaction (an application of the Saga pattern mentioned in the previous section) — a new ledger entry that reverses the outstanding balance and, if the shopper had already made a payment, triggers a real money transfer back to them through the payments processor. This event flows through the same asynchronous Kafka pipeline as loan approval, ensuring refunds are auditable with the same rigor as the original loan.

Why not just always call the credit bureau synchronously and wait?

Because bureau APIs are third-party dependencies with variable and sometimes high latency; making them a hard synchronous dependency for every request would make our P99 latency match theirs, which is unacceptable for a checkout experience.

How do you prevent a shopper from getting multiple simultaneous BNPL loans they can’t afford?

The Feature Store maintains a near-real-time view of a shopper’s total open exposure (updated via the async event pipeline after every approval), and the Risk Engine factors this into both the ML score and hard rules before approving a new loan.

What happens during a total regional outage?

The Global Load Balancer detects the region as unhealthy via health checks and routes all traffic to the remaining healthy region(s); the async replication of the ledger ensures the failover region has near-real-time data, though a small window of very recent loans may need reconciliation once the failed region recovers.

Why solely shard by shopper_id and not by merchant_id?

Sharding by shopper_id keeps a single shopper’s full loan history and exposure on one shard, which is what the risk decision needs to read quickly and consistently; sharding by merchant would spread one shopper’s history across many shards, making risk lookups slower and more complex.

How is the ML model kept accurate over time as fraud patterns and shopper behavior change?

Every decision (and its eventual real-world outcome — did the shopper repay on time, default, or commit fraud) flows into the Data Warehouse via the async event pipeline, forming a continuously growing labeled training set. Models are retrained on a regular cadence, evaluated offline against a held-out dataset, and only promoted to production through the canary rollout process described earlier, so a regression in real-world approval quality is caught on a small fraction of traffic before it affects everyone.

What’s the difference between a “decline” and a “step-up”?

A decline is a final “no” for this checkout attempt. A step-up means the risk score fell into a borderline zone where more information would change the answer — for example, asking the shopper to verify their identity with a one-time SMS code, or to link a bank account for real-time income verification. Step-up exists because a hard decline on a borderline case loses a potentially good customer and a sale for the merchant, while a step-up converts some of those borderline cases into safe approvals.

How would you test this system before a real Black Friday?

Through regular load testing that replays realistic, anonymized traffic patterns (including the exact mix of approve/decline/step-up outcomes and downstream call latencies) at increasing multiples of expected peak traffic, combined with periodic “game day” chaos exercises where engineers deliberately kill a region, degrade the bureau’s response time, or fill up a Kafka partition to verify the graceful-degradation ladder behaves as designed rather than assumed.

Could this architecture be simplified for a smaller startup that doesn’t yet have millions of requests?

Yes — the core principles (idempotency, timeouts on every external call, separating hot and cold paths) matter at any scale and cost little to build in early. What can reasonably be deferred until traffic actually demands it are multi-region active-active deployment, database sharding, and a dedicated feature store — a single well-indexed database and a simple in-memory cache can serve a smaller startup’s real-time decisioning needs perfectly well, with a clear, well-understood upgrade path to the fuller architecture described here as volume grows.

Why use both a rules engine and a machine learning model instead of just the ML model?

Hard rules encode absolute, auditable business and legal constraints (“never lend to a shopper under 18,” “never exceed a merchant’s configured maximum loan size”) that must never be violated no matter what a statistical model outputs. ML models are excellent at nuanced risk ranking across a huge feature space but are probabilistic and can behave unpredictably on edge cases; layering deterministic rules first, then the model, gives both safety and sophistication.

Designing a real-time BNPL credit approval system is fundamentally about reconciling two competing needs: the sub-second responsiveness shoppers expect at checkout, and the strong consistency and auditability that financial regulation demands. The solution is architectural separation — a fast, timeout-bounded, cached hot path that makes the actual approve/decline decision and one atomic ledger write, with everything else (notifications, deep fraud analysis, analytics, compliance reporting) pushed onto an asynchronous event pipeline. At scale — a million requests a minute and beyond — the winning patterns are the ones that isolate failure (circuit breakers, bulkheads), avoid unnecessary synchronous work (pre-computed features, parallel calls with fallbacks), and scale horizontally and statelessly at every layer, from the API Gateway down to the sharded ledger database.