Designing a Real-Time Settlement Risk Management System
How payments platforms decide, transaction by transaction, exactly how much settlement credit a merchant can safely be given — without ever letting risk exposure cross the line.
Introduction & History
Every time a customer pays a merchant through a payments platform like Stripe, Adyen, Razorpay, or PayPal, something interesting happens behind the scenes. The customer’s money doesn’t land in the merchant’s bank account instantly. Card networks and banking rails typically take one to five business days to actually move funds between banks. Yet merchants expect to see their money quickly, sometimes even instantly, because slow payouts hurt their cash flow and their trust in the platform.
To bridge this gap, payments platforms often advance money to merchants before the underlying funds have actually settled. This is effectively a short-term loan: the platform is extending settlement credit to the merchant, betting that the money will indeed arrive from the card networks or banks as expected. If it doesn’t — because of a chargeback, a fraud dispute, a merchant going out of business, or a processing error — the platform is left holding the loss.
This is exactly the problem a Real-Time Settlement Risk Management System solves. It is the component of a payments platform responsible for constantly tracking how much money has been advanced to each merchant ahead of settlement, comparing that against a risk tolerance limit calculated for that merchant, and making an instant decision — approve, hold, or decline — on every new transaction so that the platform’s exposure never crosses a safe threshold.
The origins of this problem are as old as merchant acquiring itself. Traditional acquiring banks have always maintained underwriting departments that assess a merchant’s risk profile before granting them a merchant account, and they have always maintained reserves against future chargebacks. What changed with the rise of platforms like PayPal, Stripe, Square, and Adyen in the 2010s is the shift from periodic, batch-style risk review (weekly or monthly manual assessments) to continuous, automated, real-time risk decisioning that has to keep pace with tens of thousands of transactions per second across millions of merchants globally.
This shift was driven by three forces. First, the sheer scale of transaction volume made manual review impossible. Second, competitive pressure pushed platforms toward instant or same-day payouts, which increases the amount of unsettled credit outstanding at any moment and therefore increases risk. Third, the diversity of merchants — from a single freelancer selling t-shirts to a global airline — meant that a one-size-fits-all risk policy was both too loose for risky merchants and too strict for safe ones, demanding a dynamic, per-merchant, real-time system instead of static rules.
In this tutorial we will design such a system from the ground up: the architecture, the internal algorithms, the databases, the concurrency model, the failure modes, and the trade-offs, at a level of depth suitable for system design interviews and for engineers who will actually build systems like this in production.
Think of the platform as a shopkeeper who lets a trusted customer walk out with the goods now and pay next week from a paycheck that hasn’t landed yet. The shopkeeper keeps a running tab per customer and a personal limit for each one — enough to keep the relationship easy but not so much that a single unpaid tab could wipe out the shop. Real-time settlement risk management is that shopkeeper’s ledger, running at machine speed across millions of customers at once.
Conceptual Foundations: What Are We Actually Building?
Before drawing boxes and arrows, we need a crisp definition of the problem, because “settlement risk” means slightly different things in different contexts. In our system, we define the following core concepts.
Settlement credit exposure
At any point in time, a merchant’s exposure is the total amount of money the platform has effectively advanced to that merchant which has not yet been recovered from the actual settled funds. Exposure grows every time the platform authorizes a transaction and either pays the merchant early or guarantees the merchant a payout, and it shrinks when the underlying transaction actually settles cleanly, or when a merchant’s rolling reserve absorbs a loss.
Risk tolerance / exposure limit
Each merchant is assigned a risk tolerance, which is the maximum exposure the platform is willing to carry for that merchant at any given moment. This limit is not arbitrary. It is computed from underwriting data (business type, time in business, processing history), behavioural signals (chargeback rate, refund rate, dispute rate), external signals (credit bureau data, industry risk category, geographic risk), and the merchant’s historical settlement performance. High-risk categories, such as travel, ticketing, or subscription businesses with long delivery windows, typically get tighter limits relative to their volume than low-risk categories like grocery or utility payments.
The core guarantee
The single sentence that defines this entire system is: at no point in time should any merchant’s real-time exposure exceed their assigned risk tolerance. Every architectural decision in this tutorial — from how we shard data to which consistency model we pick — flows from the fact that this guarantee must hold even under massive concurrent load, node failures, and network partitions.
This sounds like a simple “check a number against a limit” problem, but it becomes genuinely difficult at scale because of three compounding factors: transactions for the same merchant can arrive concurrently from many sources at once, the check-and-update must be atomic to avoid a race condition where two transactions both pass the check before either update lands, and the system must make this decision in single-digit milliseconds without becoming a bottleneck for the entire payments platform.
Q: Why can’t we just calculate exposure with a nightly batch job and refresh limits once a day?
A: Because exposure changes with every transaction, and a merchant’s risk can spike within minutes — for example, during a flash sale, a fraud ring targeting the merchant, or a sudden surge of chargebacks. A batch system would let a merchant’s exposure balloon far past a safe limit before the next batch run catches it, exposing the platform to potentially unbounded loss. Real-time systems close that window from hours to milliseconds.
Architecture & Components
At a high level, the system sits in the critical path of every transaction that could add to a merchant’s exposure, evaluates it against that merchant’s live risk state, and returns a decision before the transaction is allowed to proceed toward settlement. Let’s look at the major components.
Component Breakdown
API Gateway
The entry point for all transaction and payout events. It handles TLS termination, request authentication, rate limiting per client, and routing. In a system processing millions of requests per minute, the gateway is horizontally scaled behind a load balancer and is stateless, so any instance can serve any request.
Real-Time Risk Check API
A thin, latency-sensitive service that receives a request of the form “merchant X wants to add Y amount of exposure” and orchestrates the rest of the risk decision flow. This layer is intentionally kept simple; the heavy logic lives in the Exposure Calculation Engine and Decision Engine so that this API can remain fast and easy to scale.
Exposure Calculation Engine
The heart of the system. It reads the merchant’s current live exposure from the state store, applies the proposed change (a new transaction typically increases exposure; a settlement event typically decreases it), and produces a candidate new exposure value. Crucially, this read-modify-write must be atomic per merchant to avoid race conditions, which we discuss in depth in the concurrency section.
Risk Limit Service
Owns the business logic for computing and serving each merchant’s risk tolerance. Limits are not static; they are recalculated periodically (for example, nightly, or triggered by significant events like a spike in chargebacks) by an underwriting and risk-scoring pipeline that may itself use machine learning models. The Risk Limit Service caches these limits aggressively since they change far less often than exposure does.
Decision Engine
Compares the candidate new exposure against the merchant’s current limit and returns one of three decisions: Approve (proceed as normal), Hold (delay payout, route to manual or automated secondary review, but do not block the underlying transaction from being processed by the card network), or Decline (reject the additional exposure outright, typically reserved for severe breaches or high fraud-risk transactions).
Ledger Service
The system of record for every financial movement. Every approved transaction and every settlement event is written here as an immutable, append-only entry. The ledger is the ultimate source of truth that the live exposure state store is derived from or reconciled against.
Event Streaming Bus
A distributed log (typically Kafka or a similar system) that decouples the write path (approving a transaction) from downstream consumers like settlement processing, reconciliation, and monitoring. This lets the hot path stay fast while slower, heavier processing happens asynchronously.
Settlement, Reconciliation & Alerting Services
The Settlement Service talks to actual banking rails to move money and clears exposure once funds are confirmed settled. The Reconciliation Service continuously compares the live exposure state against the ledger’s true state to detect and correct drift. The Alerting Service notifies risk analysts and automated systems the moment a merchant approaches or breaches their limit.
Q: Why separate the Exposure Calculation Engine from the Decision Engine instead of merging them into one service?
A: Separation of concerns and independent scaling. The Exposure Calculation Engine is stateful and needs to be co-located with, or tightly coupled to, the state store for low-latency atomic updates. The Decision Engine is largely stateless policy logic (thresholds, rules, ML model scoring) that can scale independently and evolve its logic — for example, adding new risk rules — without touching the sensitive state-mutation code path. This also makes the state-mutation code path smaller, simpler, and easier to formally reason about for correctness, which matters enormously when real money is on the line.
Internal Working: How a Risk Decision Actually Gets Made
Let’s zoom into the exact sequence of steps for a single transaction to understand how the system enforces the core guarantee in practice.
The most important detail in this sequence is that the state store update is a single atomic conditional operation, not a separate read followed by a separate write. In pseudo-terms, the operation says: “increase merchant X’s exposure counter by Y, but only if the resulting value would not exceed the current limit; otherwise leave it unchanged and tell me it failed.” This is the linchpin of correctness, and we explore why in the concurrency section.
Idempotency
Payments systems are notorious for retries — a network blip, a timeout, or a client-side retry can cause the same transaction event to arrive at the Risk Check API more than once. If each arrival independently increased the exposure counter, a single real transaction could be double- or triple-counted, silently corrupting the merchant’s exposure state. To prevent this, every incoming request carries a unique idempotency key (typically the transaction ID), and the Exposure Engine checks whether that key has already been processed before applying any change. Processed keys are stored with a time-to-live long enough to cover realistic retry windows.
The Hold State
Real systems rarely use a strict binary approve/decline. A “Hold” state exists because declining a transaction outright can be commercially damaging and is often unnecessary — many breaches are temporary or borderline. A held transaction is typically still authorized with the card network (so the customer’s purchase is not disrupted) but its payout to the merchant is delayed, routed into a secondary review queue, or funded from the merchant’s rolling reserve instead of being counted against fresh settlement credit.
Platforms such as Stripe and Square commonly use a “rolling reserve” mechanism, where a percentage of each transaction (say 10 percent) is held back for a fixed period (say 90 days) before being released to the merchant. This reserve acts as a self-funding buffer against future chargebacks and directly reduces the platform’s real-time settlement exposure, because it means less money has actually left the platform’s control at any given time.
Q: How would you design the system to avoid double-counting a transaction that is retried by the client after a timeout?
A: Use an idempotency key derived from the transaction ID, store a record of processed keys (with the decision that was made) in a fast-access store such as Redis with a TTL, and short-circuit any repeat request with that key by returning the cached decision instead of re-applying the exposure delta. This must be checked atomically alongside the exposure update, ideally within the same atomic operation or transaction, to avoid a race where two retries both pass the idempotency check simultaneously.
Data Flow & Lifecycle of an Exposure Record
It helps to trace a merchant’s exposure through its full lifecycle, from the moment risk is created to the moment it is resolved.
- Transaction Authorization: A customer pays the merchant. The payments platform authorizes the transaction with the card network or bank, and simultaneously the risk system evaluates whether to advance settlement credit for it.
- Exposure Increase: If approved, the merchant’s live exposure counter increases by the transaction amount (or a risk-weighted portion of it, since not all approved transactions carry the same eventual loss probability).
- Interim Monitoring: While the transaction is in flight toward settlement, it contributes to the merchant’s real-time exposure and is visible to the Alerting Service and dashboards.
- Settlement Event: Days later, the underlying funds actually settle from the card network or bank into the platform’s account. The Settlement Service publishes a settlement-confirmed event.
- Exposure Decrease: The Exposure Engine consumes the settlement event and reduces the merchant’s live exposure counter by the settled amount, freeing up “room” for new transactions.
- Exception Paths: If a chargeback or dispute occurs instead of clean settlement, the exposure is not simply cleared; it may convert into a realized loss, get deducted from the merchant’s reserve, or trigger a debit against the merchant’s linked bank account.
- Reconciliation: Periodically, the Reconciliation Service replays the ledger’s true state and compares it to the live exposure counters, correcting any drift caused by bugs, partial failures, or replayed events.
Notice the use of a warning band (for example, 80 percent of limit) rather than only reacting at the hard limit. This gives risk analysts and automated systems lead time to act — perhaps by tightening payout schedules, requesting additional underwriting documentation, or proactively reaching out to the merchant — before a hard breach forces a Hold or Decline decision that would visibly disrupt the merchant’s business.
Q: How do you handle exposure when a settlement event and a new transaction for the same merchant arrive at almost the same instant?
A: Both are just deltas applied to the same atomic counter — a settlement event applies a negative delta, and a new transaction applies a positive delta. As long as the underlying state store guarantees atomic, serializable updates per merchant key (for example, through compare-and-swap or a distributed lock scoped to that merchant), the order in which the two deltas are applied does not affect correctness of the final value, only possibly the intermediate decision if the transaction is evaluated before the settlement delta lands. Because we only need eventual correctness of the final counter and immediate correctness of each individual decision relative to the state at that instant, this is safe as long as each operation is atomic.
Concurrency, Algorithms & Data Structures
This is where the system’s real engineering difficulty lives. Payments platforms process a huge volume of concurrent requests, and popular merchants can receive dozens or hundreds of simultaneous transaction events. If our exposure check-and-update is not handled correctly, we get a classic race condition.
The Race Condition Problem
Imagine a merchant has a limit of 100,000 and a current exposure of 95,000. Two transactions of 4,000 each arrive at nearly the same instant on two different service instances. If each instance independently reads the current exposure (95,000), independently checks that 95,000 + 4,000 = 99,000 is under the limit, and independently writes the new value, both will approve. The final written value depends on which write lands last, but either way the true combined exposure should be 103,000 — a limit breach that neither individual check caught, because the read-check-write was not atomic.
Solving It: Atomic Conditional Updates
The fix is to make the entire read-check-write sequence a single atomic operation at the data layer, rather than three separate application-level steps. This is typically implemented with one of the following techniques.
| Technique | How it works | Best fit |
|---|---|---|
| Compare-and-Swap (CAS) | Read a value along with a version number; write succeeds only if the version has not changed since the read; retry on failure. | Low to moderate contention per merchant key. |
| Atomic Increment with Guard | A single server-side operation increments a counter and simultaneously checks it against a stored limit, rejecting the increment if it would breach, all within one round trip (for example, a Lua script in Redis or a stored procedure). | High-throughput, low-latency hot paths. |
| Distributed Lock | Acquire an exclusive lock scoped to the merchant ID before performing read-check-write, then release it. | Complex multi-step updates that cannot be expressed as one atomic primitive. |
| Single-Writer Partition (Actor Model) | All updates for a given merchant are routed to exactly one in-memory actor or partition owner, which processes them strictly sequentially. | Extremely high contention on “hot” merchants. |
In practice, most production systems favour the atomic increment with guard approach for the hot path because it avoids the retry storms of optimistic CAS under high contention and avoids the latency and failure-mode complexity of explicit distributed locks. A single atomic script executed at the data store (for example, in Redis or a strongly consistent key-value store that supports conditional writes) can perform the entire “increase counter if result stays under limit” logic in one indivisible step.
Sharding by Merchant ID
Because every operation that matters is scoped to a single merchant, the natural partitioning strategy is to shard exposure state by merchant ID using consistent hashing. This ensures that all operations for merchant X are always routed to the same partition, which both localizes contention (only concurrent transactions for the same merchant contend with each other) and enables horizontal scalability (adding shards linearly increases capacity for handling more distinct merchants).
Handling Hot Merchants
A single hugely popular merchant (think a flash sale event) can generate transaction volume that overwhelms a single shard even though the overall cluster has spare capacity. This is a classic hot key problem. Common mitigations include: further sub-partitioning a single hot merchant’s counter into several sub-counters that are periodically aggregated (trading a small amount of staleness for parallelism), using an in-memory single-threaded actor dedicated to that merchant during the spike, or pre-computing a “fast approve” buffer where a portion of the merchant’s headroom is pre-allocated to a local node to avoid a round trip to the central store for every single transaction, reconciling the buffer periodically.
Sliding Window and Token Bucket Concepts
While the core exposure counter is a running balance rather than a rate limiter, many real systems combine it with rate-limiting-style algorithms for related controls, such as capping the velocity of transactions (for example, no more than N transactions or $M in value within any rolling 60-second window) to catch fraud bursts even when cumulative exposure is still technically under the limit. A sliding window log or a sliding window counter data structure is commonly used here, since it gives more accurate real-time behaviour than a simple fixed-window counter, which can allow up to double the intended rate at window boundaries.
Consensus for Leader Election
Where the architecture uses a single-writer-per-shard model (an actor or partition owner), the cluster needs a reliable way to elect and fail over that owner when a node crashes. This is a textbook use case for a consensus protocol such as Raft, typically provided by a coordination service like etcd or ZooKeeper, ensuring that exactly one node believes it owns a given merchant partition at any time, which prevents the exact race condition we are trying to avoid from re-appearing at the infrastructure level.
Q: Why is sharding by merchant ID better than sharding by, say, transaction ID or a round-robin strategy?
A: Because the invariant we must protect (“exposure for merchant X must never exceed X’s limit”) is entirely scoped to a single merchant. Sharding by anything other than merchant ID would scatter a single merchant’s concurrent transactions across multiple shards, which reintroduces the exact cross-node race condition we are trying to eliminate and forces us back to expensive distributed locks or transactions. Sharding by merchant ID means the atomicity we need is local to a single partition, which is dramatically cheaper and simpler to guarantee correctly.
CAP Theorem, Consistency & Partitioning
The CAP theorem states that a distributed data store can only guarantee two of the following three properties simultaneously during a network partition: Consistency, Availability, and Partition tolerance. Since network partitions are a fact of life in any real distributed system, the meaningful choice in practice is between prioritizing consistency or availability when a partition actually occurs.
For the exposure state store specifically, we must choose consistency over availability. If a network partition splits the cluster and we allowed both sides to keep accepting writes for the same merchant independently (favouring availability), we could easily approve transactions on both sides of the partition that, combined, breach the merchant’s risk limit — precisely the outcome this whole system exists to prevent. A brief service disruption (rejecting or holding new transactions for affected merchants until the partition heals) is a far safer failure mode than silently allowing unlimited exposure.
When the Exposure Engine cannot confidently verify a merchant’s current exposure against their limit — due to a partition, a timeout, or a replica lagging beyond an acceptable threshold — the correct default behaviour is to fail closed (Hold or Decline), not fail open (Approve). Losing a small amount of merchant goodwill from a delayed transaction is a far smaller cost than an unbounded financial loss from an uncontrolled exposure breach.
Where Eventual Consistency Is Acceptable
Not every part of the system needs the same consistency guarantee. The live exposure counter that gates approve/hold/decline decisions needs strong, linearizable consistency. However, downstream consumers like the analytics dashboard, the merchant-facing reporting UI, or the long-term data warehouse can safely use eventually consistent, asynchronously replicated data, because a few seconds of staleness there does not risk a financial breach — it only risks a slightly stale dashboard number, which is an acceptable trade-off for much higher read throughput and lower operational cost.
Replication Strategy
Within a shard, the exposure state is typically replicated synchronously to at least one standby replica before an update is acknowledged as successful, using a consensus-backed replication protocol. This protects against losing the most recent state if the primary node for that shard crashes immediately after acknowledging a write, at the cost of added write latency, which is an acceptable trade-off given the small size of each write (a single counter update) and the criticality of not losing state.
Partition Tolerance in Practice
Because merchant partitions are independent of each other, a partition event affecting one shard’s replicas does not need to bring down the entire system — only transactions for merchants on the affected shard need to fail closed while other shards continue operating normally. This is one of the strongest arguments for the merchant-ID sharding strategy: it contains the blast radius of a consistency incident to a subset of merchants rather than the whole platform.
Q: If you choose consistency over availability, doesn’t that mean the entire risk system becomes a single point of failure for the whole payments platform?
A: Not if the architecture decouples the risk decision from the ability to process the payment itself. A well-designed system can still authorize the underlying card transaction (protecting the customer experience) even if the settlement risk check is temporarily degraded, simply by defaulting all uncertain transactions into the Hold state rather than blocking payment authorization outright. This means unavailability of the risk system degrades to “more transactions get held for later review” rather than “the whole platform stops processing payments,” which is a much more acceptable failure mode.
Databases, Caching & Load Balancing
Different parts of this system have very different data access patterns, so a single database technology is rarely the right answer for all of them. Let’s break down each data store by its role.
Live Exposure State Store
Requirements: extremely low latency (single-digit milliseconds), strong consistency per key, atomic conditional writes, high write throughput. This is typically an in-memory data store with persistence and replication (such as Redis Cluster with Lua-scripted atomic operations, or a distributed key-value store purpose-built for strong per-key consistency). The data model here is intentionally simple: a merchant ID maps to a current exposure value, a version or timestamp, and a small amount of recent transaction metadata for idempotency checks.
Merchant Risk Profile Database
Requirements: moderate write frequency (limits change on underwriting events, not on every transaction), high read throughput, ability to store rich structured data about a merchant’s risk factors. A traditional relational database or a document database works well here, since strong per-request consistency is less critical — a limit that is a few seconds stale is generally acceptable, especially since limits move much more conservatively than exposure does.
Ledger Database
Requirements: absolute durability, immutability (append-only), strong audit trail, ability to reconstruct historical state at any point in time. This is usually backed by a strongly durable, replicated relational or distributed SQL database, often using an event-sourcing style schema where every row is an immutable fact rather than a mutable record, which makes reconciliation and auditing dramatically simpler.
Caching Strategy
The Risk Limit Service caches merchant limits aggressively in a fast in-memory cache local to each service instance or in a shared distributed cache, with a short TTL and active invalidation triggered by the underwriting pipeline whenever a limit changes. Because limits change far less frequently than they are read, this cache dramatically reduces load on the Merchant Risk Profile Database and shaves latency off the hot decision path. Importantly, the live exposure value itself is never served from a stale cache for decisioning purposes — only the limit, which is a slower-moving and less safety-critical value, is cached this way.
Load Balancing
Stateless components (API Gateway, Risk Check API, Decision Engine) sit behind standard layer-7 load balancers using round-robin or least-connections algorithms. The stateful Exposure Engine, however, cannot use naive load balancing, because every request for a given merchant must reach the specific shard (and specific leader replica) that owns that merchant’s state. This is achieved with a consistent-hashing-aware routing layer that inspects the merchant ID in each request and forwards it deterministically to the correct shard, rather than a generic load balancer that would route requests arbitrarily.
Q: Why not just use a single large relational database with row-level locking for exposure, instead of an in-memory store?
A: A relational database with row-level locking can technically provide the correctness guarantees needed, but it typically cannot match the latency and throughput requirements of a payments hot path processing millions of requests per minute, because disk-backed transactional writes with locking are inherently slower than in-memory atomic operations. In practice, teams often use an in-memory store like Redis for the hot-path exposure counters, backed by asynchronous, durable persistence and replication, while the relational or distributed SQL database is reserved for the ledger, where durability and rich querying matter more than raw latency.
APIs & Microservices
The system is naturally decomposed into microservices because each component has different scaling characteristics, different consistency requirements, and different rates of change. Let’s look at the key API contracts and interaction styles.
Synchronous APIs (Request-Response)
The transaction risk check itself must be synchronous, because the caller (typically the core payments processing flow) needs an immediate decision before proceeding. This is exposed as a low-latency internal API, typically gRPC for internal service-to-service calls given its lower serialization overhead compared to REST/JSON, with a strict latency service-level objective, often in the single-digit-to-low-double-digit milliseconds.
POST /v1/risk/decision
Content-Type: application/json
Idempotency-Key: txn_9f2c-att-01
{
"merchant_id": "mch_2Kb7",
"transaction_id": "txn_9f2c",
"amount": { "value": 4200, "currency": "USD" },
"kind": "authorization",
"risk_weight": 1.0,
"channel": "card-online"
}
// Response
{
"decision": "approve" | "hold" | "decline",
"exposure_after": 42800,
"limit": 50000,
"utilization": 0.856,
"reason_code": "under_limit" | "warning_band" | "would_exceed_limit" | "unknown_state",
"policy_version": "risk-2026.03",
"trace_id": "tr_..."
}
Asynchronous APIs (Event-Driven)
Settlement confirmations, chargeback notifications, and limit updates are naturally asynchronous events rather than request-response calls, since they originate from external systems (banking rails, card networks, the underwriting pipeline) on their own schedule. These flow through the event streaming bus, allowing the Exposure Engine to consume them at its own pace while still processing hot-path requests with priority.
Webhook Notifications
When a merchant crosses into a Warning or Breached state, the platform typically needs to notify internal risk analysts and sometimes the merchant themselves (for transparency and to prompt corrective action, like resolving open disputes). This is implemented as outbound webhooks or internal event notifications consumed by the Alerting Service and any merchant-facing notification system.
Microservice Boundaries
A good rule of thumb for drawing service boundaries in this domain is to separate services by their consistency and latency requirements rather than purely by business capability. The Exposure Engine’s boundary is drawn tightly around the atomic state mutation logic specifically because that logic has uniquely strict correctness requirements; everything that doesn’t need that level of rigour (limit computation, alerting, reporting) is deliberately kept in separate services so it can evolve, scale, and even fail independently without threatening the correctness of the core guarantee.
| Service | Style | Latency target | Consistency need |
|---|---|---|---|
| Risk Check API | Synchronous (gRPC) | Single-digit ms | Strong |
| Exposure Engine | Synchronous, internal | Sub-millisecond to low ms | Strong, linearizable |
| Limit Service | Synchronous, cached | Low ms | Bounded staleness acceptable |
| Settlement Service | Asynchronous, event-driven | Seconds to minutes | Eventual, reconciled |
| Reconciliation Service | Batch / streaming | Minutes | Eventual, self-healing |
| Alerting Service | Asynchronous, event-driven | Seconds | Eventual, at-least-once |
Q: Would you use REST or gRPC for the internal risk check call, and why?
A: gRPC, because it is built on HTTP/2 with binary protocol buffer serialization, which gives lower latency and lower CPU overhead than JSON-over-REST for high-throughput internal service calls, and it provides strongly typed contracts via .proto definitions, which reduces integration errors between services. REST/JSON remains a reasonable choice for external-facing or lower-throughput APIs, such as merchant-facing dashboards, where human readability and broad client compatibility matter more than shaving off microseconds.
Design Patterns & Anti-Patterns
Useful Design Patterns
Immutable event log
Rather than storing only the current exposure value, the system can store every exposure-changing event (transaction approved, settlement confirmed, chargeback applied) as an immutable, ordered log. The current exposure is then simply the sum of all events for that merchant. This gives a complete audit trail for free, makes the Reconciliation Service straightforward to build (replay events and compare), and makes debugging disputes (“why was this transaction declined?”) far easier since the full history is preserved.
Command Query Responsibility Segregation
The write path (updating exposure atomically) has very different performance characteristics and consistency needs than the read path (dashboards, reporting, merchant-facing views). Separating these into distinct models — a tightly consistent write model and a denormalized, eventually consistent read model — lets each be optimized independently without compromise.
Fail fast, not slow
If the Risk Limit Service, or any downstream dependency the Exposure Engine relies on, becomes slow or unavailable, a circuit breaker prevents the failure from cascading and taking down the entire hot path. Once the failure rate crosses a threshold, the circuit “opens” and the system fails closed quickly (Hold decisions) rather than piling up timeouts that could exhaust thread pools or connection limits.
Multi-system settlement coordination
Settlement often spans multiple systems — the platform’s own ledger, the banking rail, and potentially the merchant’s linked bank account for reserve debits. Rather than a single distributed transaction across all of these (which is impractical across organizational and system boundaries), a saga coordinates a sequence of local transactions with compensating actions if a later step fails, ensuring the overall settlement process remains consistent even when it spans independent systems.
Isolate blast radius
Resources (thread pools, connection pools, shard capacity) are partitioned so that an issue with one merchant, one shard, or one downstream dependency cannot exhaust resources needed to serve unrelated merchants or shards — directly mirroring the bulkheads in a ship’s hull that prevent one breach from sinking the whole vessel.
Anti-Patterns to Avoid
| Anti-pattern | Why it fails |
|---|---|
| Read-then-write without atomicity | The single most dangerous anti-pattern in this domain: performing a read of current exposure, checking it in application code, and issuing a separate write, with no atomicity guarantee across the three steps. As covered earlier, this is a textbook race condition under concurrent load and is a real cause of production financial losses in immature payments systems. |
| Treating limit updates as real-time critical | Some teams over-engineer the limit-update path to have the same strict consistency guarantees as the exposure-update path. This is usually unnecessary and adds latency and complexity to a path that doesn’t need it — limits change infrequently and moving conservatively (a slightly stale, slightly lower cached limit) is actually safer than a slightly stale, slightly higher one, so caching with modest TTLs plus active invalidation is normally sufficient. |
| Single global lock | Using one global lock or one giant table row to serialize all exposure updates across all merchants (instead of sharding by merchant) is a classic scalability anti-pattern. It trivially guarantees correctness but creates a severe bottleneck and a single point of contention that defeats the purpose of building a horizontally scalable system in the first place. |
| Silent failure to availability | Configuring the system to default to Approve when the risk check itself fails or times out (rather than defaulting to Hold or Decline) trades a rare availability inconvenience for an unbounded financial risk — this is the single most consequential anti-pattern specific to this domain, and it is worth repeating from the CAP theorem discussion because production incidents caused by “fail open” defaults are common and expensive. |
Q: How would event sourcing help you debug a merchant dispute about why a payout was held six months ago?
A: With event sourcing, you can replay the exact sequence of exposure-changing events for that merchant up to the timestamp in question and reconstruct precisely what the live exposure and limit were at that instant, along with which specific transaction pushed it over the threshold. Without event sourcing, you would only have the current aggregate value and would have to rely on separate, possibly incomplete logs to reconstruct history, which is far less reliable for a financial audit.
Performance & Scalability
Designing this system for a payments platform operating at global scale means assuming millions of requests per minute, with sharp, unpredictable spikes around sales events, holidays, and regional business hours. Let’s walk through how the design holds up.
Horizontal Scalability
Because exposure state is sharded by merchant ID, adding capacity is largely a matter of adding more shards and rebalancing the consistent hash ring. Stateless layers (gateway, decision engine) scale trivially by adding more instances behind the load balancer. The main scaling challenge is the state store tier, since it must maintain strong consistency while growing — this is why the atomic per-key operations and merchant-scoped sharding are so central to the design; they keep each unit of work small and independent.
Latency Budget
In a payments hot path, every additional millisecond in the risk check compounds with every other step (fraud scoring, authorization, network calls to card networks) to determine overall checkout latency, which directly affects conversion rates for merchants. A realistic latency budget might allocate a small single-digit number of milliseconds to the risk check specifically, which rules out any component in the hot path that depends on slow disk I/O, cross-region network calls, or synchronous calls to systems with unpredictable tail latency.
Handling Spiky Load
Flash sales and seasonal peaks (a major holiday shopping event, for example) can push a small number of merchants to many multiples of their normal transaction rate. Beyond the hot-key mitigations discussed earlier, capacity planning for this system typically involves auto-scaling stateless tiers aggressively based on request rate, pre-warming caches for merchants with known upcoming high-volume events, and maintaining headroom in each shard’s capacity rather than running shards near saturation.
Batching and Backpressure
While the core per-transaction decision cannot be batched (each one needs an individual answer), downstream asynchronous work like settlement confirmation processing and reconciliation benefit heavily from batching, since they can process many events together more efficiently than one at a time. The event streaming bus provides natural backpressure: if downstream consumers fall behind, events queue up in the log rather than being lost or forcing the hot path to slow down, decoupling burst absorption from the latency-critical write path.
Read Scaling for Dashboards
Risk analyst dashboards and merchant-facing reporting views are read-heavy and can tolerate slight staleness, so they are served from denormalized, horizontally scalable read replicas or a separate read-optimized store (per the CQRS pattern), which keeps this read traffic from ever competing with the latency-critical write path for the same underlying resources.
Q: If your system needs to handle millions of transactions per minute globally, would you run a single global cluster or regional clusters?
A: Regional clusters, generally, for both latency and regulatory reasons — keeping the risk check physically close to where the transaction originates minimizes network latency, and many jurisdictions have data residency requirements for financial data. The complexity this introduces is around merchants that operate across multiple regions, which requires either designating a home region that owns the authoritative exposure state for that merchant (with other regions forwarding requests there) or a more complex multi-region consensus approach, which is usually only justified for the platform’s very largest global merchants.
High Availability & Reliability
Replication and Failover
Each exposure shard is replicated across multiple nodes, typically spread across separate availability zones, with a consensus-backed leader election mechanism so that if the current leader for a shard fails, a replica is automatically promoted within a short, bounded time window. During this failover window, requests for the affected merchants fail closed (Hold) rather than being silently dropped or, worse, approved without verification.
Write-Ahead Logging and Durability
Every atomic update to the exposure state store is first written to a durable write-ahead log before being acknowledged, so that even if a node crashes immediately after an update, the change is not lost and can be recovered by replaying the log during restart or failover.
Disaster Recovery
Beyond node-level failover, the system needs a strategy for larger-scale failures, such as an entire region becoming unavailable. This typically involves asynchronous cross-region replication of the ledger and periodic snapshots of exposure state, combined with a documented and regularly tested failover runbook, since financial systems cannot afford to discover gaps in their disaster recovery plan during an actual incident.
Graceful Degradation
Rather than a binary “fully up” or “fully down” status, the system is designed with graceful degradation in mind: if the Limit Service’s cache is stale beyond a safety threshold, the system can fall back to a more conservative, pre-approved baseline limit rather than failing every request; if the Exposure Engine cannot reach a specific shard, only merchants on that shard are affected, not the whole platform; if the event bus backs up, the hot path continues to function using the last-known-good state while reconciliation catches up once the backlog clears.
Self-Healing Reconciliation
Because distributed systems inevitably experience partial failures, message duplication, or replay scenarios, the Reconciliation Service acts as a continuous self-healing mechanism, periodically recomputing exposure from the authoritative ledger and correcting any drift in the live state store, with automated alerts if drift exceeds an expected tolerance, which would indicate a deeper bug rather than routine eventual-consistency noise.
Q: What happens to in-flight transactions during a shard leader failover?
A: Requests that were in flight to the failed leader will time out and, per the fail-closed principle, should be retried against the newly elected leader once failover completes, or resolved to a Hold decision if the failover takes longer than the caller’s timeout budget. Because the write-ahead log is replicated before acknowledgment, any update that was actually acknowledged to the caller before the failure is guaranteed to be present on the new leader, so no successfully-approved transaction is lost; only unacknowledged, in-flight requests need to be retried or held.
Security
This system sits at the intersection of financial risk and sensitive merchant and transaction data, so security considerations span authentication, data protection, auditability, and abuse prevention.
Service-to-Service Authentication
All internal calls between services (Gateway to Risk API, Risk API to Exposure Engine, and so on) are authenticated using mutual TLS or short-lived service tokens, ensuring that only legitimate internal services can query or mutate exposure state, and every call is attributable to a specific calling service for audit purposes.
Encryption
Data is encrypted in transit using TLS across all network hops, and sensitive data at rest — particularly ledger records and merchant risk profiles — is encrypted using strong, industry-standard encryption, with key management handled through a dedicated key management service rather than embedded application secrets.
Tamper-Evident Ledger
Because the ledger is the ultimate financial source of truth, many production systems add cryptographic hash chaining between consecutive ledger entries, so that any unauthorized modification to historical records becomes detectable, similar in spirit to how blockchain systems guarantee tamper evidence, without necessarily requiring a full distributed blockchain architecture.
Least Privilege and Segregation of Duties
Access to directly modify a merchant’s risk limit is restricted to the underwriting and risk pipeline, with strict role-based access control, and any manual override by a human risk analyst is logged with the analyst’s identity, timestamp, and justification, both for accountability and to satisfy financial compliance audits.
Fraud and Abuse Considerations
The risk system itself can become a fraud target — for example, a bad actor might attempt to rapidly test many small transactions to probe a merchant’s exposure limit, or attempt to exploit timing windows around limit recalculation. Rate limiting, anomaly detection on transaction patterns, and short, tightly bounded windows between limit changes and their propagation all help reduce this attack surface.
Regulatory Compliance
Payments platforms operating this kind of system are typically subject to financial regulations such as PCI DSS for cardholder data handling, and in many jurisdictions, specific regulatory requirements around reserve funds and merchant fund safeguarding. The system’s audit trail (via the ledger and event sourcing) is designed from the outset to support the kind of detailed reporting that regulators and auditors require.
Q: How would you prevent a compromised internal service from silently inflating a merchant’s risk limit to bypass exposure controls?
A: Through defense in depth: strict mutual TLS authentication limiting which services can even call the limit-update API, role-based access control ensuring only the underwriting pipeline’s credentials are authorized to write limits, an immutable audit log of every limit change including which service or user made it, and automated anomaly detection that flags unusual limit changes (for example, a large limit increase with no corresponding underwriting event) for human review. No single control is perfect, so layering several independent controls is what makes the system resilient to a single point of compromise.
Monitoring, Logging & Metrics
Key Metrics to Track
- Decision latency (p50, p95, p99): How long the risk check takes end to end; tail latency matters enormously in a payments hot path.
- Approve/Hold/Decline rates: Tracked overall and per merchant segment, to spot unusual shifts that might indicate a bug, a policy misconfiguration, or an actual fraud event.
- Exposure utilization distribution: What percentage of merchants are near their limit at any given time, which is a leading indicator of systemic risk building up across the platform.
- Reconciliation drift: The size and frequency of discrepancies found between the live state store and the ledger’s true state; growing drift is an early warning of a correctness bug.
- Shard health and replication lag: Per-shard leader status, replica lag, and failover events.
- Idempotency hit rate: How often duplicate requests are detected, which can reveal upstream retry storms or network issues.
Structured Logging and Tracing
Every risk decision is logged with structured, queryable fields (merchant ID, transaction ID, decision, current exposure, limit, latency), and distributed tracing (for example, using a standard like OpenTelemetry) threads a single trace ID through the Gateway, Risk API, Exposure Engine, and Ledger Service, so a single slow or failed transaction can be traced end to end across service boundaries.
Real-Time Alerting
Alerts are configured on both system health signals (elevated latency, elevated error rate, replication lag) and business signals (a specific merchant crossing into the Breached state, an unusual platform-wide spike in Decline rates, aggregate exposure across the whole platform approaching an internal risk appetite ceiling). Business-signal alerts typically route to risk analysts, while system-health alerts route to on-call engineers, since the appropriate response differs significantly between the two.
Dashboards
A risk-analyst-facing dashboard typically shows, per merchant, the current exposure, the limit, recent trend, and any active holds, while an engineering-facing dashboard focuses on system health metrics like latency percentiles, error rates, and shard status. Keeping these separate avoids overwhelming either audience with irrelevant detail.
Q: What is the single most important early-warning metric for this system, and why?
A: Reconciliation drift is arguably the most important, because it is a direct, quantitative measure of whether the core correctness guarantee is actually holding in production. Latency and error-rate metrics tell you about availability and performance, but only reconciliation drift tells you whether the live exposure numbers the Decision Engine is trusting are actually accurate. A small but persistent nonzero drift, even if every other metric looks healthy, indicates a subtle bug that could eventually allow real limit breaches.
Deployment & Cloud Architecture
Containerization and Orchestration
Stateless services are packaged as containers and deployed on an orchestration platform such as Kubernetes, which handles scheduling, auto-scaling based on load, and self-healing by restarting failed instances. The stateful Exposure Engine and its backing store require careful orchestration configuration, typically using StatefulSets with stable network identities and persistent volumes, since state placement and identity matter far more here than for stateless services.
Multi-Region Deployment
For a global payments platform, the system is deployed across multiple cloud regions, with each merchant’s exposure state homed in a specific region close to where most of their transactions originate, and asynchronous cross-region replication of the ledger for disaster recovery and global reporting purposes, as discussed in the high availability section.
Progressive Delivery
Given the financial sensitivity of this system, deployments use progressive delivery techniques: canary releases that route a small percentage of traffic to a new version while closely monitoring key correctness and performance metrics, followed by a gradual, automated or manually gated rollout to the rest of the fleet, with automatic rollback triggers if error rates or reconciliation drift increase during the canary phase.
Infrastructure as Code
The entire deployment topology — service definitions, scaling policies, network policies, secrets management configuration — is defined declaratively as code and version controlled, which supports reliable, repeatable deployments across environments and provides an audit trail of infrastructure changes, which is often itself a compliance requirement in regulated financial environments.
Cost Optimization
The latency-critical, always-on tiers (Exposure Engine, state store) justify their always-provisioned cost given their criticality, while less latency-sensitive, bursty workloads like reconciliation batch jobs and reporting pipelines are well suited to more cost-efficient, elastically scaled or spot-instance-backed compute, since brief delays or interruptions there do not threaten the core correctness guarantee.
Q: How would you safely roll out a change to the core exposure-update logic given how financially sensitive it is?
A: Very conservatively: extensive testing including property-based tests that simulate concurrent transaction storms to verify the atomicity guarantee holds, a canary rollout limited to a small, carefully chosen subset of low-risk merchants first, real-time monitoring of reconciliation drift specifically during the canary window since that is the most sensitive correctness signal, and an automated rollback trigger tied to any drift or error-rate anomaly. Given the cost of a mistake, teams often also maintain a manual “kill switch” to instantly revert to the previous version or fail closed platform-wide if something looks wrong post-deployment.
Advantages, Disadvantages & Trade-offs
Advantages of this design
- Bounds the platform’s financial exposure to any single merchant precisely and in real time, rather than relying on slow, error-prone batch review.
- Sharding by merchant ID localizes contention and allows the system to scale horizontally as the merchant base grows.
- Fail-closed defaults and strong consistency on the critical path protect the platform even during infrastructure failures.
- Clean separation of the hot, latency-critical path from slower asynchronous processing (settlement, reconciliation, alerting) keeps the core decision fast while still supporting rich downstream functionality.
- Event sourcing and an immutable ledger provide a strong audit trail, which is valuable both for regulatory compliance and for debugging disputes.
Disadvantages and costs
- Strong consistency requirements add engineering complexity and operational overhead compared to a simpler, eventually consistent design.
- Fail-closed behaviour means legitimate merchants can occasionally experience unnecessary holds or declines during infrastructure incidents, which is a real business cost even though it is the safer default.
- Sharding by merchant ID can create hot-key problems for very large merchants, requiring additional engineering effort (sub-partitioning, dedicated capacity) to handle gracefully.
- Maintaining accurate, well-calibrated risk limits is itself a hard machine learning and data problem; the real-time enforcement system is only as good as the limits it is enforcing.
- Multi-region deployment for global scale introduces cross-region consistency and data residency complexity that a single-region design would not need to solve.
Key Trade-offs Summary
| Decision | Chosen approach | Trade-off accepted |
|---|---|---|
| Consistency vs Availability | Consistency (fail closed) | Occasional unnecessary holds during incidents |
| State store technology | In-memory, strongly consistent | Higher operational complexity than a simple SQL table |
| Partitioning key | Merchant ID | Hot-key risk for very large merchants |
| Limit caching | Aggressive caching with TTL | Small window of using a slightly stale limit |
| Hold vs Decline default | Prefer Hold over Decline | More operational review workload, better merchant experience |
Q: If you had to relax one of the constraints in this system to significantly simplify it, which would you pick and what would you accept in exchange?
A: A reasonable answer is to relax the real-time latency requirement slightly for a specific class of lower-risk merchants — for example, small merchants well within their limits with a long clean processing history — by evaluating their exposure on a slightly delayed, near-real-time basis (seconds rather than milliseconds) using a simpler, eventually consistent path, while keeping the strict, low-latency, strongly consistent path only for merchants near their limit or in higher-risk categories. This trades a small amount of risk precision for large merchants outside the fast path in exchange for meaningfully reduced infrastructure cost and complexity for the long tail of low-risk merchants, which in most real portfolios make up the majority of merchant count but a small fraction of aggregate risk.
Best Practices & Common Mistakes
Best Practices
- Always make the read-check-write sequence a single atomic operation at the data layer; never assemble it from separate application-level steps.
- Default to fail-closed behaviour anywhere the system cannot confidently verify current exposure against the limit.
- Shard by the entity whose invariant you are protecting, not by an arbitrary or convenient key like transaction ID.
- Separate the latency-critical write path from slower, asynchronous downstream processing using an event streaming backbone.
- Build reconciliation as a first-class, continuously running component, not an afterthought bolted on after an incident.
- Track reconciliation drift as a primary correctness metric, not just system-level latency and error rates.
- Design idempotency into every state-mutating API from day one, since retries in payments systems are the norm, not the exception.
- Keep the Hold state as a first-class citizen alongside Approve and Decline, since a strict binary decision model is usually too blunt an instrument for real merchant risk scenarios.
Common Mistakes
- Implementing exposure checks as separate read and write calls to the database, introducing race conditions under concurrent load.
- Defaulting to Approve when the risk check times out or fails, prioritizing short-term availability over the platform’s actual financial safety.
- Treating the risk limit and the live exposure value with the same caching strategy, when in fact they have very different staleness tolerances.
- Under-provisioning capacity for a small number of very large merchants, leading to hot-key bottlenecks during peak events.
- Neglecting to build robust reconciliation, leading to silent drift between live state and ledger truth that goes undetected until a major incident forces a manual audit.
- Coupling the risk decision tightly to the payment authorization flow such that any risk-system outage also blocks all payment processing, rather than allowing graceful degradation to a conservative default.
Q: You mentioned reconciliation as critical. How often should it run, and why?
A: Ideally continuously, as a streaming process consuming the same event log that feeds the live exposure state, rather than a periodic batch job. Continuous reconciliation catches drift within seconds to minutes rather than hours, which meaningfully limits how large an undetected discrepancy can grow before it is caught and corrected. A periodic batch reconciliation (for example, nightly) can serve as an additional, independent cross-check using a different code path than the streaming reconciliation, which helps catch bugs that might affect both the live system and the streaming reconciliation identically.
Real-World Industry Examples
Dynamic payout schedules and rolling reserves
Stripe manages settlement risk across millions of merchants with widely varying risk profiles by combining automated underwriting, dynamic payout schedules (new or higher-risk merchants often start on a longer payout delay, such as seven days, before graduating to faster payouts as trust is established), and rolling reserves that hold back a percentage of volume as a buffer against future disputes, all continuously informed by real-time processing signals like chargeback and dispute rates.
Processor and acquirer under one roof
As both a payments processor and, in many markets, the underlying acquiring bank, Adyen carries settlement risk directly and applies real-time risk and compliance decisioning at the point of transaction, using continuously updated merchant risk profiles that factor in transaction patterns, industry category, and processing history to determine how aggressively funds can be advanced ahead of full settlement.
Reserve accounts and holds as core controls
PayPal has long used reserve accounts and payment holds as core risk controls, particularly for new sellers, sellers in higher-risk categories, or sellers experiencing unusual account activity, temporarily limiting available balance and delaying access to funds in a way that is functionally very similar to the exposure-versus-limit model described throughout this tutorial.
Real-time pattern anomaly detection
Square, serving a large base of small and medium merchants including many with limited processing history, places significant emphasis on real-time transaction pattern analysis to detect anomalies quickly, since a sudden deviation from a small merchant’s typical volume is often the earliest and most reliable signal of either fraud or a rapidly changing risk profile that should trigger tighter real-time settlement controls.
Common Threads Across These Platforms
Despite different technical implementations, every major platform managing settlement risk at scale converges on the same core ideas covered in this tutorial: dynamic, per-merchant risk limits rather than static, one-size-fits-all rules; continuous, near-real-time recalculation of exposure rather than periodic batch review; reserves and payout delays as practical mechanisms for actually enforcing a computed limit; and a strong bias toward conservative, fail-closed behaviour when signals are ambiguous, because the asymmetry between the cost of a false decline and the cost of an unrecovered loss strongly favours caution in this specific domain.
FAQ, Summary & Key Takeaways
Is this the same thing as fraud detection?
Related but distinct. Fraud detection focuses on identifying whether a specific transaction is illegitimate. Settlement risk management focuses on whether the platform’s cumulative financial exposure to a merchant is within safe bounds, regardless of whether any individual transaction is fraudulent. A merchant can have entirely legitimate transactions and still breach their exposure limit simply through high volume, and conversely, fraud signals are one important input into setting risk limits and individual transaction decisions, so the two systems are closely integrated in practice even though their core questions differ.
Why not just set very conservative limits for every merchant to avoid the problem entirely?
Overly conservative limits directly hurt legitimate merchants by delaying their access to funds or capping their growth on the platform, which is a real competitive and commercial cost. The entire point of a dynamic, real-time system is to safely extend as much settlement credit as each merchant’s actual risk profile allows, rather than applying a blunt, universally conservative policy that would make the platform less attractive to good merchants.
How does this system interact with machine learning models?
Machine learning is typically used in the Risk Limit Service to compute and continuously refine each merchant’s risk tolerance based on a wide range of features (processing history, industry, geography, behavioural signals), and sometimes in the Decision Engine for finer-grained transaction-level risk scoring. However, the core atomic exposure-tracking mechanism described in the architecture is deliberately kept simple and deterministic, since the correctness of “did we exceed the limit” should not depend on a probabilistic model; the model informs what the limit should be, not whether the limit was breached.
What is the single hardest part of building this system in practice?
Most engineers who have built systems like this point to correctly handling concurrency at scale — guaranteeing atomicity of the exposure check-and-update under extremely high, bursty concurrent load without creating a bottleneck — as the hardest and most consequential engineering challenge, precisely because getting it subtly wrong doesn’t cause an obvious crash; it causes a slow, silent accumulation of financial risk that may not be discovered until a large loss event forces a full investigation.
Key Takeaways
- Settlement risk exists because payments platforms often advance funds to merchants before the underlying transactions actually settle, creating a real, quantifiable exposure that must be actively managed.
- The core invariant — exposure must never exceed a merchant’s risk tolerance — must be enforced through atomic, per-merchant conditional updates, not separate read-check-write steps.
- Sharding by merchant ID is the natural partitioning strategy because it localizes the contention the system must protect against and enables horizontal scaling.
- Strong consistency and fail-closed defaults are the right choice for the hot exposure-update path, even at some cost to availability, because the asymmetry of financial risk favours caution.
- Separating the latency-critical decision path from slower, asynchronous settlement, reconciliation, and reporting workflows keeps the system both fast and rich in functionality.
- Continuous reconciliation against an immutable, event-sourced ledger is what actually proves, in production, that the system’s core guarantee is holding.
- Real-world platforms like Stripe, Adyen, PayPal, and Square all converge on similar patterns — dynamic per-merchant limits, reserves and payout delays as enforcement mechanisms, and conservative defaults under uncertainty — because these are the proven, battle-tested solutions to this exact problem.
Real-time settlement risk management is not a rule engine and not a fraud model — it is a running balance defended by an atomic invariant. Everything else in the design — the sharding, the CAP choice, the reserves, the reconciliation, the fail-closed defaults — exists to make sure that a single number for each merchant never quietly crosses a single line, no matter how fast the traffic or how bad the day.