Open any modern super-app — Paytm, Venmo, Cash App, Alipay, PayPal, Google Pay — and somewhere in it sits a small number that matters more than almost anything else in the entire product: your wallet balance. Behind that single number is one of the hardest problems in distributed systems, because unlike a “like count” on a social post or a product view counter, a wallet balance is not allowed to ever be approximately right. It must be exactly right, every single time, even when millions of people are sending, receiving, topping up, and withdrawing money in the same second.

Digital wallets trace their lineage back to simple prepaid card systems and bank ledgers of the 1990s, where a single centralized mainframe processed transactions one at a time against a single database — slow, but safe, because there was only ever one place a balance could live and one thread of execution touching it. As digital payments exploded in volume through the 2010s (mobile money in Kenya’s M-Pesa, China’s Alipay and WeChat Pay, India’s UPI-linked wallets, and Western apps like Venmo and Cash App), that single-mainframe model could no longer keep up with request volume, and the industry had to solve a genuinely hard question: how do you horizontally scale a system across hundreds of machines while still guaranteeing the same all-or-nothing correctness a single mainframe gave you for free?

The answer that emerged — and the one this tutorial designs in depth — combines ideas from classical database theory (ACID transactions, double-entry bookkeeping, serializable isolation) with modern distributed-systems techniques (consensus protocols, distributed transactions, sharding with cross-shard coordination). The result is a system that can scale horizontally to handle enormous request volume while never once allowing a user’s balance to become incorrect, inconsistent, or exploitable through a race condition.

2.1 Functional requirements

• Users can hold a balance in one or more currencies/wallets, deposit money (via card, bank transfer, cash-in agent), withdraw money, and transfer money to other users.
• Every transaction must be atomic: a transfer either fully succeeds (debit sender, credit recipient) or fully fails — there is never a partial state visible to any reader.
• The system must support querying the current balance instantly and correctly, with no possibility of a user seeing a balance that reflects only half of an in-flight transaction.
• Duplicate submissions of the same transaction request (e.g., a client retrying after a network timeout) must never result in the money moving twice.
• A full, immutable audit trail of every balance-affecting event must exist, sufficient to reconstruct any account’s balance at any point in history and to satisfy financial regulatory audits.

2.2 Non-functional requirements

• Scale: the platform must sustain on the order of 1,000,000+ requests per minute (~16,700 requests/second sustained) across balance checks, deposits, withdrawals, and transfers combined, with bursts several times higher during salary-disbursement days, sale events, or festival cash-transfer spikes.
• Consistency: strict, strong consistency (not eventual consistency) for anything that touches a balance. This is the single most important non-functional requirement in the entire system and drives nearly every architectural decision that follows.
• Latency: a transfer between two wallets should complete, end-to-end, in well under 500ms at p99 for same-shard transfers, and under 1–2 seconds for cross-shard transfers.
• Durability: once a transaction is acknowledged as successful to the client, it must never be lost, even in the face of a data center failure.
• Availability: 99.99%+ uptime for balance reads and transaction submission, since a wallet that can’t be checked or used is effectively broken money.
• Auditability & compliance: the system must support regulatory requirements common to financial systems (KYC/AML checks, transaction reporting, immutable audit logs) from day one, not bolted on afterward.

Before diagramming anything, we need a shared vocabulary. Each concept below is explained from scratch — what it is, why it exists in this system, and a concrete example of how it plays out on the money path.

3.1 Double-Entry Bookkeeping

What: Double-entry bookkeeping is a centuries-old accounting technique where every transaction is recorded as two balanced entries: a debit in one account and an equal credit in another. The books are only ever considered correct if, at any point in time, the sum of all debits equals the sum of all credits across the entire system.

Why: This gives us a built-in, mathematically verifiable correctness check. If a bug ever causes money to be created or destroyed, the sum of all ledger entries stops equaling zero, and a reconciliation job can detect the discrepancy automatically, often before a human even notices.

Analogy: It’s like a seesaw that must always stay balanced — for every gram added to one side, an equal gram must be removed from the other, or from a third party’s side, but the total weight on the seesaw as a whole never mysteriously changes.

Practical example: A $50 transfer from Alice to Bob isn’t stored as “Alice.balance -= 50” and “Bob.balance += 50” as two independent updates; it’s stored as two immutable ledger rows in the same atomic transaction: (DEBIT, Alice, 50, txnId) and (CREDIT, Bob, 50, txnId). The account’s “balance” is then a materialized, cached sum of its ledger entries, not a mutable field that’s directly overwritten.

3.2 ACID Transactions & Isolation Levels

What: ACID stands for Atomicity, Consistency, Isolation, Durability — the four guarantees a database transaction makes. Isolation specifically determines how much concurrent transactions can “see” of each other’s in-progress changes, ranging from Read Uncommitted (weakest) to Serializable (strongest, behaves as if transactions ran one at a time).

Why it matters here: A wallet transfer absolutely needs Atomicity (both legs happen or neither does) and a strong Isolation level (Serializable, or Snapshot Isolation with explicit locking) to prevent the classic “lost update” bug: two concurrent withdrawals both read a balance of $100, both think there’s enough for their $80 withdrawal, and both commit — leaving the account at -$60 instead of correctly rejecting the second withdrawal.

Practical example: Our Ledger Service wraps every balance-affecting operation in a database transaction using SELECT ... FOR UPDATE (pessimistic row locking) or optimistic concurrency control with a version/row-timestamp check, ensuring two concurrent operations on the same account can never both proceed against a stale balance.

3.3 Optimistic vs Pessimistic Concurrency Control

What: Pessimistic concurrency control locks a row before reading it, blocking any other transaction from touching it until the lock is released. Optimistic concurrency control instead reads a row along with a version number, proceeds without locking, and only at commit time checks whether the version is still unchanged — retrying if someone else modified it in the meantime.

Trade-off: Pessimistic locking is simpler to reason about and avoids wasted work on conflict, but can create contention and reduced throughput on “hot” accounts (e.g., a popular merchant account receiving thousands of payments per second). Optimistic concurrency control scales better under low-contention workloads but wastes work and adds retry latency when contention is high.

Practical example: For typical peer-to-peer wallet transfers (low contention per account), we default to optimistic concurrency control with a version column. For known high-contention “hot” accounts (large merchants, payroll disbursement accounts), we switch that specific account to pessimistic locking or, better, a sharded sub-ledger with periodic aggregation, discussed further in the Scalability section.

3.4 Distributed Transactions: Two-Phase Commit & Sagas

What: When a transfer involves two accounts that live on different database shards, a single local ACID transaction isn’t enough — we need a distributed transaction protocol. Two-Phase Commit (2PC) has a coordinator ask all participants to “prepare” (lock resources and confirm they can commit), and only after every participant agrees does the coordinator tell everyone to actually “commit.” The Saga pattern instead breaks a distributed transaction into a sequence of local transactions, each with a corresponding compensating action that can undo it if a later step fails.

Why both matter here: 2PC gives strong consistency but at the cost of holding locks across a network round-trip, reducing throughput and creating a risk of blocking if the coordinator crashes mid-protocol. Sagas avoid holding locks across the network (better throughput) but only provide eventual consistency during the saga’s execution window, requiring careful compensating-transaction design so a partially-completed transfer is never visible as a permanent, incorrect balance.

Practical example: For cross-shard transfers, we use a variant sometimes called a “sagas with a durable coordinator log” — a Transaction Coordinator records each step’s intent durably (backed by a Raft-based consensus store) before executing it, so that even if the coordinator process crashes mid-transfer, a recovery process can read the durable log and either complete or compensate the transfer deterministically, rather than leaving it in an ambiguous state.

3.5 Idempotency Keys

What: An idempotency key is a unique identifier a client attaches to a request (often a UUID generated once per logical user action) that the server uses to detect and safely ignore duplicate submissions of the same request.

Why: Networks are unreliable. If a client submits a $50 transfer and the network drops the response before the client sees “success,” the client’s natural instinct is to retry. Without idempotency keys, that retry would move another $50, double-charging the sender.

Practical example: Every transfer request carries a client-generated idempotencyKey. The Wallet Service checks this key against a durable idempotency store (with the same strong consistency guarantee as the ledger itself) before processing; if the key has been seen before, it returns the original, already-committed result instead of processing the transfer again.

3.6 Consensus Protocols: Raft and Paxos

What: Consensus protocols are algorithms that let a group of independent machines agree on a single, shared sequence of values or decisions, even when some machines are slow, crash, or are temporarily unreachable — and, critically, without ever letting two conflicting decisions both “win.” Raft and Paxos are the two most widely-used consensus protocols in production systems; Raft was explicitly designed to be more understandable than Paxos while providing the same core guarantees.

Why it matters here: every shard in our distributed SQL cluster is internally a Raft group — a set of replicas (typically 3 or 5) that use Raft to agree on the order of every write to that shard, and to automatically elect a new leader if the current one fails. This is what gives each shard its strong consistency guarantee without requiring a single, unreplicated machine to be the sole keeper of the truth.

Analogy: Picture three referees watching the same race, each with their own stopwatch. Raft is the rulebook that lets them agree on one official finishing order even if one referee’s stopwatch briefly glitches or one referee steps away for a moment — as long as at least two of the three are present and functioning, the group can still produce one single, trusted, agreed-upon result, and the temporarily-absent referee can catch up and adopt the agreed result once they return.

Practical example: when a shard’s current Raft leader crashes mid-transaction, the remaining replicas detect the failure (via missed heartbeats), hold a leader election, and a new leader takes over — typically within one to a few seconds — after which writes to that shard resume automatically, with no data loss for any write that had already been acknowledged as committed (since Raft only considers a write committed once a majority of replicas have durably stored it).

3.7 CAP Theorem, Applied Deliberately

What: as introduced briefly earlier, the CAP theorem says a distributed system can provide at most two of Consistency, Availability, and Partition tolerance at the same time during an actual network partition. Because partitions are a fact of life in any real distributed system (network links do fail), the meaningful choice in practice is really between prioritizing Consistency (a “CP” system) or Availability (an “AP” system) when a partition occurs.

Why we choose CP for the wallet core, deliberately and explicitly: an AP system, when partitioned, keeps serving both sides of the split with whatever local data each side has — which for a wallet balance means two different, disconnected parts of the system could each independently approve a withdrawal against what they believe is the current balance, with no way to detect the conflict until the partition heals. That is precisely a double-spend. A CP system instead refuses to serve (or explicitly errors on) writes on any side of a partition that cannot reach a quorum of replicas, guaranteeing that whichever side does keep operating has a truthful, single, agreed-upon view of the balance.

Practical example: if a network partition isolates one availability zone containing a minority of a shard’s Raft replicas, that isolated minority simply cannot commit new writes (it can’t reach quorum), and any request routed there fails fast with a clear, retryable error — while the majority partition continues operating normally with full consistency guarantees intact.

Below is the full architecture, drawn as a component diagram. Every box names the actual technology or role it plays, including the API Gateway and Load Balancer explicitly, as they sit at the very front door of every request in this system.

4.1 Component-by-component breakdown

5.1 Flow A — Same-shard transfer (the common case)

5.2 Flow B — Cross-shard transfer

1. The Wallet Service determines sender and recipient live on different shards and delegates to the Transaction Coordinator instead of attempting a local transaction.
2. The Coordinator durably logs its intent (“begin transfer txnId, debit shard 1 account X by $50, credit shard 2 account Y by $50”) to its Raft-replicated durable log before doing anything else — this is the critical step that makes crash recovery deterministic.
3. Prepare phase: the Coordinator asks shard 1 to tentatively reserve/lock $50 from account X (without yet committing the debit) and asks shard 2 to confirm account Y can accept the credit. Both shards respond “prepared” only if they can guarantee the operation will succeed if asked to commit.
4. Commit phase: once both shards acknowledge “prepared,” the Coordinator durably logs the decision to commit, then instructs both shards to finalize — shard 1 commits the debit, shard 2 commits the credit.
5. If either shard fails to prepare (e.g., insufficient balance), the Coordinator durably logs a decision to abort and instructs any already-prepared shard to roll back its tentative reservation — no money moves.
6. If the Coordinator itself crashes between the prepare and commit phases, a recovery process reads the durable log on restart, sees the last recorded decision, and re-drives the remaining shards to the same conclusion, guaranteeing the transfer is never left in an ambiguous, half-completed state.

5.3 Sequence diagram — cross-shard transfer with Two-Phase Commit

Every transaction moves through a well-defined lifecycle, recorded immutably at each stage:

An important design decision: a transaction is never “soft” or “reversible” by directly mutating a past ledger entry. If a transaction needs to be reversed (a refund, a disputed chargeback), this is modeled as a brand new, separate transaction that moves money back — never as an edit or deletion of the original entry. This preserves a complete, tamper-evident audit trail, which is both a regulatory requirement in most jurisdictions and a powerful debugging tool when investigating discrepancies.

The central trade-off worth dwelling on: this architecture deliberately accepts lower raw throughput and higher latency per transaction than a system that allowed eventual consistency, in exchange for a correctness guarantee that a wallet system cannot function without. In an interview, this is worth stating explicitly and confidently — it is not a weakness of the design, it’s the correct prioritization for the domain, and reviewers specifically look for candidates who recognize that not every system should optimize for the same axis.

It’s also worth noting where this trade-off is deliberately not applied uniformly across the platform. A broader digital wallet product typically includes many features adjacent to the core ledger — spending analytics, budgeting suggestions, promotional cashback calculations, loyalty points — that touch money-adjacent numbers without being the authoritative balance itself. Applying the full weight of strict consistency, distributed locking, and durable coordinator logging to every one of these adjacent features would be a significant over-engineering mistake: it would slow down feature velocity and infrastructure cost for properties that don’t need it. The discipline this architecture calls for is knowing precisely where the strongly-consistent boundary should sit — around the ledger and nothing more — and resisting the temptation to either shrink that boundary (which risks real financial bugs) or expand it unnecessarily (which risks unjustified complexity and cost elsewhere in the platform).

The requirement is to comfortably support over one million requests per minute — roughly 16,700 requests/second sustained, with bursts several times higher during salary-disbursement windows or major sale/cashback events. Here’s how each layer scales while preserving strict consistency.

8.1 Sharding strategy

Accounts are sharded by accountId using consistent hashing across the distributed SQL cluster (CockroachDB-style range/hash sharding). Each shard is itself a Raft-replicated group (typically 3–5 replicas) providing local strong consistency and automatic leader failover. Because most transfers in a real wallet product are between users who are geographically and socially close (friends, family, local merchants), locality-aware sharding — placing accounts likely to transact together on the same shard, based on historical transaction graphs — meaningfully reduces the fraction of transfers that require the more expensive cross-shard 2PC path.

8.2 Reducing hot-account contention

A small number of accounts (large merchants, payroll disbursement accounts, popular peer-to-peer payment collectors) can receive an extremely disproportionate share of transactions, creating lock contention that a single account row can’t sustain even on a well-provisioned shard. The standard technique is sub-ledger sharding: split a single hot account’s balance across N internal sub-accounts (e.g., 16 or 64), route incoming credits round-robin or hash-based across sub-accounts, and only aggregate the true total balance lazily on read (with a background job periodically consolidating sub-accounts back down when load is low). This converts a single point of write contention into N independently-lockable rows, multiplying achievable write throughput for that account by roughly N.

8.3 Read scaling for balance checks

Pure balance-read requests (far more frequent than writes in most wallet products) are served from the distributed SQL cluster’s local replicas with a “read your own writes” consistency mode tied to the session, rather than always routing to the current Raft leader — this spreads read load across replicas while still guaranteeing a user never sees a balance older than their own most recent confirmed transaction. For extremely read-heavy, low-criticality use cases (e.g., a dashboard widget showing an approximate balance), a short-TTL cache (a few hundred milliseconds) can be layered in front, but never for any balance value used to authorize a subsequent debit.

8.4 Scaling the Transaction Coordinator tier

The Transaction Coordinator is horizontally scaled as a stateless-per-request service (each transfer gets assigned to any available Coordinator instance), with all durable state living in the Raft-replicated coordinator log rather than in-process memory — this means any Coordinator instance can pick up recovery of an in-flight transaction left behind by a crashed peer, since the durable log, not the process, is the source of truth for in-flight cross-shard transfers.

8.5 Rough capacity planning at target scale

It’s worth walking through concrete numbers, since interviewers often want to see that “1 million requests per minute” translates into an actual capacity plan rather than staying an abstract phrase. At ~16,700 requests/second sustained, if we estimate roughly 60% are pure balance reads, 30% are same-shard transfers, and 10% are cross-shard transfers, that’s approximately 10,000 reads/second, 5,000 same-shard writes/second, and 1,700 cross-shard writes/second at steady state. A well-tuned distributed SQL shard can typically sustain a few thousand serialized writes per second per Raft group before latency starts climbing meaningfully, which suggests provisioning on the order of low double-digit shard counts at baseline, with auto-scaling headroom to roughly double that during known peak events (salary day, major promotional windows). The Transaction Coordinator tier, handling only the 1,700 cross-shard writes/second, needs comparatively few instances, since each instance can typically drive many hundreds of concurrent two-phase-commit flows given the coordination work is largely I/O-bound waiting on shard round-trips rather than CPU-bound.

This kind of back-of-envelope sizing is exactly the sort of reasoning a strong system design interview answer demonstrates — not because the specific numbers need to be precisely correct, but because it shows the candidate can translate a scale requirement into concrete per-component load estimates, identify which tier is likely to become the bottleneck first (usually the Raft-replicated shards handling same-shard writes, given they bear the serialization cost directly), and reason about where to add capacity headroom ahead of known traffic spikes rather than purely reactively.

• Multi-AZ, Raft-replicated shards: every distributed SQL shard runs as a Raft group across at least three availability zones, so the loss of a single AZ never loses committed data and triggers automatic leader re-election within a couple of seconds.
• Durable coordinator log: as covered above, this is what makes cross-shard transaction recovery deterministic rather than a manual, error-prone incident-response exercise.
• Circuit breakers on the async periphery: if the Fraud Detection or Notification services degrade, this never blocks or slows the core transfer path, since they only consume already-committed events from Kafka — a clean illustration of the bulkhead pattern protecting the critical path from non-critical failures.
• Graceful rejection over silent inconsistency: during a severe network partition that prevents a shard’s Raft group from reaching quorum, the system deliberately rejects writes to that shard (returning a clear “temporarily unavailable” error) rather than allowing a minority partition to accept writes that could later conflict — a direct, intentional application of choosing consistency over availability (the “CP” choice in CAP) for the money-moving path specifically.
• Disaster recovery: cross-region asynchronous replication of the fully-committed ledger stream provides a warm standby region; a full regional failover is treated as a deliberate, tested runbook operation (with a small acceptable RPO measured in seconds) rather than an automatic failover, given the extremely high cost of an incorrect automatic failover decision in a financial system.

• Strong authentication: multi-factor authentication (MFA) required for high-value transfers and for any change to withdrawal destinations (linked bank accounts, cards), with step-up authentication challenges triggered by the Fraud Detection Service’s risk score.
• Encryption in transit and at rest: TLS everywhere between services (enforced via mutual TLS in a service mesh like Istio), and column-level encryption for sensitive fields (linked account numbers, KYC documents) in the distributed SQL cluster.
• Least-privilege access to the ledger: only the Ledger Service itself has write access to ledger tables; every other service, including the Wallet Service’s own orchestration logic, must go through it, minimizing the blast radius of a compromised service.
• Idempotency as a security control, not just a correctness one: beyond preventing accidental double-processing from retries, idempotency keys also blunt a class of replay attacks where a captured request is resent maliciously.
• Real-time fraud scoring: the Fraud Detection Service evaluates velocity (transactions per minute per account), device fingerprinting, and behavioral anomalies, and can place a temporary hold on an account’s ability to initiate new withdrawals pending manual review, without ever needing to touch or slow down the core ledger write path.
• Audit logging & regulatory compliance: every ledger mutation, every administrative action, and every authentication event is logged immutably, satisfying AML (Anti-Money-Laundering) and KYC (Know Your Customer) audit requirements common across financial regulators globally.
• Rate limiting and abuse prevention: per-account and per-device rate limits at the API Gateway prevent both scripted abuse and a class of denial-of-service attempts aimed at exhausting Coordinator or lock-manager capacity.

Security in a system like this is not a bolted-on layer added after the core transaction logic is built; it is inseparable from the correctness guarantees discussed throughout this tutorial. An attacker who can bypass idempotency enforcement can double-spend just as effectively as a genuine concurrency bug can, and an attacker who can read a stale, cached balance to authorize a fraudulent withdrawal exploits exactly the same weakness a badly-designed cache-invalidation race would create accidentally. This is why the strongly-consistent core described throughout this document — single-writer ledger access, transactional idempotency, serializable isolation — should be understood as security infrastructure every bit as much as it is correctness infrastructure; the two concerns converge almost completely in a financial system.

• Metrics (Prometheus + Grafana): transaction throughput and latency (p50/p95/p99) split by same-shard vs. cross-shard, 2PC prepare/commit success and abort rates, distributed lock wait times, Raft leader-election frequency per shard, and reconciliation discrepancy counts (which should be zero, always).
• Logging (ELK/EFK stack): structured logs correlated by transactionId and idempotencyKey across every service involved in a transfer, enabling an engineer to reconstruct the exact sequence of events for any single transaction on demand.
• Distributed tracing (Jaeger/Zipkin): a single trace spans the API Gateway, Wallet Service, Transaction Coordinator (including individual prepare/commit calls to each shard), and Ledger Service, making cross-shard latency bottlenecks immediately visible.
• Alerting: PagerDuty/OpsGenie alerts fire on any non-zero reconciliation discrepancy (treated as a Sev-1 incident, given the financial correctness implications), Raft quorum loss on any shard, elevated 2PC abort rates, and Coordinator recovery events (a crash-recovery cycle firing is itself worth investigating even if it resolved correctly).
• Business dashboards: total transaction volume, average transfer value, cross-shard transaction percentage (a useful signal for whether the sharding/locality strategy is working as intended), and fraud-hold rate.

• Containerized stateless services: the Wallet Service, Transaction Coordinator, Fraud Detection, and Notification services run on Kubernetes, built via a CI/CD pipeline with mandatory automated testing (including chaos and consistency-invariant tests) gating every deployment.
• Careful rollout for the Ledger and Coordinator services: given the correctness-critical nature of these components, deployments use canary rollouts with automatic rollback triggered by any reconciliation anomaly or elevated transaction failure rate — not just generic latency/error-rate thresholds.
• Multi-region posture: the distributed SQL cluster’s Raft groups are typically deployed within a single region (for latency reasons — cross-region Raft consensus adds significant round-trip latency to every write), with asynchronous cross-region replication of the committed ledger stream feeding a standby region for disaster recovery rather than for active-active writes.
• Infrastructure as Code: the entire stack, including Raft group topology, shard counts, and Kubernetes node pools, is defined in Terraform for reproducibility and auditable change history — itself a compliance-relevant property in financial systems.
• Change management: schema migrations to ledger tables follow a strict backward-compatible, multi-step rollout process (expand, migrate, contract) to avoid any window where old and new code paths disagree about the shape of a financial record.

13.1 Why a distributed SQL database (not a traditional NoSQL store)

The wallet’s core data — accounts, balances, ledger entries — is exactly the kind of workload relational databases were built for: strong schemas, multi-row ACID transactions, and rich consistency guarantees. Traditional single-node relational databases (PostgreSQL, MySQL) don’t horizontally scale writes easily, while classic NoSQL stores (Cassandra, DynamoDB) trade away the strong, multi-row transactional guarantees this domain requires. Distributed SQL databases (CockroachDB, Google Spanner, YugabyteDB) were built specifically to close this gap: they provide the horizontal scalability of NoSQL with the ACID transactional guarantees of traditional relational databases, using Raft (or Paxos, in Spanner’s case) consensus under the hood for each shard’s replication.

13.2 Why caching is used sparingly, and never for balances

Caching is the default first instinct for scaling reads in most systems, but here it must be applied with real discipline: a cached balance is, by definition, a value that could already be stale relative to the true, currently-committed source of truth. We do use caching for genuinely non-critical, read-only reference data (currency exchange rates refreshed every few seconds, user profile metadata, transaction history pagination), but every balance check that will be used to authorize a subsequent write reads directly from the strongly-consistent data layer, with no cache in between.

13.3 Load balancing strategy

The L7 Load Balancer uses least-connections balancing for the stateless Wallet Service tier. Within the distributed SQL cluster itself, each shard’s Raft group elects a leader that serves all writes for that shard, while read load can be spread across replicas using the database’s native follower-read capability for non-authorizing reads, giving read scalability without compromising the write-path’s strong consistency guarantee.

Below is a representative Java implementation sketch of the Wallet Service’s core transfer endpoint for the same-shard case, showing idempotency handling and optimistic concurrency control via a version column.

@RestController
@RequestMapping("/api/v1/transfers")
public class TransferController {

    private final AccountRepository accountRepository;
    private final LedgerRepository ledgerRepository;
    private final IdempotencyRepository idempotencyRepository;

    public TransferController(AccountRepository accountRepository,
                              LedgerRepository ledgerRepository,
                              IdempotencyRepository idempotencyRepository) {
        this.accountRepository = accountRepository;
        this.ledgerRepository = ledgerRepository;
        this.idempotencyRepository = idempotencyRepository;
    }

    @Transactional(isolation = Isolation.SERIALIZABLE)
    @PostMapping
    public ResponseEntity<TransferResponse> transfer(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @Valid @RequestBody TransferRequest request) {

        Optional<IdempotencyRecord> existing =
                idempotencyRepository.findByKey(idempotencyKey);

        if (existing.isPresent()) {
            return ResponseEntity.ok(existing.get().getCachedResponse());
        }

        Account sender = accountRepository.findByIdForUpdate(request.getSenderId());
        Account recipient = accountRepository.findByIdForUpdate(request.getRecipientId());

        if (sender.getBalance().compareTo(request.getAmount()) < 0) {
            throw new InsufficientFundsException(sender.getId());
        }

        String transactionId = UUID.randomUUID().toString();

        LedgerEntry debit = new LedgerEntry(transactionId, sender.getId(),
                EntryType.DEBIT, request.getAmount(), Instant.now());
        LedgerEntry credit = new LedgerEntry(transactionId, recipient.getId(),
                EntryType.CREDIT, request.getAmount(), Instant.now());

        ledgerRepository.save(debit);
        ledgerRepository.save(credit);

        sender.setBalance(sender.getBalance().subtract(request.getAmount()));
        sender.setVersion(sender.getVersion() + 1);
        recipient.setBalance(recipient.getBalance().add(request.getAmount()));
        recipient.setVersion(recipient.getVersion() + 1);

        accountRepository.save(sender);
        accountRepository.save(recipient);

        TransferResponse response = new TransferResponse(transactionId, "COMPLETED");
        idempotencyRepository.save(new IdempotencyRecord(idempotencyKey, response));

        return ResponseEntity.ok(response);
    }
}

And here is a simplified sketch of the Transaction Coordinator’s prepare phase for a cross-shard transfer, showing the durable-log-first pattern:

public class TransferCoordinator {

    private final CoordinatorLogRepository logRepository;
    private final ShardClient shardClient;

    public TransferCoordinator(CoordinatorLogRepository logRepository,
                               ShardClient shardClient) {
        this.logRepository = logRepository;
        this.shardClient = shardClient;
    }

    public TransferResult executeCrossShardTransfer(TransferRequest request) {

        String transactionId = UUID.randomUUID().toString();

        // Durably record intent BEFORE touching either shard
        logRepository.save(new CoordinatorLogEntry(
                transactionId, CoordinatorPhase.BEGIN, request));

        // Prepare phase across both shards
        boolean senderPrepared = shardClient.prepareDebit(
                request.getSenderShard(), request.getSenderId(),
                request.getAmount(), transactionId);
        boolean recipientPrepared = shardClient.prepareCredit(
                request.getRecipientShard(), request.getRecipientId(),
                request.getAmount(), transactionId);

        if (!senderPrepared || !recipientPrepared) {
            logRepository.save(new CoordinatorLogEntry(
                    transactionId, CoordinatorPhase.ABORT, request));
            shardClient.rollback(request.getSenderShard(), transactionId);
            shardClient.rollback(request.getRecipientShard(), transactionId);
            return TransferResult.failed(transactionId,
                    "insufficient funds or shard unavailable");
        }

        // Durably record the commit decision BEFORE executing it
        logRepository.save(new CoordinatorLogEntry(
                transactionId, CoordinatorPhase.COMMIT, request));

        shardClient.commit(request.getSenderShard(), transactionId);
        shardClient.commit(request.getRecipientShard(), transactionId);

        return TransferResult.success(transactionId);
    }
}

15.1 Patterns used

15.2 Anti-patterns to avoid

As with any system design, most of the serious failure modes here are not exotic — they are ordinary-looking shortcuts (skipping a transaction wrapper “just this once,” caching a balance “just for this one dashboard”) that happen to interact catastrophically with concurrency and failure conditions. The discipline of never taking those shortcuts on the core balance path, even under deadline pressure, is arguably the single most important cultural practice for a team operating a system like this.

A useful heuristic for any engineer joining a team that operates a system like this: before touching any code path that reads or writes a balance, ask “what happens if this exact code runs twice, concurrently, on the same account, right now?” If the answer isn’t immediately and confidently “nothing bad, because of X guarantee,” that code path needs more scrutiny before it ships — this single habit of mind catches the overwhelming majority of the subtle correctness bugs that plague financial systems in practice.

Across all of these examples, the same core principle recurs: every production financial system that has scaled successfully has done so by keeping a small, rigorously strongly-consistent ledger core, and pushing everything that can tolerate delay or approximation (notifications, analytics, fraud model training, customer-facing dashboards) into a separate, eventually-consistent periphery — the exact separation of concerns this tutorial’s architecture is built around.

A single instance, however large, has a hard ceiling on write throughput and represents a single point of failure for the entire platform’s balances. Distributed SQL databases provide the same transactional guarantees while allowing both storage and write throughput to scale horizontally by adding more shards, and provide automatic failover within each shard’s Raft group rather than requiring manual intervention if the single instance fails.

This is modeled as a two-step process: first, a strongly-consistent internal transaction debits the user’s wallet and records the withdrawal as PENDING_EXTERNAL; second, an asynchronous worker calls the external banking rail (ACH, SWIFT, a card network) and, upon confirmation, marks the withdrawal COMPLETED, or, upon failure, executes a compensating credit transaction back to the user’s wallet. The core wallet ledger’s strong consistency is preserved throughout; only the “did the external bank actually receive the money” step is inherently asynchronous, because it depends on a system outside our control.

This is precisely what Serializable isolation (or explicit row locking) prevents: the database guarantees that the two concurrent transactions behave as if they executed one after another, not interleaved, so the second transaction to actually execute sees the already-reduced balance from the first and correctly fails with insufficient funds, rather than both succeeding against a stale read.

Yes — each currency is modeled as a logically separate ledger/balance for the same account (or as separate accounts entirely, linked by ownership), with currency conversion handled as an explicit, auditable transaction pair (debit source currency, credit destination currency, at a recorded exchange rate) rather than an implicit conversion, preserving the same double-entry auditability for currency exchange as for any other transfer.

Beyond standard load testing, this system specifically needs invariant-based property testing: run many thousands of concurrent, randomly-interleaved transfers between a fixed pool of test accounts under injected failures (killed Coordinator processes, simulated network partitions, forced Raft leader elections), then assert that the sum of all account balances plus all pending transaction amounts exactly equals the sum before the test began. This kind of invariant-checking chaos test is far more valuable here than simple throughput benchmarking, because it directly targets the correctness property the whole architecture exists to guarantee.

A single global lock would indeed make correctness trivial to reason about, but it would also mean the entire platform’s transaction throughput is bottlenecked by whatever a single lock and a single execution thread can process — nowhere close to the required 1,000,000+ requests per minute. The entire design of sharding by accountId, together with fine-grained per-account (or per-sub-ledger) locking rather than a single global lock, exists specifically to let unrelated transactions (Alice paying Bob, and Charlie paying Dana, on unrelated accounts) proceed fully in parallel with zero contention between them, while still serializing only the specific operations that actually touch the same account.

A blockchain achieves agreement on transaction ordering through a public, often computationally expensive consensus mechanism (proof-of-work, proof-of-stake) designed to work without any trusted central operator, which is powerful for decentralized trust but comes with materially lower transaction throughput and higher latency than a permissioned, centrally-operated Raft-based distributed database. Since a company operating its own digital wallet platform is already a trusted central operator (it holds the KYC relationship and regulatory responsibility for its users), it can use the much faster, purpose-built consensus approach described in this tutorial rather than paying blockchain’s decentralization costs for a property (trustless operation) the product doesn’t actually need.

Taken together, these decisions describe a system that treats “a balance” not as a number to be updated but as a running sum of an immutable, append-only history — and one where every hard problem (concurrency, cross-shard atomicity, retry safety, fault recovery) is resolved by pushing the guarantee down to the lowest possible layer (ACID transactions, Raft consensus, durable coordinator logs) rather than reinventing correctness at the application layer. A strong system design answer for this question is one that names, at every step, exactly which layer is responsible for which guarantee, and why the strongly-consistent core is deliberately kept as small as it can be while still covering every path where money actually moves.