Open any banking app and tap the home screen. A number appears — your balance. It feels like the simplest possible feature: read one number from a database, show it on screen. But ask yourself a harder question: what happens if, at the exact millisecond you tap that button, the bank’s primary database server catches fire (figuratively — a disk fails, a data centre loses power, a deployment goes wrong) and a brand-new server has to take over as the primary? Do you see a stale number? An error? Or, worst of all, a wrong number that’s higher than what you actually have — leading you to spend money you don’t have?

This is the exact problem this tutorial solves: designing a real-time account balance query system for a digital bank that must (a) answer at massive scale, comparable to millions of requests per minute during peak hours like payday or a major shopping event, (b) respond in well under a second, and (c) never — under any circumstances, including an active database failover — show a balance that is stale in a way that could mislead the customer or allow an overdraft that shouldn’t happen.

1.1 A Short History of the Problem

Traditional banks built their core systems on mainframes in the 1970s–1990s — enormously reliable but slow-changing systems where a “balance inquiry” was a batch-friendly operation, often only updated once per day (this is why older bank statements show “as of end of day” balances). The rise of digital-only “neobanks” (Monzo, Revolut, Chime, N26) in the 2015–2020 era forced a shift: customers now expect their balance to reflect a transaction the instant it happens — tap to pay for coffee, and the balance updates before you’ve put your phone back in your pocket. This shift from “batch-accurate” to “real-time-accurate” banking is what makes this a genuinely hard distributed systems problem, because real-time freshness and rock-solid correctness are usually in tension, and that tension gets sharply worse during a database failover, which is exactly when a naive system is most likely to serve wrong data.

It’s worth being explicit about what “in doubt” means operationally, since it’s easy to state as a principle but harder to implement precisely: it means any moment where the system cannot mathematically prove — via a version number checked against a known-good replication state — that the data it’s about to serve reflects every transaction that was ever acknowledged as committed. Outside of a failover, this bar is met essentially all the time, which is exactly why the system can serve the overwhelming majority of its million-plus requests per minute from a fast cache with confidence. The entire point of the design that follows is to make that same bar hold true even in the handful of seconds when it’s hardest to meet.

1.2 Why This Matters Beyond Banking

The core lesson generalises to any system where a stale read is actively dangerous rather than merely annoying: inventory counts for the last unit of a product, seat availability for a flight, or a medical dosage record. Wherever “slightly wrong” is worse than “briefly unavailable,” the patterns in this tutorial — synchronous replication, fencing tokens, read staleness bounds, and fail-closed design — apply directly.

2.1 The Ask

Design a system for a digital bank to support real-time account balance checks that must remain accurate even during a database failover event.

2.2 Why This Is Hard

• Massive read volume, spiky by nature. Balance checks spike hard around payday, weekends, and major sale events — a system built for steady load will fall over exactly when it matters most.
• Correctness during failure is the actual requirement, not a nice-to-have. Any distributed database can and will fail over — a disk dies, a network partition happens, an operator makes a mistake during a deploy. The system must be explicitly designed so that a failover event, which might last anywhere from a few seconds to a couple of minutes, never produces an incorrect balance.
• Caching is both the solution and the danger. Caching is the only realistic way to hit sub-second latency at huge scale, but a cache is exactly the kind of component that can confidently serve very wrong, very stale data during and after a failover if not designed carefully.
• Regulatory expectations. Banking regulators in most countries require operational resilience — demonstrable evidence that critical customer-facing functions (balance, payments) degrade safely and recover quickly from infrastructure failures.
• Read/write coupling. Balance isn’t an independent fact — it’s the sum of a transaction ledger. The read path (balance check) and write path (transactions posting) must stay in sync even while the underlying storage topology is changing underneath them.

2.3 Functional Requirements

1. Given an authenticated account holder, return the current account balance (available balance and ledger/posted balance) in real time.
2. The balance must reflect all transactions that have been confirmed/posted as of the read, with a clearly bounded and documented maximum staleness in any degraded mode.
3. During a database failover, the system must either serve a correct balance or clearly indicate temporary unavailability — it must never silently serve an incorrect number.
4. Support multiple account types (checking, savings, credit) with different balance semantics (available vs. posted vs. pending holds).
5. Provide an audit trail sufficient to reconstruct exactly what balance was shown to a customer at any point in time, for dispute resolution.

2.4 Non-Functional Requirements

2.5 What “Real-Time” Actually Means Here

It’s worth pinning down “real-time” precisely, since the term is often used loosely. In this system, real-time means the balance shown reflects every transaction that has been fully posted (settled) as of the moment the customer taps the button, within the P99 latency budget defined above — it does not mean the balance instantly reflects a transaction that is still pending or mid-flight at another bank or card network, which is a separate, well-understood distinction (available vs. posted balance, covered in Section 3.2) rather than a failure of real-time-ness. Being precise about this distinction matters both for setting the right engineering targets and for setting the right customer expectations in the app’s UI.

2.6 Sizing the Problem

Before designing anything, it helps to sanity-check the scale we’re solving for against a realistic customer base. A digital bank with, say, 20 million active customers, where a meaningful fraction check their balance multiple times around payday within a short window, can plausibly generate the million-plus-requests-per-minute peak this tutorial targets — this isn’t a hypothetical worst case invented for the exercise, it’s a genuinely representative peak for a bank of that size, which is exactly why the architecture treats it as the design point rather than an unlikely edge case to handle apologetically.

Before we draw the architecture, let’s build shared vocabulary — each term explained with a real-life analogy, a beginner example, and how it shows up in this system.

3.1 Ledger vs. Balance

What: The ledger is the append-only list of every transaction (deposits, withdrawals, holds). The balance is a derived number — the running sum of that ledger.
Analogy: Think of your chequebook register — every line is a transaction, and the number at the bottom is the balance. You never erase a line to “fix” the balance; you add a new line.
In our system: The database’s real source of truth is the transaction ledger table; the “balance” shown on the home screen is either computed on read or maintained as a continuously-updated running total that’s mathematically derivable from the ledger at any time — this dual view is what lets us reconcile and prove correctness after any incident, including a failover.

3.2 Available Balance vs. Posted (Ledger) Balance

What: The posted balance reflects fully settled transactions. The available balance additionally subtracts pending holds (e.g., a gas station’s temporary $100 hold when you swipe your card, before the actual $40 charge settles).
In our system: Both numbers are computed from the same ledger plus a separate holds table, and the API always makes clear which one it’s returning, since showing the wrong one is a very common source of customer confusion and complaints.

3.3 Synchronous vs. Asynchronous Replication

What: In synchronous replication, a write is only acknowledged as “committed” once at least one replica has confirmed it received the write too — slower per-write, but guarantees no committed data is lost if the primary dies immediately after. In asynchronous replication, the primary acknowledges the write immediately and replicates in the background — faster, but a crash before replication catches up can lose the most recent writes.
Analogy: Synchronous is like mailing a letter and waiting for a signed receipt before considering it “sent.” Asynchronous is like dropping it in the mailbox and trusting it’ll arrive.
In our system: Every balance-changing write (a transaction posting) uses synchronous replication to at least one standby — this single decision is the foundation of failover accuracy, because it guarantees that whichever node is promoted to primary after a failure has every committed transaction.

3.4 Failover and Fencing Tokens

What: Failover is the process of promoting a standby database replica to become the new primary after the old primary fails. A fencing token is a strictly increasing number issued to whichever node currently holds the “I am the primary” lease — if an old, supposedly-dead primary suddenly comes back to life (a “zombie” primary), any write it tries to make carries an old fencing token and is rejected, preventing two nodes from both believing they’re the primary at once (split-brain).
Analogy: Think of a relay race baton with a serial number stamped on it. If a runner who dropped the baton earlier tries to keep running with a photocopy of an old baton, the finish-line judge checks the serial number and rejects it, because a runner with a higher-numbered real baton has already passed.

3.5 Quorum Reads and Writes

What: In a cluster of N replicas, a quorum write requires acknowledgement from a majority (e.g., 2 of 3) before being considered committed, and a quorum read requires reading from a majority before trusting the result — mathematically guaranteeing that a read quorum and a write quorum always overlap by at least one node, so a read can never miss the latest committed write.
In our system: Used for the small, security- and correctness-critical metadata about “who is currently the primary,” coordinated via a consensus system like etcd (backed by the Raft algorithm), separate from the bulk transaction data itself.

3.6 Read-Your-Writes Consistency

What: A guarantee that after a user (or the system, on their behalf) performs a write, any subsequent read by that same user reflects that write — even if other users might briefly see stale data on a lagging replica.
Analogy: If you hand a librarian a new book to add to the shelf and then immediately ask “do you have this book?”, she should say yes — even if the catalogue system hasn’t finished updating for everyone else yet.
In our system: Achieved by tagging each write with a version/sequence number and having the read path check that the replica it’s about to read from has caught up to at least that version before trusting it — otherwise routing to a more current source.

3.7 CQRS (Command Query Responsibility Segregation)

What: Separating the “write model” (how transactions are recorded) from the “read model” (how balances are queried), often using different, independently-scaled storage optimised for each access pattern.
In our system: Transaction posting (writes) goes through the core ledger database; balance reads are served from a purpose-built, heavily-cached read path that is kept in sync with the ledger via a change-data-capture stream — this lets us scale reads independently of writes, which matters enormously at a million-requests-a-minute read volume.

3.8 Change Data Capture (CDC)

What: A technique for streaming every row-level change (insert/update) out of a database’s transaction log in near-real-time, so downstream systems (caches, search indexes, read replicas) can stay continuously synchronised without the source database needing to know they exist.
Analogy: Like a court stenographer’s live transcript feed being piped out to multiple screens in real time, rather than each screen having to separately ask the judge “what did you just say?”

3.9 Write-Ahead Log (WAL)

What: Before a database changes any actual data on disk, it first writes a record of the intended change to an append-only log file. If the database crashes mid-operation, it can replay the WAL on restart to recover to a consistent state.
Analogy: A pilot’s pre-flight checklist read aloud and recorded before touching any control — if something goes wrong, investigators can replay exactly what was intended, step by step.
In our system: The WAL is the actual mechanism synchronous replication ships to standbys — a standby is only “caught up” once it has applied the same WAL entries as the primary, and CDC tools like Debezium work by tailing this same WAL rather than polling tables directly, which is both faster and lower-overhead.

3.10 Split-Brain

What: A dangerous failure mode in a distributed system where a network partition or failed health check causes two nodes to simultaneously believe they are the sole authoritative primary, both accepting writes that can directly conflict with each other.
Analogy: Imagine two co-signers on a joint account, each unaware the other is also currently approving withdrawals, both thinking they have the final say — the account can be drained twice over before anyone notices the conflict.
In our system: Prevented entirely by the combination of consensus-based leader election (only one node can win a Raft election at a time) and fencing tokens (even if an old primary doesn’t know it’s been demoted, its writes are rejected by every other component that checks the token).

3.11 Idempotency in the Write Path

What: As covered generally in distributed systems, ensuring the same operation applied twice has the same effect as applying it once.
In our system: Every transaction posting carries a unique transaction ID; if a client retries a posting request (for example, because a network timeout occurred right as a failover was happening and the client isn’t sure if the original request succeeded), the Transaction Posting Service recognises the duplicate ID and returns the original result rather than posting the same deposit or withdrawal twice — a duplicated transaction would be just as damaging to balance accuracy as a lost one.

3.12 Two-Phase Commit vs. This System’s Approach

What: Two-Phase Commit (2PC) is a classic protocol for coordinating an atomic write across multiple independent databases — every participant must “prepare” before any can “commit,” guaranteeing all-or-nothing across systems that don’t share a single log.
Why we don’t use it here: 2PC is notoriously fragile under partial failures (a coordinator crash mid-protocol can leave participants blocked indefinitely) and adds significant latency. Because our ledger and its replicas share a single write-ahead log rather than being independent databases, we get atomicity “for free” from PostgreSQL’s own transaction guarantees plus synchronous replication, without needing a separate distributed-commit protocol at all — a good example of designing the data model to avoid a hard distributed-systems problem rather than solving it head-on.

Now let’s build the system. Every box below explicitly states its architectural layer (API Gateway, Load Balancer, Service, Cache, Database, etc.), as requested.

4.1 High-Level Architecture Diagram

4.2 Component Responsibilities

4.3 Balance Read Sequence (Normal Operation)

4.4 Balance Read Sequence (During Failover)

5.1 The Balance Query Service

This stateless service is the front door for every balance read. On each request, it first attempts the fast path — a cache lookup — but never trusts that cache blindly. Every cached entry carries a monotonically increasing ledger version number (effectively a sequence number tied to the write-ahead log position at which it was computed). The Staleness Guard compares that version against the version the currently-healthy replica pool has confirmed applying; only if they’re within an acceptable bound does the service serve the cached value.

@Service
public class BalanceQueryService {

    private final BalanceCacheClient cache;
    private final StalenessGuard stalenessGuard;
    private final ReadReplicaClient replicaClient;
    private final FailoverCoordinatorClient failoverClient;

    private static final long MAX_ACCEPTABLE_LAG_MS = 2000; // 2 second freshness bound

    public BalanceResponse getBalance(String accountId) {
        // 1. Check whether the cluster is mid-failover before anything else
        ClusterState state = failoverClient.currentState();
        if (state.isFailoverInProgress()) {
            return degradedResponse(accountId, state);
        }

        // 2. Try the cache first - the fast path for the overwhelming majority of traffic
        Optional<CachedBalance> cached = cache.get(accountId);
        if (cached.isPresent()) {
            boolean fresh = stalenessGuard.isFresh(
                cached.get().getLedgerVersion(), MAX_ACCEPTABLE_LAG_MS);
            if (fresh) {
                return BalanceResponse.ok(cached.get());
            }
        }

        // 3. Cache miss or stale - fall back to a versioned replica read
        ReplicaBalanceResult result = replicaClient.readWithVersionCheck(accountId);
        if (result.isTrustworthy()) {
            cache.putAsync(accountId, result.toCachedBalance()); // refresh cache, non-blocking
            return BalanceResponse.ok(result);
        }

        // 4. Even the replica can't be trusted right now - fail closed
        return degradedResponse(accountId, state);
    }

    private BalanceResponse degradedResponse(String accountId, ClusterState state) {
        CachedBalance lastKnownSafe = cache.getLastKnownSafe(accountId);
        return BalanceResponse.degraded(
            lastKnownSafe.getBalance(),
            lastKnownSafe.getAsOfTimestamp(),
            "TEMPORARY_REDUCED_FRESHNESS");
    }
}

5.2 The Staleness Guard

The guard’s job is narrow but critical: answer “can I trust this data right now?” It maintains a lightweight, frequently-refreshed view of each replica’s applied write-ahead-log position, compared against the primary’s latest committed position (obtained cheaply from the Failover Coordinator’s health-check stream rather than querying the primary directly on every request, which would create its own bottleneck).

@Component
public class StalenessGuard {

    private final ReplicaLagMonitor lagMonitor;

    public boolean isFresh(long dataVersion, long maxAcceptableLagMs) {
        ReplicaLagSnapshot snapshot = lagMonitor.currentSnapshot();

        if (snapshot.isFailoverInProgress()) {
            return false; // never trust anything mid-failover
        }

        long lagMs = snapshot.estimateLagMs(dataVersion);
        return lagMs <= maxAcceptableLagMs;
    }
}

5.3 The Reconciliation Service

Even with synchronous replication, fencing tokens, and a versioned cache, the system doesn’t rely on those mechanisms alone for correctness — it continuously verifies its own work. The Reconciliation Service runs on a tight schedule (e.g., every few seconds for recently-active accounts, less frequently for dormant ones), independently recomputing each account’s true balance directly from the ledger’s transaction rows and comparing it against whatever the cache and replicas are currently serving.

@Service
public class ReconciliationService {

    private final LedgerRepository ledgerRepository;
    private final BalanceCacheClient cache;
    private final AlertingClient alerting;

    @Scheduled(fixedDelay = 5000) // every 5 seconds for active accounts
    public void reconcileActiveAccounts() {
        List<String> activeAccountIds = ledgerRepository.recentlyActiveAccountIds();

        for (String accountId : activeAccountIds) {
            BigDecimal trueBalance = ledgerRepository.computeBalanceFromLedger(accountId);
            Optional<CachedBalance> served = cache.get(accountId);

            if (served.isPresent() &&
                served.get().getBalance().compareTo(trueBalance) != 0) {

                // Any drift at all is treated as a serious correctness incident
                alerting.pageOnCall(
                    "BALANCE_DRIFT_DETECTED",
                    accountId, trueBalance, served.get().getBalance());

                // Self-heal: overwrite the incorrect cached value immediately
                cache.put(accountId, CachedBalance.fromLedger(trueBalance,
                    ledgerRepository.currentVersion(accountId)));
            }
        }
    }
}

This service is deliberately independent of the request path — it doesn’t wait for a customer to check their balance to discover a problem. Any drift it finds is both a signal to page an engineer immediately and an opportunity to self-heal the affected cache entry before a customer ever sees the incorrect value, turning what could be a customer-facing incorrect-balance incident into an internal, silently-corrected anomaly the vast majority of the time.

Let’s trace the complete life of a single balance change — from the moment a transaction posts to the moment a customer’s app shows the new number — through the six discrete stages the system relies on.

8.1 Capacity Math

1,000,000 requests/minute ≈ 16,667 requests/second on average, with payday and weekend evening peaks pushing well above that — we design for roughly 60,000–80,000 requests/second at peak, similar in shape to other real-time financial systems, but here nearly all of that volume is reads, which changes the scaling story significantly compared to a write-heavy system.

8.2 Read Scaling via CQRS

Because balance is a read-dominated workload (customers check their balance far more often than they transact), separating the read model from the write model is the single biggest scalability lever. The read path — cache plus replica pool — can be scaled horizontally to any size independent of the primary database, which only needs to handle the much smaller volume of actual transaction writes.

This is the crux of the entire design, so let’s go deep.

9.1 Synchronous Replication as the Foundation

The Primary Ledger DB replicates every committed transaction synchronously to at least one standby before acknowledging the write to the caller. This is a deliberate latency-for-correctness trade-off: a write takes a few extra milliseconds to complete, but in exchange we get an ironclad guarantee — whichever node becomes the new primary after a failure has every transaction that was ever acknowledged as committed. Without this, an asynchronously-replicated system could lose the last few seconds of transactions during a crash, which is exactly the scenario that could cause an incorrect balance to be shown.

9.2 Consensus-Based Failover with Fencing Tokens

When the Failover Coordinator (built on a Raft-based consensus store like etcd, via a tool such as Patroni for PostgreSQL) detects that the primary has stopped heartbeating, it runs a leader election among the healthy standbys. The winning standby is promoted and issued a new, strictly higher fencing token. Every write-path component (the Transaction Posting Service, and any replication client) is required to include the current fencing token with every operation; if the old primary ever comes back online confused about its own status, its writes carry a stale token and are rejected outright — preventing the classic and dangerous “split-brain” scenario where two nodes both believe they’re the primary and accept conflicting writes.

9.3 What Happens to Reads During the Failover Window

This is where the Staleness Guard and degraded-mode response (Section 4.4) do their work. From the moment the Failover Coordinator detects a failure until a new primary is confirmed and fencing is in place — typically single-digit seconds with a well-tuned setup — the Balance Query Service refuses to serve any read it cannot vouch for, falling back to the last version it can mathematically prove was correct, clearly labelled with its timestamp. Once the new primary is confirmed, the CDC pipeline and replica pool catch up within milliseconds, and normal fast-path service resumes automatically, with no special “recovery mode” logic needed in the read path — it simply starts trusting fresh data again as soon as the Staleness Guard’s checks pass.

9.4 Multi-Region Design

For disaster recovery beyond a single data centre failure, the ledger is asynchronously replicated to a secondary region — asynchronous here because synchronous cross-region replication would add too much latency to every write. This means a full regional disaster could lose the last few seconds of transactions in the worst case; the system compensates with a fast, well-drilled runbook and, crucially, treats a full regional failover as a much rarer, higher-severity event than an in-region primary failover, with correspondingly different (and more conservative) read-availability trade-offs communicated clearly to customers if it ever occurs.

9.5 Disaster Recovery & Backup

Continuous write-ahead-log archiving to durable, geographically-separate object storage supports point-in-time recovery, combined with daily full snapshots retained per financial record-keeping requirements (often seven years or more). Recovery Point Objective (RPO) for a single-node failure is effectively zero, thanks to synchronous replication; for a full regional disaster, RPO is bounded by the asynchronous cross-region replication lag, typically a few seconds. Recovery Time Objective (RTO) for an in-region primary failure is targeted at under 15 seconds end-to-end (detection + election + fencing + traffic resumption); for a full regional failover, RTO is targeted at a few minutes, given the additional coordination required.

9.6 Handling the “Zombie Primary” Scenario in Detail

It’s worth walking through the most subtle failure mode this design guards against, since interviewers often probe exactly this: a primary database experiences a long garbage-collection pause or a brief network partition that makes it unreachable from the Failover Coordinator, but the process itself is still alive and still believes it’s the primary. The Failover Coordinator, seeing missed heartbeats, promotes a standby and issues it a new fencing token. When the original primary’s pause ends or its network connectivity is restored, it may attempt to continue accepting writes — but every downstream component (replication targets, the Transaction Posting Service if it still holds a connection to the old primary) is configured to check the current fencing token before accepting anything from it, and rejects it because its token is now stale. The “zombie” primary is thus fenced off from doing any further damage, even though nothing ever explicitly told it to stop — it’s forced into irrelevance by every other component simply refusing to trust it anymore. This is a more robust guarantee than relying on the zombie process itself to notice it’s been demoted and gracefully step down, since a process that’s confused enough to still think it’s primary cannot be trusted to reliably notice its own confusion.

9.7 Circuit Breaking on the Read Path

Just as an external dependency like a credit bureau needs a circuit breaker in other financial systems, the Balance Query Service applies the same pattern to its own replica pool. If a growing fraction of replica reads are failing or timing out — a leading indicator that something is wrong even before the Failover Coordinator has formally declared a failover — the circuit breaker trips and the service proactively shifts into degraded mode rather than continuing to send doomed requests to an unhealthy replica and eating the full timeout latency on each one. This shortens the window during which customers experience elevated latency, even if it slightly widens the window during which they see degraded (but still honest and correct) data.

• mTLS between all services: Prevents a compromised pod from impersonating the Balance Query Service or Failover Coordinator.
• Strong authentication on every request: Balance data is amongst the most sensitive information a bank holds; every request requires a valid, short-lived session token, re-validated at the API Gateway on every call.
• Fencing tokens double as a security control: Beyond correctness, rejecting writes from an unfenced or stale-token source also protects against certain classes of infrastructure-level attacks where an attacker might try to resurrect an old, compromised database instance.
• Encryption at rest and in transit: The ledger, replicas, cache, and CDC stream are all encrypted, with especially strict key management for the primary ledger given its regulatory sensitivity.
• Audit logging of every balance read: Not just for security forensics, but because regulators and dispute-resolution processes may require proof of exactly what balance was shown to a customer and when.

10.1 Compliance & Regulatory Considerations

Real-time balance accuracy sits directly inside banking regulation, not adjacent to it. Most jurisdictions’ operational resilience frameworks (for example, the UK’s PRA/FCA operational resilience rules, or similar frameworks elsewhere) explicitly require banks to identify their “important business services” — balance and payment availability are almost always on that list — define maximum tolerable outage/impact thresholds for them, and demonstrate through testing (not just documentation) that the bank can stay within those thresholds during severe but plausible disruptions, including infrastructure failures exactly like the database failover this system is built around. This is why the failover drills described in Section 9.5 aren’t just good engineering practice here — they’re often a direct regulatory expectation, with evidence retained for examiners. Similarly, financial record-keeping rules typically require that the ledger (the true source of truth) be retained, auditable, and reconstructable for many years, which is part of why the architecture treats the ledger’s WAL-based durability as sacred and treats the cache and replicas purely as accelerators built on top of it, never as an alternate source of truth.

Distributed tracing tags every request with a trace ID that flows from the API Gateway through the cache, staleness guard, and replica/primary calls, making it possible to reconstruct exactly which data source served a specific customer’s balance at a specific moment — essential both for debugging and for dispute investigations.

11.1 Alerting Philosophy

Reconciliation drift and any degraded-mode response outside of a known, planned failover event are treated as the highest-severity alert category the platform has — paging immediately, day or night — because they represent the one failure mode (an incorrect balance shown to a real customer) that the entire architecture exists to prevent. Latency and capacity alerts, while important, are tiered below this, since they’re recoverable without direct customer harm in the way a wrong balance is.

• Containerised microservices on Kubernetes, with the Balance Query Service and Staleness Guard scaled independently from the Transaction Posting Service, reflecting their very different traffic shapes.
• Canary deployments for the read path: Any change to caching or staleness logic is rolled out to a small percentage of traffic first, with reconciliation drift monitored closely before wider rollout — this is one of the highest-risk categories of change in the whole system.
• Database topology changes are never “just deployed”: Adding or removing a replica, or changing replication mode, goes through a controlled runbook with an explicit failover drill validating the new topology before it takes production traffic.
• Infrastructure as Code ensures every region’s stack — including the exact replication and consensus configuration — is identical and reproducible, which matters enormously for passing banking regulatory audits.

Data-residency constraints in banking are strict; deployments are often regionally isolated with careful attention to which region a specific customer’s data may legally live in, rather than freely spreading a single dataset across the globe as some other systems in this tutorial series can. Every deployment change to the data tier goes through the same regulatory-audit-friendly runbook process, because the same infrastructure changes that could accidentally trigger a failover during normal traffic could, if uncontrolled, also produce the very customer-facing incident the whole architecture exists to prevent.

13.1 Why PostgreSQL with Synchronous Standbys (Not a Pure NoSQL Store)

The ledger is stored in a relational database (e.g., PostgreSQL) chosen specifically for its mature support for ACID transactions and synchronous streaming replication with fine-grained control (e.g., synchronous_commit settings, quorum-based synchronous replica sets). While a NoSQL store might offer higher raw write throughput, the ledger’s write volume is comparatively modest (it’s the read volume that’s enormous), and the correctness guarantees a mature relational database provides for exactly this failover scenario are hard to replicate elsewhere without significant custom engineering.

13.2 Cache Design: Versioned, Not Just TTL-Based

Each cache entry stores the balance alongside the ledger version it reflects and a wall-clock as_of timestamp. This lets the Staleness Guard reason precisely about freshness rather than guessing based on a fixed expiration window, and it lets the client app render an honest “as of” indicator to the customer whenever the data isn’t guaranteed to be instantaneous.

{
  "account_id": "acc_9f21ac",
  "available_balance": 1284.32,
  "posted_balance": 1310.00,
  "currency": "USD",
  "ledger_version": 40213,
  "as_of": "2026-08-04T09:12:41.203Z",
  "freshness": "REALTIME"
}

13.3 Load Balancing Algorithm Choice

The regional load balancer uses least-outstanding-requests rather than round-robin, since balance-check latency is variable depending on whether a request resolves from cache (sub-10 ms) or falls through to a replica read (higher latency) — routing new requests away from already-busy instances keeps tail latency lower under peak load.

13.4 Sharding the Ledger and Read Replicas

Sharding by account_id keeps a single customer’s full transaction history and balance state on one shard, so every balance calculation is local and fast, while total load is spread evenly across the cluster as the customer base grows.

13.5 Algorithms & Data Structures Worth Knowing Here

• Vector clocks / version vectors: In systems with multiple writers or complex multi-region topologies, version vectors generalise the simple “ledger version number” used in this design to detect concurrent, potentially conflicting updates across nodes — worth mentioning in an interview as the natural next step if the design grew to support multi-region active-active writes rather than single-region active-passive.
• Merkle trees for reconciliation at scale: Rather than the Reconciliation Service recomputing every single account’s balance from scratch on every pass, a Merkle-tree-based summary (hashing balances in batches, then hashing those hashes together) lets it cheaply detect “something changed in this range of accounts” and drill down only where needed — a well-known technique from distributed databases like Cassandra and DynamoDB for efficient anti-entropy repair.
• Consistent hashing: Used for distributing accounts across cache shards and Kafka partitions, so that adding or removing a shard only reshuffles a small fraction of accounts rather than nearly all of them.
• Raft consensus algorithm: Powers the Failover Coordinator’s leader election — worth understanding at a conceptual level (a leader is elected by majority vote among nodes, and only a leader with an up-to-date log can win) since it’s directly why split-brain is prevented rather than just “handled.”

13.6 Testing Strategy

Beyond the failover chaos drills already described, the system relies on deterministic replay tests that feed a fixed, versioned sequence of transactions and concurrent balance-check requests through a test environment, asserting the exact expected balance at each point — including at instants deliberately chosen to fall during a simulated failover. Contract tests verify that the Balance Query Service, Staleness Guard, and Reconciliation Service all agree on the meaning of a “ledger version,” since a subtle mismatch in how one service interprets versioning could silently reintroduce the exact staleness risk the whole design exists to prevent. Load tests replay realistic payday-scale traffic patterns against a staging cluster while a failover is deliberately triggered mid-test, verifying both the latency and correctness targets hold simultaneously under real concurrent load rather than only in isolated unit tests.

Headers:
  Authorization: Bearer <customer_session_token>

Response (200 OK - Real-time):
{
  "account_id": "acc_9f21ac",
  "available_balance": 1284.32,
  "posted_balance": 1310.00,
  "currency": "USD",
  "as_of": "2026-08-04T09:12:41.203Z",
  "freshness": "REALTIME"
}

Response (200 OK - Degraded, e.g. during failover):
{
  "account_id": "acc_9f21ac",
  "available_balance": 1284.32,
  "posted_balance": 1310.00,
  "currency": "USD",
  "as_of": "2026-08-04T09:12:33.900Z",
  "freshness": "DEGRADED",
  "message": "Showing balance as of 09:12:33. Refreshing automatically."
}

Each internal microservice exposes a narrow, single-purpose API (Balance Query, Staleness Guard, Transaction Posting, Failover Coordinator) rather than one monolithic service, letting the team owning replication and failover logic ship changes independently from the team owning the customer-facing balance API, while both share the same underlying correctness contract expressed through the ledger version number.

15.1 Patterns Used

15.2 Anti-patterns to Avoid

15.3 The Saga Pattern for Balance-Affecting Reversals

Not every balance-affecting event is a simple, single-service write — a disputed transaction reversal, for example, may need to touch the ledger, notify the customer, and update a linked card-network dispute system. These multi-step flows use the Saga pattern: each step is a local transaction with a defined compensating action, coordinated asynchronously through the same CDC/event pipeline already used for cache updates, so a failure partway through never leaves the ledger and its downstream consumers inconsistent with each other.

• Always tie cache freshness to a real, verifiable version number tied to the write-ahead log — never to a fixed timer alone.
• Treat “fail closed” as a design requirement from day one, not a patch added after an incident — retrofitting it is much harder.
• Run regular, automated failover drills in a realistic environment; a failover mechanism that has never actually been triggered under load should not be trusted.
• Keep the write path’s synchronous replication scope as small and fast as possible — one standby acknowledgement, not a large quorum, to keep write latency acceptable without sacrificing the core durability guarantee.
• Make degraded-mode responses a designed, tested part of the API contract, not an afterthought error case.

17.1 A Worked Example: Payday Traffic Meets a Mid-Day Failover

Consider the last Friday of the month, when balance-check traffic surges from a 25,000 req/min baseline to over 1,100,000 req/min as millions of customers check their balance right after being paid. In the middle of this peak, a hardware fault takes down the primary database in one region. Here’s how the system responds: the CDN and API Gateway see no change at all, since neither is affected by the database event. The Balance Query Service’s cache continues serving the vast majority of requests instantly, since most recently-active accounts’ balances are already cached with recent versions. For accounts whose cache entries are near the staleness boundary right as the failover begins, the Staleness Guard detects the failover state from the Failover Coordinator and serves a clearly-labelled degraded response instead of guessing. Within roughly 10–15 seconds, the Failover Coordinator completes leader election and fencing, the new primary resumes accepting writes, the CDC pipeline catches up in milliseconds, and the Staleness Guard resumes trusting fresh reads automatically — with zero manual intervention and, critically, zero incorrect balances shown at any point during the event.

17.2 A Worked Example: A Botched Deployment Triggers an Unplanned Failover

Not every failover is caused by hardware failure — a surprisingly common trigger in real production systems is a routine deployment going wrong, for example a configuration change that causes the primary database process to restart unexpectedly. From this system’s perspective, the cause is almost irrelevant: the Failover Coordinator doesn’t need to know why heartbeats stopped, only that they did. This is a deliberately valuable property of the design — by building failover handling around a generic “the primary is unreachable” signal rather than a list of specific failure causes, the same protection applies equally to a disk failure, a botched deploy, a network blip, or an operator mistake, without needing separate handling logic for each. This is also why deployment changes to the database tier go through the controlled runbook mentioned in Section 12 — not because failover would fail to protect the system, but because triggering an unplanned failover during a peak traffic window, even one handled correctly, still means a slice of customers briefly see degraded-mode responses, which is worth avoiding through careful change management even though the system tolerates it safely when it does happen.

17.3 Balancing Customer Experience Against Strict Correctness

It’s worth acknowledging directly that this design accepts a real, measurable cost — a small number of customers will occasionally see a “balance as of a few seconds ago, refreshing” message rather than an instantaneous number, specifically during the rare windows when a failover is in progress. Product and engineering teams at real digital banks generally treat this as a deliberate, acceptable trade-off once it’s explained clearly: a customer who understands their balance is refreshing after an infrastructure hiccup, and sees the correct number a few seconds later, has a far better experience and far more trust in the bank than a customer who is shown a confidently wrong number that later turns out to have let them overdraw. Making this trade-off explicit — both in the architecture and in how it’s communicated to the customer through the app’s UI — is itself a form of good system design, since a purely technical solution that isn’t paired with the right user-facing messaging would still leave customers confused even when the underlying engineering is working exactly as intended.

The most common questions engineers ask when first meeting this design — each answered in the same spirit as an interview response.

At a million-plus requests per minute, routing every read to the single primary would overwhelm it and directly compete with write traffic for the same resources — and it wouldn’t even help during a failover, since the primary is exactly the component that’s unavailable at that moment. The versioned cache-and-replica approach gives near-primary freshness for the vast majority of traffic while remaining scalable and, crucially, failure-aware.

Most large-scale caches accept unbounded (or loosely bounded) staleness because the cost of a stale read is low. Here, staleness is explicitly bounded, measured, and enforced per-request via the Staleness Guard, and the system is designed to refuse to serve data that falls outside that bound rather than silently accepting looser consistency for the sake of higher cache hit rates.

A single-node primary failover is handled entirely within a region by the Failover Coordinator in seconds, with zero committed-transaction loss thanks to synchronous replication. A full regional disaster is a much rarer, higher-severity event requiring cross-region failover, with a small, bounded, and explicitly acknowledged possibility of losing the last few seconds of transactions due to the necessarily asynchronous nature of cross-region replication — the two scenarios intentionally have different RPO/RTO targets and different customer-facing behaviour.

Through scheduled chaos-engineering drills that kill the primary under realistic concurrent load and assert three things automatically: zero lost committed transactions (verified against a pre-recorded expected ledger state), zero balance-check responses that overstate available funds, and a measured recovery time within the target RTO — treating any violation of the first two as a release-blocking failure, not just a warning.

The core failover-safety mechanisms (synchronous replication, fencing, versioned caching, fail-closed reads) stay identical, since they protect the integrity of any ledger-derived number, not specifically a debit balance. What changes is the direction of the “safe” default: for a credit account, understating the amount owed during a degraded window is the riskier direction (it could let a customer believe they have more available credit than they do), so the degraded-mode response would round in the more conservative direction for that account type — a good example of how the same architecture supports different business-specific safety policies on top of the same correctness guarantees.

It’s tempting given how much of the read path already depends on Redis, but the primary ledger specifically needs mature, battle-tested ACID transaction guarantees and durable, disk-backed write-ahead logging that survives a full process crash without data loss — properties an in-memory-first store either lacks or bolts on less maturely than a purpose-built relational database. Using Redis as an accelerating cache layered on top of a durable primary, as this design does, captures nearly all of the latency benefit while keeping the actual money-of-record on infrastructure built for exactly that responsibility.

The reconciliation drift count, without question — it’s the one metric that directly measures the property the entire system exists to guarantee (a correct balance), rather than a proxy for it like latency or cache hit rate. A system with perfect latency numbers and a nonzero drift count is failing at its actual job; a system with occasionally elevated latency during a failover but zero drift is succeeding at it.

The core principles are cloud-agnostic — synchronous replication, consensus-based failover, and versioned caching are all implementable on any major cloud provider’s managed database and caching offerings, or on self-managed infrastructure. A genuine multi-cloud active-active setup would add meaningful complexity around cross-provider network latency for the synchronous replication link specifically, since that link is the most latency-sensitive part of the whole design; most real-world digital banks instead run multi-region within a single cloud provider and treat true multi-cloud primarily as a disaster-recovery posture rather than an active-active one, for exactly this reason.

Designing a real-time balance-check system for a digital bank is fundamentally about accepting that failures — a database primary crashing, a network blip, a botched deploy — are not edge cases to handle apologetically, but a normal operating condition the architecture must be built around from day one. The winning combination is: synchronous replication so no committed transaction can ever be lost, consensus-based failover with fencing tokens so two nodes can never both believe they’re in charge, a versioned cache and staleness guard so freshness is a measured fact rather than a hopeful assumption, and a “fail closed” philosophy so that when correctness genuinely cannot be guaranteed for a few seconds, the system says so honestly instead of guessing.

If you take one idea into your next system design interview from this tutorial, let it be this: identify the one property your system can never violate — here, “never show an incorrect balance” — design every failure path around protecting that property first, and only then optimise for speed and scale on top of it. Applied consistently, that instinct is what lets a system serve a customer’s balance in milliseconds on a normal Tuesday and still tell them the truth, seconds after a database catches fire underneath it. That is, in the end, the whole job — and every architectural decision in this document, from the choice of synchronous replication down to the wording of a degraded-mode API response, exists in service of that one job.