Every loan in existence — a mortgage, a personal loan, a buy-now-pay-later installment, a credit line, a peer-to-peer loan — has one thing in common: money grows (or the debt grows) every single day it sits unpaid. That growth is called interest accrual. Somewhere, every single day, a piece of software wakes up, looks at millions of loan accounts, and asks a deceptively simple question for each one: “how much interest did this loan earn today?”

That question sounds trivial until you consider the scale and the correctness bar. A bank the size of a mid-tier regional lender might carry 5 to 20 million active loans. A large fintech lender or credit card issuer can carry hundreds of millions of open balances. Every one of those loans can have a different interest rate, a different day-count convention, a different compounding frequency, a different currency, and a different regulatory jurisdiction. And unlike a “like” button on a social app, an interest calculation that’s off by even a fraction of a cent — multiplied across millions of accounts, over years — becomes a regulatory incident, a class-action lawsuit, or a multi-million dollar reconciliation nightmare.

Historically, interest accrual was a nightly, single-threaded COBOL batch job running on a mainframe, processing loans sequentially against a single ledger tape. That approach worked when banks had thousands of loans and an eight-hour overnight window. It does not work when a modern lending platform must onboard millions of new loans through instant digital origination, recalculate accruals intraday when a customer makes a payment or a rate changes, expose real-time “current balance” APIs to millions of concurrent mobile app users, and do all of this while remaining auditable to regulators who can ask “show me exactly how you calculated interest on loan X, on date Y” five years later.

This tutorial designs that system from first principles: the domain concepts that make lending math tricky, the architecture that lets a batch-and-real-time hybrid system scale horizontally, and the engineering discipline (idempotency, event sourcing, precise arithmetic, sharding) required to make the numbers always add up — even under a traffic profile that spikes to a million requests per minute during peak hours like month-end statement generation or a marketing-driven surge in loan applications.

By the end of this tutorial, you’ll be able to whiteboard the full system end to end: draw the component diagram with every box correctly labeled (API Gateway, load balancer, orchestrator, worker fleet, sharded stores, caches), explain precisely why each piece exists and what breaks without it, and defend the design against the kinds of follow-up questions a senior interviewer throws at exactly this class of problem — “what if two events race,” “what if a worker dies mid-batch,” “how do you prove to a regulator this number is correct.” Those follow-ups are woven throughout as dedicated interviewer callouts, so you can treat this as both a design reference and interview preparation in one pass, whether you’re preparing for a staff-level system design round or actually building this system for a real lending platform.

Let’s define the problem precisely before drawing a single box.

2.1 Functional requirements

• Daily accrual: For every active loan, compute the interest that accrued for “today” based on the loan’s outstanding principal, its annual interest rate, and its day-count convention (e.g., Actual/365, 30/360, Actual/360).
• Heterogeneous terms: Support fixed-rate, variable-rate, tiered-rate, and promotional-rate (e.g., 0% APR for 6 months) loans, each with its own compounding schedule (daily, monthly, or none until maturity).
• Mid-cycle events: Correctly re-accrue when a payment posts, a rate changes, a loan is paid off early, or a loan goes delinquent (and possibly stops accruing, or starts accruing penalty interest) — all mid-day, not just at the nightly batch boundary.
• Real-time balance API: Expose a “what do I owe today” endpoint to mobile/web clients that reflects accrued-but-not-yet-posted interest, with low latency, at massive read concurrency.
• Immutable audit trail: Every cent of interest ever calculated must be traceable — which formula, which inputs, which code version, which timestamp — for 7+ years, to satisfy regulators (e.g., Truth in Lending Act, Regulation Z in the US, or equivalent regimes elsewhere).
• Idempotent, exactly-once posting: A retried or replayed accrual job must never double-post interest to a loan’s ledger.

2.2 Non-functional requirements

2.3 Why “just scale the database” isn’t a real answer

A natural first instinct is to throw a bigger database at the problem — more CPU, more RAM, a faster disk. This works until it very suddenly doesn’t. A single database instance, no matter how large, has a ceiling on write IOPS and connection concurrency. At 200 million loans needing a write each night within a 2-hour window, the required sustained write rate exceeds what even a very large single instance can sustain reliably, and it creates a single point of failure for the entire lending business’s most critical nightly process. Vertical scaling buys time; it doesn’t remove the ceiling. The architecture in this tutorial is built around horizontal scaling from the outset specifically to avoid hitting that wall later, at the worst possible time — during a growth spurt when the business can least afford an emergency re-architecture.

2.4 Regulatory weight behind the numbers

Interest accrual isn’t just a technical calculation — in most jurisdictions it’s directly governed by consumer protection law (for example, Regulation Z / Truth in Lending Act disclosures in the US, or equivalent regimes elsewhere), meaning the method of calculation, not just the final number, must match what was disclosed to the borrower at origination. This is why the system must persist the day-count convention, rate, and formula version used for every single calculation rather than just the resulting dollar amount — a regulator or auditor can ask to see the exact method five years later, and “we don’t have that level of detail anymore” is not an acceptable answer.

Before touching architecture, we need shared vocabulary. Think of these as the “physics” of the lending world — the rules the system must obey no matter how it’s built.

3.1 Principal, rate, and accrual

What: Principal is the amount of money still owed. The interest rate (Annual Percentage Rate, or APR) is the yearly cost of borrowing that money, expressed as a percentage. Accrual is the daily “drip” of interest that builds up on top of the principal.

Analogy: Imagine filling a bathtub with a slowly dripping faucet. The tub is the loan balance. Every day, a few more drops (interest) fall in, even if nobody touches the tub. The daily accrual job is the “drop counter” — it doesn’t fill the tub in one go, it counts drops one day at a time.

Formula (simple daily interest):

Daily Interest = Outstanding Principal × (Annual Rate / Day Count Basis)

3.2 Day-count conventions

What: A rule for how many days are assumed to be in a month or year when computing interest. This sounds like trivia, but it’s the single most common source of “off by a few cents” bugs in lending software.

Practical example: A $100,000 loan at 6% APR accrues $16.44/day under Actual/365, but $16.67/day under Actual/360 — a small daily gap that compounds into thousands of dollars of difference over a loan’s lifetime. The system must store the convention as loan-level metadata and never hardcode it.

3.3 Compounding

What: Whether accrued-but-unpaid interest itself starts earning interest. Simple interest never compounds; the principal alone accrues. Compound interest (common in credit cards) adds unpaid interest into the balance that then accrues further interest.

Analogy: Simple interest is like a savings jar where only the original coins earn interest. Compound interest is a jar where yesterday’s earned interest coins get added to the jar and start earning interest too.

3.4 Ledger & immutability

What: A ledger is an append-only, chronological record of every financial event on an account — every accrual, payment, fee, and adjustment. Unlike a typical CRUD table, ledger rows are never updated or deleted; corrections are made by inserting a new offsetting entry.

Why it matters: Immutability is what makes a financial system auditable. If a bug caused an over-accrual, you don’t silently “fix” the old row — you write a reversing entry, exactly like double-entry bookkeeping in accounting has done for 500 years.

3.5 Amortization vs. simple daily accrual

What: Amortization is a pre-computed schedule that splits each fixed payment into a principal portion and an interest portion over the life of a loan (common in mortgages and auto loans). Daily simple accrual, by contrast, doesn’t rely on a fixed schedule — it computes interest fresh every day off the actual outstanding principal, which is what makes it correctly handle early payments, extra principal payments, and payoffs without needing to regenerate an entire schedule.

Why this system uses daily accrual, not a static amortization table: A static schedule assumes the borrower pays exactly on time, every time. Real portfolios don’t behave that way — partial payments, late payments, and payoffs are the norm at scale, so the amortization schedule becomes a display convenience for the customer, while the daily accrual engine remains the actual source of truth for how much interest is owed.

3.6 Grace periods, delinquency, and charge-off

What: A grace period is a window (e.g., 15 days after a due date) during which a late payment doesn’t yet trigger penalty interest or fees. Delinquency status kicks in once a loan is meaningfully overdue (e.g., 30/60/90 days past due), and often changes the accrual rule itself — some products stop accruing standard interest and start accruing a higher penalty rate; others pause accrual entirely pending a regulatory or accounting requirement. Charge-off is the point where a lender writes the loan off as a loss for accounting purposes; the system typically stops accrual entirely at that point, even though collections activity may continue.

Why it matters for architecture: Accrual isn’t one universal formula — it’s a small state machine per loan (current → grace → delinquent → charged-off), and the Interest Rate and Terms Service must expose not just a rate, but the current accrual policy for that loan’s status, which the worker consults before applying any formula.

3.7 Prepayment and payoff

What: A payoff is when a borrower pays the full remaining balance early. From that moment forward, principal is zero and accrual must stop precisely at the payoff timestamp — not at midnight, not at the next batch run. Some loans also apply a prepayment penalty, itself a special one-time interest-like charge computed by a different formula.

Analogy: Think of the daily accrual job as a faucet that drips into the tub every day. A payoff event isn’t just “empty the tub” — it’s “turn the faucet off at 2:47pm today,” and the system has to know to only count drops up to that exact moment, not the whole day.

3.8 Consistency model: why eventual consistency is acceptable here

What: The CAP theorem says a distributed system facing a network partition must choose between consistency and availability. This system deliberately chooses different points on that spectrum for different data: the ledger write path favors strong consistency within a shard (a single loan’s entries must never be lost or duplicated), while the balance read path favors availability and low latency, accepting a small, bounded eventual-consistency window (typically under one second) between a ledger write and the read-side cache reflecting it.

Why this split is safe: The kind of “wrong answer” a customer-facing balance API can tolerate for a few hundred milliseconds (a slightly stale but never fabricated number) is fundamentally different from what the ledger must never tolerate (a lost or duplicated financial entry). Separating these concerns is precisely why CQRS is used — a single system can’t uniformly optimize for both, but two cooperating models can.

The system splits cleanly into two operational modes that share the same domain logic but very different traffic shapes:

• Batch Accrual Pipeline — a nightly (and optionally intraday) distributed job that walks every active loan and posts a daily interest entry.
• Real-Time Event & Read Path — handles payments, rate changes, payoffs, and serves millions of “what do I owe” reads per minute without ever touching the batch pipeline’s write path directly.

Below is the full end-to-end architecture. Every box names the concrete component responsible for it — this is deliberately drawn the way you’d whiteboard it in an interview.

4.1 Component responsibilities

4.2 Why the Kafka topic is partitioned by shard key, not by loan ID directly

A natural instinct is to partition the accrual topic by individual loan_id for maximum parallelism. In practice, the system partitions by the same shard key used for the database (e.g., loan_id % 4096, then grouped into a smaller number of Kafka partitions) so that a single consumer processes a contiguous, database-co-located batch of loans per message rather than one tiny message per loan. This dramatically reduces message volume (thousands of shard-trigger messages instead of hundreds of millions of per-loan messages), lets each worker do efficient batched database reads and writes against loans it already owns a connection pool for, and keeps the mapping between “which worker touches which data” stable and predictable — valuable both for debugging and for reasoning about concurrency.

4.3 Why a separate idempotency store instead of relying solely on the database’s unique constraint

The database’s unique constraint on (loan_id, idempotency_key) is the ultimate source of truth and the real safety net — it’s what actually prevents a duplicate row from ever being committed. But hitting the database to discover “was this already processed” for every single loan, every single day, adds unnecessary load to the most contended resource in the system. The Redis-backed idempotency store acts as a fast-path check: if a key is already marked complete, the worker skips the expensive database round trip entirely. If the fast-path check is wrong (e.g., the cache entry expired early or was never written due to a crash), the database constraint still catches the duplicate at insert time — so correctness never depends on the cache being right, only on it being a helpful optimization.

Let’s zoom into how a single day’s accrual actually runs across the fleet.

5.1 Job fan-out and sharding

At the scheduled trigger time (say, 00:05 UTC), the Distributed Scheduler doesn’t process 200 million loans in one job. It fans the work out: the loan population is partitioned into shards (e.g., by loan_id % 4096), and one lightweight “accrual trigger” message is published per shard to Kafka. Each message says, in effect, “go accrue interest for shard 137, for business date 2026-08-04.” This turns one giant sequential job into thousands of small, independent, parallelizable units of work — the same principle as MapReduce.

5.2 Worker execution

Each worker in the auto-scaled consumer pool picks up a shard message, queries the Loan Master DB (or a warmed cache) for all active loans in that shard, and for each loan:

5.3 Idempotency: the non-negotiable guarantee

Kafka consumers can redeliver messages (at-least-once delivery is the norm). A worker can crash mid-batch and be restarted. Without idempotency, a redelivered message would double-post interest — effectively fabricating money. The fix: every accrual write uses a deterministic idempotency key, and the ledger insert is guarded by a unique constraint (or a conditional write) on that key, so a duplicate attempt is a safe no-op rather than a duplicate row.

public class AccrualPoster {

    private final LedgerRepository ledgerRepo;
    private final IdempotencyKeyStore idempotencyStore;

    public AccrualResult postDailyAccrual(LoanSnapshot loan, LocalDate businessDate) {
        String idemKey = buildIdempotencyKey(loan.getLoanId(), businessDate);

        // Fast-path check against Redis/DynamoDB before touching the DB
        if (idempotencyStore.exists(idemKey)) {
            return AccrualResult.alreadyProcessed(idemKey);
        }

        BigDecimal dailyInterest = InterestCalculator.calculateDaily(
                loan.getOutstandingPrincipal(),
                loan.getAnnualRate(),
                loan.getDayCountConvention(),
                businessDate
        );

        LedgerEntry entry = LedgerEntry.builder()
                .loanId(loan.getLoanId())
                .businessDate(businessDate)
                .principalUsed(loan.getOutstandingPrincipal())
                .rateUsed(loan.getAnnualRate())
                .interestAmount(dailyInterest)
                .idempotencyKey(idemKey)
                .formulaVersion("v3-actual365")
                .build();

        try {
            // Unique constraint on (loan_id, business_date, formula_version)
            // makes this insert safe to retry
            ledgerRepo.insertIfAbsent(entry);
            idempotencyStore.markComplete(idemKey, Duration.ofDays(3));
            return AccrualResult.success(entry);
        } catch (DuplicateKeyException dup) {
            // Another worker/retry already committed this exact entry
            return AccrualResult.alreadyProcessed(idemKey);
        }
    }

    private String buildIdempotencyKey(String loanId, LocalDate date) {
        return loanId + ":" + date + ":daily-accrual";
    }
}

public class InterestCalculator {

    private static final int SCALE = 10;
    private static final RoundingMode ROUNDING = RoundingMode.HALF_EVEN;

    public static BigDecimal calculateDaily(BigDecimal principal,
                                              BigDecimal annualRatePercent,
                                              DayCountConvention convention,
                                              LocalDate businessDate) {
        BigDecimal rateDecimal = annualRatePercent.divide(BigDecimal.valueOf(100), SCALE, ROUNDING);
        BigDecimal dayCountBasis = convention.basisFor(businessDate); // 360, 365, or 366

        BigDecimal dailyRate = rateDecimal.divide(dayCountBasis, SCALE, ROUNDING);
        BigDecimal dailyInterest = principal.multiply(dailyRate);

        // Round to the currency's minor unit only at the final posting step,
        // never during intermediate compounding math - avoids drift
        return dailyInterest.setScale(2, ROUNDING);
    }
}

5.4 Concurrency control: batch accrual vs. concurrent payment posting

A subtle but critical race condition: what happens if a payment posts for a loan at the exact moment the nightly batch is calculating that loan’s accrual? Two writers touching the same loan’s principal at once is exactly the kind of bug that causes real financial loss if handled naively.

The system resolves this with optimistic concurrency control at the loan level. Each loan’s mutable “current state” row (principal, last-accrued date) carries a version number. Both the batch worker and the real-time payment handler read the version, compute their update, and write back conditionally on that version being unchanged (UPDATE ... WHERE loan_id = ? AND version = ?). If the conditional update affects zero rows, the writer knows it lost the race, re-reads the fresh state, and retries its calculation against the new principal. This avoids taking a heavyweight distributed lock per loan while still guaranteeing no lost updates.

public class LoanStateUpdater {

    private final LoanStateRepository repo;
    private static final int MAX_RETRIES = 5;

    public void applyAccrualWithOptimisticLock(String loanId, BigDecimal accruedInterest) {
        int attempt = 0;
        while (attempt < MAX_RETRIES) {
            LoanState current = repo.fetchCurrentState(loanId); // includes version
            LoanState updated = current.withAccruedInterestApplied(accruedInterest);

            int rowsAffected = repo.updateIfVersionMatches(
                    loanId,
                    updated,
                    current.getVersion()
            );

            if (rowsAffected == 1) {
                return; // success
            }

            attempt++;
            sleepWithJitter(attempt); // small backoff before re-reading and retrying
        }
        throw new OptimisticLockExhaustedException(loanId, MAX_RETRIES);
    }

    private void sleepWithJitter(int attempt) {
        try {
            long baseMs = (long) Math.pow(2, attempt) * 10;
            long jitter = ThreadLocalRandom.current().nextLong(0, baseMs);
            Thread.sleep(baseMs + jitter);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

5.5 Batch fetching for throughput

Fetching one loan at a time from the database per worker iteration would drown the database in round trips at this scale. Instead, each worker pulls loans in bulk pages (e.g., 500–1000 rows per query) within its assigned shard, processes the page in memory, and writes ledger entries in a single batched insert rather than one insert per loan — reducing both network round trips and transaction overhead by two to three orders of magnitude compared to row-by-row processing.

public class ShardAccrualProcessor {

    private static final int PAGE_SIZE = 750;

    public void processShard(int shardId, LocalDate businessDate) {
        String cursor = null;
        List<LedgerEntry> batchToInsert = new ArrayList<>(PAGE_SIZE);

        do {
            Page<LoanSnapshot> page = loanRepo.fetchActiveLoansPage(shardId, cursor, PAGE_SIZE);
            for (LoanSnapshot loan : page.getItems()) {
                BigDecimal interest = InterestCalculator.calculateDaily(
                        loan.getOutstandingPrincipal(),
                        loan.getAnnualRate(),
                        loan.getDayCountConvention(),
                        businessDate
                );
                batchToInsert.add(buildLedgerEntry(loan, businessDate, interest));
            }
            ledgerRepo.bulkInsertIfAbsent(batchToInsert); // single round trip per page
            batchToInsert.clear();
            cursor = page.getNextCursor();
        } while (cursor != null);
    }
}

The sequence diagram below traces one loan’s daily accrual from trigger to a client seeing the updated balance.

6.1 Mid-cycle events

Payments, payoffs, and rate changes don’t wait for the nightly batch. They flow through a separate real-time event path: a payment event lands in a Payment Ingestion Service, which publishes an event to the same Kafka ecosystem (a different topic), consumed by the Ledger Service to post a payment entry and trigger an immediate partial re-accrual for that single loan if the business rules require it (e.g., “interest stops accruing on paid-off principal from the payment timestamp forward”).

7.1 Sharding strategy

With 200M+ loans, a single database instance is out of the question. The Loan Master DB and Ledger Store are both horizontally sharded by loan_id hash, typically into a few thousand logical shards mapped onto a smaller number of physical database clusters (e.g., 4096 logical shards across 64 physical Postgres clusters, each holding 64 logical shards). This gives room to re-balance physical placement later without changing the hashing scheme clients depend on.

7.2 Caching layers

7.3 Ledger storage model

The ledger uses an event-sourced, append-only table partitioned by business date (for efficient archival of old partitions to cold storage) and indexed by loan_id for fast per-loan history reads. Every row is immutable; corrections are new rows referencing the original entry’s ID.

CREATE TABLE ledger_entries (
    entry_id            BIGINT GENERATED ALWAYS AS IDENTITY,
    loan_id              BIGINT NOT NULL,
    business_date        DATE NOT NULL,
    entry_type           VARCHAR(20) NOT NULL,  -- ACCRUAL, PAYMENT, ADJUSTMENT, REVERSAL
    principal_used        NUMERIC(18,2),
    rate_used             NUMERIC(9,6),
    amount                NUMERIC(18,2) NOT NULL,
    formula_version       VARCHAR(20) NOT NULL,
    idempotency_key       VARCHAR(100) NOT NULL,
    reverses_entry_id      BIGINT,
    created_at            TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (business_date, entry_id),
    UNIQUE (loan_id, idempotency_key)
) PARTITION BY RANGE (business_date);

8.1 Key endpoints

GET  /v1/loans/{loanId}/balance          -> real-time current balance (cache-backed)
GET  /v1/loans/{loanId}/ledger?from&to   -> paginated ledger history for audits
POST /v1/loans/{loanId}/payments          -> ingest a payment event
POST /v1/loans/{loanId}/rate-change       -> apply a new rate effective a given date
GET  /v1/accrual-runs/{date}/status       -> batch run health/progress for ops dashboards

8.2 Microservice boundaries

Each service owns its own data and communicates via well-defined APIs and asynchronous events — never by reaching into another service’s database. The Accrual Orchestrator only knows shard boundaries and job status; it delegates actual calculation to workers. The Balance Query Service never reads the ledger directly at request time; it only reads the projection cache, keeping read latency flat regardless of ledger size.

8.3 Example request / response

GET /v1/loans/LN-88213/balance
Authorization: Bearer <jwt>

200 OK
{
  "loanId": "LN-88213",
  "asOf": "2026-08-04T09:12:03Z",
  "outstandingPrincipal": "12450.32",
  "accruedInterestUnposted": "4.18",
  "currentPayoffAmount": "12454.50",
  "dayCountConvention": "ACTUAL_365",
  "annualRate": "7.250",
  "status": "CURRENT",
  "lastAccrualDate": "2026-08-03",
  "cacheAge": "312ms"
}

Note the cacheAge field — a deliberate transparency choice. Since the balance API is backed by an eventually-consistent projection, the response tells the caller exactly how fresh the number is, rather than silently pretending it’s a live, strongly-consistent read. This matters for both customer trust and for internal services that may need to decide whether to force a synchronous re-read for a high-stakes operation like a payoff quote.

8.4 Multi-tenancy and portfolio isolation

A lending platform rarely serves a single homogeneous loan book — it typically serves multiple portfolios (different loan products, different originating partners, sometimes different regulatory jurisdictions), each of which can define its own day-count convention, compounding rule, and accrual policy. The Interest Rate and Terms Service models this as a portfolio configuration layer sitting above individual loan records: every loan references a portfolio_id, and the accrual worker resolves “which formula and which day-count convention applies” by looking up the portfolio’s policy, not by hardcoding rules per loan. This keeps onboarding a new lending product or partner a configuration change rather than a code change, while still letting each shard mix loans from many portfolios without any cross-contamination of rules.

Stepping back, nearly every trade-off above follows the same shape: the system gives up a small amount of simplicity or a small, bounded consistency window in exchange for horizontal scalability and operational resilience. That’s a deliberate, recurring theme rather than a coincidence — it’s the same trade-off financial infrastructure has made for decades in different technological clothing, from paper ledgers with reversing entries, to mainframe batch tapes, to today’s event-sourced, sharded, cloud-native pipelines.

A million requests per minute is roughly 16,700 requests per second sustained, with realistic bursts well above that during month-end statement generation or a marketing campaign. Here’s how each tier absorbs that load.

10.1 Edge and gateway tier

The CDN absorbs and caches cacheable GET responses (e.g., static rate disclosures, marketing content) so they never reach the origin. The WAF and Load Balancer tier scales horizontally behind an auto-scaling group, and the API Gateway enforces per-client and global rate limits using a token-bucket algorithm backed by a distributed counter (Redis), so one noisy client can’t starve others.

10.2 Read path scalability

Because balance reads hit a Redis-backed projection cache rather than the sharded transactional database, the read path scales by adding Redis replicas / shards, which is far cheaper and faster than scaling relational database read replicas for the same throughput. A well-provisioned Redis cluster can serve well over a million reads per second per cluster, giving enormous headroom above the target load.

10.3 Write / batch path scalability

The batch accrual pipeline scales by increasing Kafka partition count and worker pool size — each partition is consumed independently, so doubling partitions and workers roughly doubles throughput (up to the database’s write capacity limits). Database write throughput is protected by sharding: 4096 logical shards mean each physical cluster only needs to sustain a fraction of total write volume.

10.4 Back-of-envelope capacity math

Loan book:            200,000,000 active loans
Batch window target:  2 hours = 7,200 seconds
Required throughput:  200,000,000 / 7,200 ≈ 27,778 loan-accruals/sec

If each worker processes 500 loans/sec (DB write + calc):
  Workers needed ≈ 27,778 / 500 ≈ 56 workers minimum
  (Provision 2-3x headroom -> ~150 workers, auto-scaled)

Peak read load:       1,000,000 req/min ≈ 16,667 req/sec
  Redis cluster easily absorbs this with sub-ms reads;
  bottleneck shifts to network/API Gateway throughput,
  addressed via horizontal Gateway auto-scaling.

10.5 Cache hit ratio and its effect on database load

The Rate/Terms cache exists specifically to keep database load flat during the batch window. Consider the math: with 200 million loans and a 98% cache hit ratio on rate/terms lookups (reasonable, since a loan’s rate rarely changes day to day), only 4 million lookups per run fall through to the database rather than 200 million — a 50x reduction in read pressure on the Loan Master DB during the exact window it’s also absorbing heavy ledger writes. This is why cache warming (pre-populating the cache ahead of the nightly run, based on the previous day’s active loan set) is a standard operational step before the batch trigger fires.

10.6 Horizontal scaling levers, summarized

10.7 Handling traffic spikes beyond a million requests per minute

Predictable spikes (month-end statements, a scheduled marketing push) are handled with scheduled pre-scaling — the API Gateway and Balance Query Service fleets scale up ahead of the known event rather than reactively. Unpredictable spikes rely on fast auto-scaling triggers (scaling on request queue depth rather than just CPU, since CPU metrics lag behind an actual traffic surge) plus graceful degradation: if load exceeds provisioned capacity, the system prioritizes serving cached-but-slightly-stale balances over failing requests outright, and sheds lower-priority traffic (e.g., non-critical reporting endpoints) before shedding customer-facing balance checks.

11.1 Failure recovery flow

11.2 Multi-region and multi-AZ design

Each database shard’s primary lives in one availability zone with synchronous replicas in at least two others, enabling automatic failover within seconds. Kafka brokers and consumer groups span multiple AZs so a single AZ outage doesn’t halt the pipeline. For disaster recovery, ledger data is continuously replicated to a secondary region; a full region failover is a documented, drilled runbook rather than an aspiration.

11.3 Reconciliation as a safety net

Beyond preventing failures, the system runs a nightly reconciliation job comparing the sum of ledger entries against expected totals derived independently (e.g., from the loan master’s principal / rate history), flagging any mismatch above a cent threshold for human review before it reaches a customer statement.

11.4 Backup, RPO and RTO

Financial ledger data carries the strictest backup requirements in the platform. Continuous write-ahead-log (WAL) streaming to a secondary region keeps the Recovery Point Objective (RPO) — the maximum acceptable data loss — under a few seconds. Automated failover mechanisms target a Recovery Time Objective (RTO) — the maximum acceptable downtime — of a few minutes for a single-shard failure and under thirty minutes for a full regional failover, both figures validated through scheduled, unannounced failover drills rather than assumed from documentation.

11.5 Graceful degradation ladder

Rather than a binary “up or down,” the platform defines a ladder of degraded states it can fall back through under stress, always preserving core financial correctness even if convenience features drop first:

• AuthN / AuthZ: OAuth2 / OIDC at the API Gateway; service-to-service calls use mutual TLS and short-lived tokens (SPIFFE / SPIFFE-like identities).
• Encryption: TLS in transit everywhere; AES-256 at rest for the ledger and loan master databases; field-level encryption for PII (SSNs, account numbers).
• Least privilege: Accrual workers have write access only to the ledger tables they need; no service has blanket database admin rights.
• Immutable audit logging: All writes to financial tables are logged to a WORM (write-once-read-many) store, satisfying regulatory tamper-evidence requirements.
• Rate limiting and abuse prevention: Per-client quotas at the gateway prevent a single compromised or buggy integration from overwhelming the platform.
• PCI / PII scoping: Payment card data, if handled, is isolated to a narrowly scoped, separately audited subsystem rather than spread across the general architecture.

12.1 Segregation of duties

Beyond technical access controls, the platform enforces segregation of duties at the process level: the engineer who can deploy a change to the interest calculation formula is never the same person who can approve that formula’s golden-file test results, and any manual ledger adjustment requires a second approver before it posts — mirroring the “maker-checker” controls that traditional banking back-offices have used for decades, now encoded into the deployment and adjustment workflows themselves rather than left as a manual policy.

12.2 Data minimization in logs and traces

Distributed traces and structured logs are invaluable for debugging, but a stack trace or log line that accidentally includes a full account number or a customer’s SSN becomes a compliance liability the moment it’s written to a log aggregator with broader access than the production database itself. Logging middleware at the service boundary automatically redacts known sensitive fields before anything is emitted, and code review checklists explicitly call out “does this log statement include PII” as a required check for any change touching the loan or payment services.

12.3 Secrets and credential management

Database credentials, API keys, and signing keys are never embedded in code or configuration files checked into source control; they’re pulled at runtime from a managed secrets store (such as a cloud KMS-backed vault), rotated on a defined schedule, and scoped so that a compromised credential for one microservice cannot be used to access another service’s data store.

13.1 The batch run dashboard: what ops actually watches

During the nightly window, the single most important screen in the operations center is a live dashboard answering three questions at a glance: how many shards have completed, how many are in progress or stuck, and how many loans have landed in the dead letter queue. This is deliberately a business-facing dashboard, not just an infrastructure one — it’s built from Orchestrator status events rather than raw infrastructure metrics, so a non-engineer on the finance operations team can look at it at 2am and know whether statements will go out on time.

13.2 Alerting philosophy

Alerts are tiered by financial blast radius, not just technical severity. A single loan failing accrual and landing in the DLQ is a warning-level alert reviewed the next business day. A shard-wide failure that threatens the batch completion window is a page-immediately, wake-someone-up alert. A reconciliation mismatch of any non-zero amount is treated with the same urgency as a production outage, because unlike most software bugs, a financial miscalculation actively gets worse (compounds) the longer it goes unnoticed.

Stateless services (API Gateway, microservices, workers) run as containerized workloads on Kubernetes, scaled by Horizontal Pod Autoscalers keyed on Kafka consumer lag and CPU. The Accrual Worker Pool in particular scales aggressively before the nightly batch window (a scheduled pre-warm) and scales back down afterward to control cost. Blue-green or canary deployments are mandatory for the Ledger and Accrual services given the financial blast radius of a bad release — new formula versions are shipped behind a version flag (formula_version) so a regression is caught on a small canary shard before touching the full book. Infrastructure is defined as code (Terraform) so shard topology, Kafka topic configuration, and database cluster sizing are reproducible and reviewable.

14.1 Cost optimization

The Accrual Worker Pool is the single biggest driver of variable compute cost, since it’s provisioned for a burst (the nightly run) rather than steady-state load. Scheduled scale-down after the batch window, combined with spot / preemptible instances for the worker fleet (safe here because work is idempotent and replayable, so a preempted worker simply gets its partition reassigned), typically cuts compute cost for the batch tier substantially compared to running the peak fleet size around the clock. On the storage side, tiering ledger partitions from hot (fast SSD-backed database storage, recent ~18 months) to cold (object storage, older data retained for regulatory purposes) keeps the expensive, high-IOPS storage tier proportional to active data rather than the platform’s entire multi-year history.

14.2 Environment parity

Staging environments run against a synthetic loan population sized to a meaningful fraction of production (not just a handful of test loans), because many of the bugs that matter in a system like this — sharding hot spots, cache stampedes, batch-window overruns — only manifest at realistic scale. A staging environment with 50 loans will never catch a bug that only appears when 4,096 shards are running concurrently against contended database connections.

• Version your formulas. Store a formula_version on every ledger entry so historical entries remain explainable even after the calculation logic evolves.
• Never round in intermediate steps. Only round to currency minor units at the final posting point; round consistently with an agreed rounding mode (typically banker’s rounding).
• Treat business date, not wall-clock date, as authoritative. Loans operate on a “business day” calendar (accounting for weekends, holidays, and timezone of the servicing entity), which is often not the same as the server’s local date.
• Design for replay from day one. Being able to safely re-run any past day’s accrual (e.g., after a bug fix) is what turns an incident into a non-event.
• Common mistake: forgetting leap years in Actual/365 vs Actual/Actual conventions, silently under- or over-charging interest by a day’s worth annually.
• Common mistake: not handling loan status transitions (delinquency, charge-off, bankruptcy) as first-class accrual rules, leading to loans that keep accruing interest when regulation requires them to stop.
• Common mistake: under-provisioning the idempotency store’s TTL, so a very delayed retry no longer finds its key and double-posts.
• Best practice: make every downstream effect of an accrual (notifications, projections, analytics exports) derive from the ledger’s change stream rather than being triggered directly by the worker — this way, replaying history for a bug fix automatically replays every downstream effect correctly too, instead of requiring separate manual backfills.
• Best practice: build the reconciliation job before the system goes live with real customer money, not after the first incident — it should be considered a core deliverable of the project, not an afterthought bolted on post-launch.
• Common mistake: conflating “the batch job ran” with “the batch job succeeded.” A job that completes without errors but silently skipped a shard due to a coordination bug can look healthy on a naive dashboard while quietly missing millions of loans — always verify completion against an expected count, not just an absence of exceptions.
• Best practice: keep the accrual calculation function pure (no side effects, deterministic given its inputs) so it can be unit tested exhaustively and safely run in shadow mode without any risk of accidentally mutating state.

Financial calculation logic demands a testing discipline beyond typical CRUD services, because a subtle bug doesn’t just crash a request — it silently mis-states money across millions of accounts before anyone notices.

17.1 Golden file / reference testing

Interest formulas are tested against a large table of pre-verified expected outputs (“golden files”), independently computed by finance / actuarial teams using spreadsheet tools, covering every day-count convention, leap years, month-boundary edge cases, and compounding rule the platform supports. Any change to InterestCalculator must reproduce every golden value exactly before it can ship — this is the single highest-value test suite in the codebase.

17.2 Property-based and fuzz testing

Beyond fixed examples, property-based tests assert invariants that must hold for any valid input — for example, “daily interest is never negative for a positive principal and positive rate,” or “the sum of all daily accruals over exactly one year approximately equals principal × rate, within rounding tolerance.” These catch edge cases human-written examples miss.

17.3 Shadow runs before formula changes go live

Before a new formula_version is promoted from canary to the full loan book, it runs in “shadow mode” — computing results alongside the live formula without posting them — so any divergence is caught and investigated against real production data before it can affect a single customer’s balance.

17.4 Reconciliation testing in CI/CD

The nightly reconciliation job itself (comparing ledger totals against independently-derived expected totals) is exercised in staging against synthetic loan populations sized to mirror production, as part of the deployment pipeline — catching regressions in the reconciliation logic itself, not just the accrual logic.

It’s correct but doesn’t scale — summing years of daily ledger entries per request at a million-requests-per-minute load would overwhelm any database. The projection cache trades a small, bounded eventual-consistency window for massive read scalability.

Incomplete shards are tracked by the Orchestrator; the run auto-continues past the target window rather than truncating, while alerting on-call. The window is a target SLA for capacity planning, not a hard cutoff that drops loans.

The rate-change event carries an effective timestamp; the day’s accrual is split into two ledger entries — interest at the old rate up to the effective moment, and at the new rate afterward — preserving full auditability of exactly which rate applied when.

Kafka’s partitioned log gives natural sharding for parallel consumption, durable replay for recovery and reconciliation, and high sustained throughput — all directly needed here, versus a simpler queue that would need bolt-on replay and partitioning support.

Ledger table partitions older than an active window (e.g., 18 months) are archived to cheaper, immutable object storage (e.g., WORM-configured cold storage) while remaining queryable for audits, keeping the hot transactional database lean.

Golden-file reference testing against independently-verified expected values, property-based tests for universal invariants, and shadow-mode runs of any new formula version alongside the live one before cutover — all three layers described in the Testing section, because unit tests alone don’t catch domain-specific edge cases like leap years or convention mismatches.

Every event carries a business-effective timestamp, not just an arrival timestamp. The Ledger Service orders processing by effective time within a loan, and the accrual calculation always re-reads the loan’s current authoritative state (from the database, not a possibly-stale cache) at the moment of posting, so out-of-order delivery doesn’t produce out-of-order financial effects.

A global lock would serialize all writes across the entire loan book, destroying the horizontal scalability the whole design exists to achieve. Per-loan optimistic concurrency control keeps contention scoped to the rare case where two writers touch the same loan at the same moment — a tiny fraction of total traffic — rather than serializing unrelated loans against each other.

The portfolio and day-count convention are treated as time-versioned attributes of the loan, not fixed-forever values. A modification event writes a new effective-dated policy record, and every accrual calculation resolves “which convention applied on this specific business date” from that history rather than always reading the loan’s current setting — the same effective-dating principle used for rate changes, applied consistently to any policy attribute that can change over a loan’s life.

Fair challenge, and the honest answer is: partly, yes, if taken wholesale on day one. A smaller lender can start with a single well-indexed database, a simpler scheduled job instead of a full Kafka-based fan-out, and a single-region deployment — while still keeping the core correctness principles (immutable ledger, idempotent writes, BigDecimal arithmetic, versioned formulas) from the very first line of code. Those principles cost almost nothing at small scale and save an enormous, painful re-architecture later; the sharding, Kafka fan-out, and CQRS projection layer are the pieces that should be introduced incrementally as the loan book and traffic actually demand them, not built speculatively in advance of real need.

A daily interest accrual engine for millions of loans is fundamentally a distributed batch-processing problem wrapped around a financial-grade correctness core. The architecture succeeds by turning what began, decades ago, as a single mainframe conveyor belt into thousands of small, parallel, independently-verifiable units of work — while preserving the same append-only, double-entry ledger discipline that classical accounting settled on centuries before any of this ran on a computer.

Master this design and you’ve internalized patterns — CQRS, event sourcing, idempotent consumers, and sharding — that apply far beyond lending, to any system where correctness and internet-scale throughput must coexist. The same building blocks show up in payment processing, inventory management, and any domain where “the number must be exactly right” and “the number must be available to millions of users instantly” are both non-negotiable requirements at the same time.