Imagine you run a lemonade stand, but instead of one stand, you have thousands of them across the world and instead of collecting cash yourself, you let dozens of different banks collect the money on your behalf and send it to you later. Every evening, you need to answer one simple question: “Did I actually receive all the money that my records say I should have received?” If a bank sends you less than expected, or sends money for a sale you never recorded, you need to catch it — fast and at massive scale.

This is exactly the problem a Transaction Reconciliation System solves for any payment platform — whether it is a digital wallet, a card network, a marketplace like Amazon or a fintech like Stripe or Razorpay. Every payment a customer makes travels through several systems: the platform’s own internal ledger (the platform’s own bookkeeping of “what happened”) and then, separately, the banking partner’s settlement records (the bank’s own bookkeeping of “what money actually moved”). These two records are created by two completely different systems, often owned by two completely different companies and they must be proven to match — every day, for millions of transactions.

Reconciliation is not a new idea. Long before computers, accountants at banks reconciled paper ledgers by hand at the end of each day — a process called “balancing the books”. As transaction volumes exploded with the rise of credit cards in the 1960s–70s and then exploded again with the internet and mobile payments in the 2000s–2010s, manual reconciliation became physically impossible. A single large payment platform today can process tens of millions of transactions per day and reconciliation had to evolve from a nightly Excel-sheet exercise into a real-time, automated, distributed system capable of handling a million or more events per minute during peak load — think Black Friday, a big cricket match with in-app payments or a flash sale.

In this tutorial, we will design a production-grade Transaction Reconciliation System from the ground up — the architecture, the databases, the matching logic, how it scales to a million requests per minute, how it stays highly available, how it stays secure and how a Staff or Principal Engineer would defend every decision in a system design interview.

Let us define the problem precisely. A payment platform has two independent sources of truth about the same set of financial events:

• Internal Ledger: The platform’s own database that records every transaction it believes happened — a customer paid $50, a merchant was credited $48 after fees, a refund of $10 was issued and so on.
• Bank Settlement Records: Files or API responses sent by banking partners (card networks, ACH operators, UPI switches, local banks) describing what money actually moved between accounts, usually delivered in batches — end-of-day settlement files, intraday API callbacks or nightly SFTP drops.

These two records can disagree for many real reasons and the system’s entire job is to find and classify every disagreement:

The motivation for building a dedicated, automated system (instead of a nightly cron job with a SQL script) comes down to three forces pulling at once: scale (millions of transactions per day, spikes to a million per minute), correctness (money math must be exact — no floating-point rounding tricks) and speed (finance teams and merchants need same-day or near-real-time visibility into discrepancies, not a report that arrives a week later).

There is also a fourth, less obvious force: partner diversity. A payment platform of any real size does not settle with just one bank — it settles with dozens of banking partners, card networks and local payment rails, each with its own file format, delivery schedule, timezone convention and quirks. A system that hardcodes assumptions about “the bank’s” file format breaks the moment a second banking partner is onboarded. This is why, as we will see in the architecture, the parsing and validation layer is deliberately built as a pluggable, per-partner adapter system rather than a single monolithic parser.

3.1 Ledger

A ledger is simply an ordered, append-only record of financial events — “who owes whom, how much and when”. Think of it as a diary where every entry, once written, is never erased or edited, only added to (and corrected with a new offsetting entry if needed). This “append-only” property is critical for auditability: regulators and auditors need to see the full, untampered history.

3.2 Settlement

Settlement is the actual movement of money between bank accounts to make a transaction final. When you pay with a card, the “authorisation” happens instantly, but the real money often moves hours or days later in a batch process called settlement. The bank sends a settlement file (or an API feed) describing exactly what was settled.

3.3 Matching (or “reconciliation matching”)

Matching is the process of pairing a ledger record with its corresponding settlement record based on shared identifiers — usually a transaction ID, but sometimes a combination of amount, date and merchant reference when IDs do not line up cleanly (called fuzzy matching).

3.4 Tolerance window

Because of currency rounding and small fee differences, reconciliation systems allow a small tolerance — for example, a $0.01 difference might still be considered “matched” rather than flagged as an exception, depending on business rules.

3.5 Idempotency

An operation is idempotent if doing it multiple times has the same effect as doing it once. This matters enormously here: if a bank file gets delivered twice (which happens more often than you would think, due to retries), the system must not double-count it.

3.6 Exception

An exception is any transaction that could not be automatically and confidently matched. These get routed to a human-reviewable queue with full context (amount, timestamps, both-sided records) so an operations analyst can resolve them.

3.7 Break / discrepancy

Industry term for a confirmed, unresolved mismatch between the two ledgers — literally, the books do not “balance”.

3.8 CAP theorem in this context

The CAP theorem says a distributed system can only guarantee two of three properties during a network partition: Consistency, Availability and Partition tolerance. For financial reconciliation, we generally favour consistency over availability for the core ledger writes (we would rather momentarily reject a write than record wrong money), while the read-heavy reporting and dashboard layers can favour availability with eventual consistency.

3.9 Distributed locking & leader election

When multiple instances of the Bank File Ingestion Service poll the same SFTP folder, we need to guarantee only one instance processes a given file. This is solved with distributed locking — before processing file X, an instance must acquire a lock named lock:file:{fileId} in Redis (using SET key value NX PX ttl, an atomic “set if not exists with expiry”) or via a coordination service like ZooKeeper / etcd. If the lock acquisition fails, another instance already owns that file and this instance simply skips it. The TTL ensures that if the owning instance crashes mid-processing, the lock eventually expires and another instance can pick up the work — trading a small window of risk for guaranteed forward progress.

3.10 Bloom filters for fast duplicate detection

Checking “have I seen this settlement record ID before?” against a database for every single incoming record, at a million-per-minute scale, would hammer the database with point lookups. A Bloom filter — a compact, probabilistic data structure — sits in front of the database as a first-pass filter. It can say “definitely not seen before” with 100% certainty (letting the record flow straight through) or “possibly seen before” (in which case and only then, we pay for the more expensive database check). Because false positives are rare and false negatives are impossible by construction, this cuts duplicate-check database load by well over 90% in practice.

3.11 Data structures behind the matching engine

Internally, the Matching Engine relies on a few well-known data structures doing a lot of quiet, heavy lifting: hash maps (transaction ID to ledger entry, O(1) average lookup) power the deterministic matching pass; a sorted index (B-tree, as used by Postgres) on (merchant_ref, date) powers the range-scan needed for fuzzy matching within a date window; and an in-memory LRU cache keeps the most recently accessed ledger entries hot without unbounded memory growth. Understanding these underlying structures is what lets an engineer reason correctly about Big-O cost when tuning the matching pipeline for scale — a naive linear scan over all of a merchant’s transactions to find a fuzzy match would be catastrophic at volume; an indexed range query is not.

Every box in this architecture that receives external or inter-service traffic sits behind a Load Balancer and is reached only through the API Gateway, which handles authentication, authorisation and rate limiting before any request touches business logic. This is called out explicitly in each node below so the flow of “how a request actually reaches a service” is never ambiguous.

4.1 Component breakdown

DNS + Global Traffic Manager

The very first hop. Routes banking partners and internal services to the nearest healthy regional deployment using GeoDNS, so a bank in Frankfurt does not send its settlement file all the way to a US region unless it has to.

Load Balancer

A Layer 7 load balancer (e.g., an ALB/NLB equivalent or Envoy-based) terminates TLS, performs health checks against backend instances and spreads incoming traffic evenly. This is the component that lets us scale horizontally — add more instances behind it and it just starts sending them traffic.

API Gateway

Sits right after the load balancer. Handles authentication (mTLS for bank partners, OAuth2 / JWT for internal callers), authorisation, per-partner rate limiting (critical, since a misbehaving bank integration should never be able to overwhelm the system), request validation and routing to the correct downstream microservice.

Bank File Ingestion Service

Polls SFTP / API endpoints of each banking partner on a schedule (or receives push webhooks), downloads settlement files and hands them to the parser. It is stateless and horizontally scalable — each instance can own a subset of bank partner polling jobs, coordinated via a distributed lock (e.g., using Redis or ZooKeeper) so two instances never process the same file twice.

File Parser and Validator Service

Parses bank-specific file formats (fixed-width, CSV, XML, ISO 20022 for many modern banking rails) into a common internal schema, validates checksums and record counts against the file’s own header / trailer records and rejects malformed files back to an error queue rather than silently ingesting bad data.

Ledger Event Consumer Service

Subscribes to the platform’s internal transaction event stream (every payment, refund, chargeback or fee event published by the core payments system) and forwards a normalised copy into the reconciliation pipeline.

Kafka streaming backbone

The backbone that decouples ingestion from processing. Ledger events and settlement events land on separate topics, partitioned by a key (typically merchant ID or transaction ID hash) so that all events for the same transaction land on the same partition — this is what lets the Matching Engine process them in order without cross-partition coordination.

Normalization Service

Converts both ledger and settlement records into one canonical internal format: standardised currency codes, UTC timestamps and a derived matching key. This is where the two very different “shapes” of data (internal ledger schema vs. 12 different bank file schemas) become comparable.

Matching Engine Service

The heart of the system. Applies deterministic matching first (exact transaction ID + amount), then fuzzy matching rules for records that did not exact-match (amount within tolerance, date within a settlement window, reference number similarity). Emits a match result: RECONCILED, PARTIALLY_MATCHED or UNMATCHED.

Exception Management Service

Owns the lifecycle of unmatched / partially-matched records — creating a case, assigning it to an operations queue, tracking resolution and re-triggering matching if a late-arriving record shows up.

Reporting and Reconciliation API

Read-optimised API that powers dashboards, daily reconciliation reports and finance / compliance exports. Deliberately separated from the write path (CQRS) so heavy reporting queries never slow down the matching pipeline.

Data layer

A Redis cache cluster for hot lookups during matching, sharded PostgreSQL for the ledger and settlement records (strong consistency, relational integrity), a Cassandra-based results store optimised for high write throughput and time-range queries and object storage for archiving raw bank files (needed for audits).

Supporting services

Notification Service (alerts operations and finance teams), Audit Log Service (immutable record of every state change, required for compliance) and the Monitoring / Observability stack (metrics, distributed tracing, centralised logs).

Let us trace exactly what happens when a bank sends a settlement file, step by step, through a sequence diagram.

5.1 The matching algorithm, in plain English

The Matching Engine works in two passes, just like how you would first look for an exact receipt match in a pile of papers before squinting at ones that are “close enough”:

1. Pass 1 — Deterministic matching: Look up the settlement record’s transaction ID directly against the ledger. If found and the amount matches exactly (or within a configured tolerance, e.g., ₹0.01), mark it RECONCILED immediately. This handles the vast majority — often 95%+ — of transactions.
2. Pass 2 — Fuzzy matching: For anything left over, search using secondary keys: merchant reference number, amount + date window, or card’s last-4 + amount + approximate timestamp. If a confident fuzzy match is found, mark it PARTIALLY_MATCHED and queue for lightweight review; if nothing is found even after the fuzzy pass, mark it UNMATCHED and create a full exception case.

5.2 Matching Engine — simplified Java implementation

public class MatchingEngine {

    private final LedgerRepository ledgerRepository;
    private final ReconciliationResultRepository resultRepository;
    private final ExceptionService exceptionService;
    private static final BigDecimal TOLERANCE = new BigDecimal("0.01");

    public MatchResult match(SettlementRecord settlement) {
        // Pass 1: Deterministic match on transaction id
        Optional<LedgerEntry> exact =
            ledgerRepository.findByTransactionId(settlement.getTransactionId());

        if (exact.isPresent() && amountsMatch(exact.get().getAmount(), settlement.getAmount())) {
            return recordMatch(settlement, exact.get(), MatchStatus.RECONCILED);
        }

        // Pass 2: Fuzzy match on merchant ref + amount + date window
        List<LedgerEntry> candidates = ledgerRepository
            .findByMerchantRefAndDateRange(
                settlement.getMerchantRef(),
                settlement.getSettlementDate().minusDays(1),
                settlement.getSettlementDate().plusDays(1)
            );

        for (LedgerEntry candidate : candidates) {
            if (amountsMatch(candidate.getAmount(), settlement.getAmount())) {
                return recordMatch(settlement, candidate, MatchStatus.PARTIALLY_MATCHED);
            }
        }

        // No match found at all -> raise an exception case
        exceptionService.createException(settlement, ExceptionReason.NO_LEDGER_MATCH);
        return recordMatch(settlement, null, MatchStatus.UNMATCHED);
    }

    private boolean amountsMatch(BigDecimal ledgerAmount, BigDecimal settlementAmount) {
        // Always compare using BigDecimal, never float/double, to avoid rounding errors
        return ledgerAmount.subtract(settlementAmount).abs()
                .compareTo(TOLERANCE) <= 0;
    }

    private MatchResult recordMatch(SettlementRecord settlement, LedgerEntry ledger, MatchStatus status) {
        ReconciliationResult result = ReconciliationResult.builder()
                .settlementId(settlement.getId())
                .ledgerId(ledger != null ? ledger.getId() : null)
                .status(status)
                .matchedAt(Instant.now())
                .build();
        resultRepository.save(result);
        return new MatchResult(status, result);
    }
}

5.3 State machine for a transaction record

5.4 Idempotent ingestion — simplified Java implementation

Bank files sometimes get re-delivered (a retry after a network blip or a bank’s own system re-sending a “just in case” copy). The ingestion service must detect and skip anything already processed, using a fast Bloom-filter pre-check backed by a durable idempotency store.

public class IdempotentFileIngestor {

    private final BloomFilter<String> recentFileFilter; // fast, in-memory pre-check
    private final IdempotencyKeyStore idempotencyStore; // durable, source of truth
    private final KafkaProducer<String, SettlementBatch> producer;

    public IngestResult ingest(SettlementFile file) {
        String idempotencyKey = buildKey(file);

        // Fast path: bloom filter says "definitely not seen" -> skip DB check
        if (recentFileFilter.mightContain(idempotencyKey)) {
            // Slow path: confirm against durable store before trusting the maybe
            if (idempotencyStore.exists(idempotencyKey)) {
                return IngestResult.skippedDuplicate(file.getFileName());
            }
        }

        List<SettlementRecord> records = parseAndValidate(file);
        SettlementBatch batch = SettlementBatch.builder()
                .idempotencyKey(idempotencyKey)
                .records(records)
                .receivedAt(Instant.now())
                .build();

        // Persist the key BEFORE publishing, so a crash after publish
        // never leaves us without a duplicate-detection record
        idempotencyStore.save(idempotencyKey, file.getChecksum());
        recentFileFilter.put(idempotencyKey);

        producer.send(new ProducerRecord<>("settlement-events", batch.getMerchantId(), batch));
        return IngestResult.accepted(file.getFileName(), records.size());
    }

    private String buildKey(SettlementFile file) {
        // Bank id + file name + checksum uniquely identifies a file delivery
        return file.getBankId() + ":" + file.getFileName() + ":" + file.getChecksum();
    }
}

A single transaction’s data touches the system twice, at two different points in time and the reconciliation pipeline’s job is to eventually bring both touches together:

1. T+0 (transaction time): Customer pays. The core payments system writes a ledger entry and publishes a “PaymentCompleted” event onto the internal event stream. The Ledger Event Consumer picks this up within seconds and stores a normalised copy.
2. T+1 to T+3 (settlement time): The bank batches transactions and, depending on the rail (card networks vs. ACH vs. UPI vs. wire), sends a settlement file anywhere from same-day to 2–3 business days later. The Bank File Ingestion Service picks this up, parses it and normalises it.
3. Matching window: Once both sides exist in the normalised store, the Matching Engine attempts to pair them. If the settlement side arrives before the ledger side (rare, but possible with fast rails), the record waits in a “pending match” state with a TTL and matching is retried when the missing side shows up.
4. Terminal state: Every record eventually reaches RECONCILED or WRITTEN_OFF (after manual review), which is what closes the books for that day / batch.

6.1 Batch vs. streaming reconciliation

Most production systems use a hybrid: streaming for same-day, low-latency matching (catch fraud and failures fast) and a nightly batch reconciliation job that re-runs matching against the full day’s data as a safety net — catching anything the streaming path missed due to late-arriving files, ordering issues or transient failures. The batch job is the “final answer” used for financial close and audit reporting.

7.1 Key trade-off: strict matching vs. fuzzy matching

Stricter matching (exact ID + exact amount only) produces fewer false positives but pushes more transactions into the manual exception queue, increasing operational headcount cost. Looser fuzzy matching reduces the exception queue but risks incorrectly pairing two different transactions that happen to share an amount and rough timeframe — a serious problem if it hides a real discrepancy. Most mature systems tune this per-partner, since a bank with clean, well-structured IDs needs less fuzzy matching than one with messy legacy file formats.

7.2 Key trade-off: synchronous vs. asynchronous matching

Matching synchronously (as soon as a settlement record arrives) gives the freshest possible view but couples ingestion throughput to matching throughput. Asynchronous matching via Kafka decouples them, letting ingestion absorb spikes without dropping data — at the cost of a small, bounded delay before a transaction’s status is known.

Let us ground this in real numbers. A million requests per minute is roughly 16,667 requests per second sustained, with realistic peaks 2–3x higher during bursts — so the system must comfortably absorb 40,000–50,000 requests / second at peak without falling over. Here is how each layer is designed to hit that number.

8.1 Partitioning strategy

Both Kafka topics and the Matching Engine instances are partitioned by hash(merchant_id + transaction_id). This guarantees all events for one transaction land on the same partition (so ordering is preserved) while spreading load evenly across hundreds of partitions. The database layer mirrors this with consistent hashing across shards, so adding a new shard only requires reshuffling a small fraction of the keyspace.

8.2 Horizontal scaling at every tier

8.3 Reducing load before it hits the database

• Cache-aside pattern: The Matching Engine checks Redis before querying Postgres for candidate ledger entries, cutting database read load dramatically for hot, recent transactions.
• Batching: Settlement files are processed in chunks (e.g., 10,000 records per batch job) rather than row-by-row API calls, reducing per-record overhead.
• Backpressure: Kafka consumer lag is monitored; if the Matching Engine falls behind, ingestion continues writing to Kafka (which absorbs the burst) rather than blocking upstream services.
• Bulk writes: Reconciliation results are written to Cassandra in batches using prepared statements, not one write per record.

8.4 Back-of-the-envelope capacity planning

A useful interview habit is deriving concrete numbers rather than waving hands at “it scales”. Let us do the math for our 1M requests / minute target:

• Sustained throughput: 1,000,000 / 60 ≈ 16,667 events/sec. With a realistic 2.5x burst multiplier during peak windows, design for ~42,000 events/sec.
• Kafka partitions needed: If a single partition consumer can sustainably process ~2,000 events/sec (including the DB / cache round-trips involved in matching), we need at least 42,000 / 2,000 ≈ 21 partitions per topic, rounded up and padded for headroom — typically provisioned at 48–64 partitions to allow future consumer scale-out without a repartition.
• Matching Engine instances: One instance per partition (at minimum) means ~48–64 pod replicas at peak, scaled down to a much smaller baseline (e.g., 8–12) during off-peak hours via Kubernetes Horizontal Pod Autoscaler.
• Database write capacity: If each matched record generates one write of roughly 500 bytes, 42,000 writes/sec × 500 bytes ≈ 21 MB/sec sustained write throughput — well within range for a properly sharded Cassandra cluster, but would saturate a single unsharded Postgres instance, confirming our earlier decision to shard.
• Redis cache sizing: If we keep 3 days of hot transactions cached, at roughly 1KB per cached entry and 20M transactions/day, that is 3 × 20M × 1KB ≈ 60 GB of cache data — comfortably fits a modestly sized Redis Cluster with room to grow.

This kind of estimation — even if the exact constants are approximate — demonstrates the ability to translate a business requirement (“a million requests a minute”) into concrete infrastructure sizing decisions, which is exactly what a Staff Engineer interview is probing for.

8.5 Concurrency model

Within a single Matching Engine instance, records from different Kafka partitions are processed on separate worker threads (or separate async event loops in a reactive framework), since ordering only needs to be preserved within a partition, not across partitions. This lets one instance fully utilise multi-core hardware — a common pattern is one consumer thread pulling from a partition, handing work off to a bounded thread pool for the actual matching logic (which includes I/O-bound cache / DB calls) and committing the Kafka offset only after the write is durably persisted, to avoid losing work on a crash between “processed” and “committed”.

A reconciliation system cannot afford to silently lose a settlement file — that is literally the definition of “money going missing”. Every layer is designed with redundancy and explicit failure handling.

9.1 No single point of failure

• Multi-AZ deployment: Every stateless service runs across at least 3 availability zones; losing one AZ does not take down the system.
• Kafka replication: Topics are replicated with a factor of 3, so broker failure does not lose in-flight events.
• Database replication: Each Postgres shard has synchronous replicas for failover, plus async cross-region replicas for disaster recovery.
• Idempotent consumers: Every consumer tracks processed message IDs (or uses Kafka’s exactly-once semantics with transactional producers) so a consumer crash-and-restart never double-processes a settlement file.

9.2 Retry & dead-letter handling

Transient failures (a downstream database timeout, a temporary network blip) are retried with exponential backoff and jitter. Messages that fail repeatedly (say, after 5 retries) are routed to a dead-letter queue rather than blocking the pipeline forever — an alert fires and an engineer investigates without the rest of the day’s transactions getting stuck behind one bad record.

9.3 Disaster recovery

Raw settlement files are archived immutably in object storage (with versioning) the moment they are received — before any parsing happens. If a bug in the parser corrupts data downstream, the system can always re-ingest from the original raw file, giving a clean recovery path. RPO (Recovery Point Objective) is targeted near-zero via synchronous replication on the ledger / settlement databases; RTO (Recovery Time Objective) is minimised with automated failover (typically under 60 seconds for stateless tiers, a few minutes for database failover).

9.4 Consensus and coordinated failure recovery

Some operations genuinely need cluster-wide agreement rather than simple per-instance locking — for example, “which instance is currently the leader responsible for triggering the nightly full-batch reconciliation job”. This is a textbook use case for a consensus protocol like Raft, typically consumed off-the-shelf via etcd or ZooKeeper rather than implemented from scratch. A leader is elected among candidate instances; if the leader crashes or becomes unreachable, the remaining nodes detect the missing heartbeat and elect a new leader within a bounded time window, ensuring the nightly job always runs exactly once even during infrastructure failures.

Failure recovery follows a layered strategy depending on blast radius:

9.5 Chaos testing

Mature reconciliation platforms periodically run controlled failure-injection exercises — deliberately killing a Matching Engine pod mid-batch, introducing artificial network latency between the service and its database, or simulating a Kafka broker loss — in a staging or shadow-production environment. This validates that the retry, dead-letter and failover mechanisms designed on paper actually behave correctly under real failure conditions, rather than discovering gaps for the first time during an actual incident.

9.6 Runbooks and operational readiness

Every alert defined in the monitoring layer maps to a documented runbook — a step-by-step guide an on-call engineer follows during an incident, rather than improvising under pressure at 3 AM. A good runbook for “Kafka consumer lag exceeding SLA”, for example, walks through checking current partition count versus consumer instance count, confirming whether autoscaling has already kicked in, identifying whether the lag is isolated to one banking partner (often a sign of a partner-specific data quality issue) or system-wide (often a sign of a genuine capacity shortfall) and the exact commands or dashboard links needed to scale out consumer instances immediately. Runbooks are treated as living documents, updated after every incident postmortem and tested periodically through game-day exercises so the team’s muscle memory stays current even for failure modes that rarely occur in practice.

This system touches financial data end-to-end, so it sits squarely under PCI-DSS and SOC 2 scope.

• Transport security: All bank file transfers use SFTP over SSH or mutual TLS; internal service-to-service traffic uses mTLS.
• Encryption at rest: Ledger, settlement and results databases are encrypted at rest (AES-256), with separate key management (KMS) per data classification tier.
• Field-level protection: Sensitive fields like full card numbers are never stored — only tokenised references or last-4 digits, keeping the system out of the highest PCI-DSS scope tiers where possible.
• AuthN / AuthZ at the gateway: Each banking partner authenticates with a unique certificate or API key; internal callers use short-lived JWTs scoped to specific operations (read-only reporting vs. write access to exception resolution).
• Least privilege: The Reporting API has read-only database credentials; only the Matching Engine and Normalization Service can write to the core tables.
• Immutable audit log: Every state transition (who resolved an exception, when and how) is written to an append-only audit log, satisfying compliance requirements for traceability.
• Rate limiting and abuse protection: Per-partner rate limits at the API Gateway prevent a compromised or misbehaving integration from flooding the pipeline.

10.1 Compliance considerations

Beyond PCI-DSS, a reconciliation system typically falls under several overlapping regulatory regimes depending on geography and the nature of the platform:

• SOC 2 Type II: Requires demonstrable, continuous controls around data access, change management and monitoring — the immutable audit log and role-based access control directly support this.
• GDPR (Europe) / data localisation laws: If settlement data includes any personal data (cardholder name, account holder details), it must be handled per regional data residency rules — often requiring reconciliation infrastructure to be regionally deployed rather than globally centralised.
• Central bank / regulator mandates: Many jurisdictions (e.g., RBI in India, the Federal Reserve in the US for certain rails) mandate same-day or next-day reconciliation with formal discrepancy reporting — which is exactly what drives the “streaming plus nightly batch safety net” design discussed earlier.

10.2 Data retention and right-to-erasure tension

Financial audit requirements typically mandate retaining transaction records for 7+ years, which can be in tension with data-subject erasure rights under privacy law. The common resolution is to tokenise or pseudonymise personally identifiable fields after an active window while retaining the financial amounts and identifiers needed for audit — satisfying both requirements without deleting financially material records.

You cannot trust a reconciliation system you cannot observe — the entire point of the system is trust in numbers, so its own health metrics matter enormously.

11.1 Key metrics to track

11.2 Tracing and logging

Every transaction carries a correlation ID from the moment it enters the ledger through to its final reconciliation status, propagated across every service via distributed tracing (OpenTelemetry-style spans). This lets an engineer answer “what happened to transaction X?” in seconds instead of grepping logs across 8 services. Structured logs (JSON) are centralised, with PII / sensitive fields redacted before indexing.

11.3 Alerting

Alerts fire on: match rate dropping below a threshold (e.g., below 98% for a given bank partner), consumer lag exceeding a time-based SLA, dead-letter queue growth and any daily balance discrepancy exceeding a configured dollar threshold — the last one typically pages a human immediately, since it is the highest-severity signal the system can produce.

11.4 Dashboards for different audiences

Not every stakeholder needs the same view into the system, so the monitoring layer is deliberately split into three tiers. Engineering dashboards focus on infrastructure health — consumer lag, error rates, pod restarts, database connection pool saturation — the signals needed to keep the pipeline itself running. Operations dashboards focus on the exception queue — open case count, average time-to-resolution, cases aged past SLA — the signals a reconciliation analyst uses to triage their day. Finance and compliance dashboards focus on business-level outcomes — daily reconciled amount, outstanding discrepancy total, trend lines by banking partner — the signals used for financial close and regulatory reporting. Building all three from the same underlying event stream (rather than three disconnected systems) ensures every audience is looking at numbers that are provably consistent with each other.

11.5 SLA definitions

A well-run reconciliation platform commits to explicit, measurable SLAs rather than vague promises of “fast reconciliation”. Typical targets look like this:

These SLAs, once published internally (and sometimes externally to banking partners), directly shape architecture decisions — for instance, the “5 minute p95 match latency” target is exactly why the design favours streaming matching via Kafka over a purely batch-oriented approach.

Services are packaged as containers and deployed on Kubernetes, giving us declarative scaling, self-healing (crashed pods restart automatically) and rolling deployments with zero downtime.

• Blue-green or canary deploys for the Matching Engine specifically — since bugs in matching logic have direct financial consequences, changes are rolled out to a small percentage of traffic first, with match-rate metrics watched closely before full rollout.
• Infrastructure as Code (Terraform or equivalent) defines every environment, so disaster recovery regions can be stood up predictably and consistently.
• Multi-region active-active for the edge / gateway tier, with the core ledger / settlement databases typically active-passive (single write region) to keep financial writes strongly consistent, with cross-region read replicas for reporting.
• Separate environments (dev, staging, prod) with staging fed by anonymised / synthetic bank file samples for safe testing of new bank integrations.

12.1 Rollback strategy

Every deployment is paired with a fast, tested rollback path. Because the Matching Engine’s behaviour is driven partly by configuration (tolerance thresholds, per-partner fuzzy-matching rules) rather than only code, configuration changes are versioned and deployed independently from code releases, using a feature-flag style system. This means a bad matching-rule change can be reverted in seconds via a config flag flip, without waiting for a full container rebuild-and-redeploy cycle — a meaningful difference when a bad rule change is actively misclassifying live financial transactions.

12.2 Cost optimisation

Given the bursty nature of payment traffic (predictable daily peaks, occasional flash-sale spikes), the platform mixes reserved / committed-use capacity for baseline load with autoscaled on-demand or spot-style capacity for burst absorption on the stateless tiers (API Gateway, Matching Engine). Stateful tiers (databases, Kafka brokers) run on reserved capacity, since they cannot scale down as elastically. Cold, older reconciliation data (beyond the active audit window) is tiered from the primary Cassandra cluster into cheaper object storage with a queryable archive layer, cutting storage cost for data accessed rarely but still legally required to be retained.

13.1 Why different databases for different jobs (polyglot persistence)

13.2 Load balancing strategy

The edge Load Balancer uses round-robin with health-check-based ejection at Layer 7. Within Kafka, “load balancing” happens through partition assignment — each consumer instance in a consumer group is assigned a subset of partitions and Kafka automatically rebalances when instances join or leave, giving us elastic scaling without manual partition management.

13.3 Caching strategy

Cache-aside for candidate ledger lookups (write-through on ledger normalisation, so the cache is warm by the time settlement records arrive), with a TTL of a few days matching the typical settlement window, after which cold data falls back to Postgres. Cache keys are structured as ledger:{merchant_id}:{txn_id} to enable fast point lookups without scanning.

13.4 Sharding strategy in detail

The ledger and settlement Postgres clusters are sharded using consistent hashing on merchant ID rather than simple modulo hashing. The difference matters at scale: with modulo hashing (shard = hash(key) % N), adding a single new shard changes the target shard for almost every existing key, forcing a massive, disruptive data migration. Consistent hashing arranges shards on a conceptual ring, so adding a new shard only reassigns the fraction of keys that fall in the region of the ring near the new shard — typically around 1/N of the data, not all of it.

Choosing merchant ID (rather than transaction ID) as the shard key is deliberate: it keeps all of a single merchant’s transactions co-located on one shard, so a merchant’s daily reconciliation report never needs a cross-shard scatter-gather query — a single shard can answer it directly. The trade-off is that very high-volume “whale” merchants can create hot shards; this is mitigated with a secondary sub-sharding scheme for the handful of merchants that individually exceed a configured transaction-volume threshold.

The system exposes a small, well-scoped set of REST / gRPC APIs behind the gateway:

Internally, services communicate via gRPC for low-latency synchronous calls (e.g., Exception Service calling Notification Service) and Kafka for asynchronous, high-volume event flows (ingestion to matching). This split — synchronous APIs for control-plane operations, async events for data-plane volume — is a common and effective microservices pattern for high-throughput systems.

14.1 API design details worth getting right

• Idempotency keys on write endpoints: POST /v1/settlements/upload requires an idempotency key header, so a retried upload (due to a client timeout, for example) never creates a duplicate settlement batch — mirroring the same idempotency principle used internally for file ingestion.
• Cursor-based pagination: GET /v1/exceptions uses opaque cursor tokens rather than page numbers, since offset-based pagination degrades badly on large, actively-changing result sets and can skip or duplicate rows as new exceptions are created mid-scroll.
• Explicit error codes: Every API error response carries a machine-readable error code (e.g., PARTNER_NOT_AUTHORIZED, FILE_CHECKSUM_MISMATCH, DUPLICATE_SUBMISSION) in addition to a human-readable message, so calling systems (including bank partner integrations) can programmatically branch on failure type rather than parsing free-text strings.
• Versioned contracts: All endpoints are versioned in the URL path (/v1/), so breaking changes ship as a new version rather than silently changing behaviour underneath existing integrations — banking partner integrations, in particular, are slow-moving and cannot tolerate silent contract changes.

15.3 Strategy pattern for per-partner parsing

Since every banking partner delivers a different file format, the File Parser and Validator Service implements a Strategy pattern: a common SettlementFileParser interface with one concrete implementation per bank (e.g., VisaFileParser, LocalBankAchParser, Iso20022Parser), selected at runtime based on the source partner’s identity. Onboarding a new banking partner then means adding one new implementation class and a routing entry, without touching the Matching Engine, the Kafka topic structure or any downstream service at all — a clean illustration of the Open-Closed Principle in a real production system.

public interface SettlementFileParser {
    List<SettlementRecord> parse(RawFile file) throws ParseException;
    boolean supports(String bankPartnerId);
}

public class ParserRegistry {
    private final List<SettlementFileParser> parsers;

    public SettlementFileParser resolve(String bankPartnerId) {
        return parsers.stream()
                .filter(p -> p.supports(bankPartnerId))
                .findFirst()
                .orElseThrow(() -> new UnsupportedPartnerException(bankPartnerId));
    }
}

16.1 Best practices

• Always persist the raw, untouched bank file before any transformation — it is your source of truth for replay and audit.
• Design matching keys and tolerance rules per banking partner, not globally, since file quality varies widely.
• Run a nightly full-batch re-reconciliation as a safety net on top of streaming matching.
• Make every write idempotent using unique settlement / ledger identifiers.
• Track the “total dollar balance” metric, not just “percentage matched” — a 99% match rate can still hide a very large dollar-value discrepancy in the remaining 1%.
• Build the Exception Management UI / API early — it is often under-invested in, yet it is what operations teams live in daily.

16.2 Common mistakes

• Treating reconciliation as a one-time nightly batch job instead of a continuously running pipeline, leading to late fraud / failure detection.
• Hardcoding bank-specific file parsing logic inline in the core matching service, making it hard to onboard new banking partners without redeploying core logic.
• Not versioning bank file format changes — banks do change their file layouts and an unversioned parser breaks silently.
• Underestimating exception queue growth during high-traffic events, leaving operations teams overwhelmed exactly when scrutiny is highest.

Across all of these, the same core pattern repeats: event-driven ingestion, a durable streaming backbone, a matching engine with deterministic-then-fuzzy logic and a human-in-the-loop exception workflow — the exact architecture we have built in this tutorial.

17.1 A composite case study: flash sale day

Consider a hypothetical but realistic scenario combining several of the patterns above. A large e-commerce platform runs a 24-hour flash sale, driving transaction volume to 8x normal levels. The Ingestion Layer, pre-scaled ahead of the known event based on historical Black Friday data, absorbs the burst into Kafka without loss. The Matching Engine, running at its pre-provisioned peak partition count, keeps consumer lag under five minutes throughout the event by temporarily reducing fuzzy-matching depth for the lowest-risk transaction categories (small consumer purchases under a configured amount threshold), while keeping full-depth matching active for high-value transactions where a missed discrepancy would be more costly. The Exception Management queue grows but stays within operational capacity because false-positive exceptions are minimised by the tuned tolerance thresholds discussed earlier. By the following morning, the nightly full-batch reconciliation job re-processes the entire day at full matching depth, closing out any records the streaming path had deprioritised — and the finance team has a fully reconciled, audit-ready daily close by the committed SLA, despite the 8x traffic spike.

Q: Why not just use a database JOIN to match ledger and settlement records?

A SQL join works fine for small, single-database datasets, but breaks down at scale: the two record sets often live in different systems entirely, arrive at different times (sometimes days apart), need fuzzy comparison logic beyond simple equality and require streaming / incremental processing rather than a full-table batch scan every time. A dedicated matching pipeline gives you incremental, event-driven matching with proper state tracking.

Q: How is this different from a general-purpose ETL pipeline?

It shares some DNA (ingest, transform, load) but reconciliation adds domain-specific concerns that generic ETL does not: financial correctness guarantees (BigDecimal math, idempotency), a formal exception workflow with human review and compliance-grade auditability requirements.

Q: What happens if a bank sends a corrupted or incomplete file?

The File Parser and Validator Service checks the file’s header / trailer record counts and checksums. If validation fails, the file is quarantined (not processed), the raw file is still archived and an alert notifies both the operations team and, typically, the bank partner’s integration contact to resend a corrected file.

Q: How do you handle multi-currency reconciliation?

Amounts are normalised to a base currency using the exchange rate in effect at transaction time (stored alongside the record, not recalculated later, since rates change) and matching compares both the original-currency amount and the normalised amount, since rounding during conversion is a common source of small discrepancies.

Q: Can this system detect fraud, not just accounting errors?

Yes, indirectly — unusual patterns like ledger entries with no corresponding settlement across many transactions from one merchant, or settlement amounts consistently higher than ledger amounts, are exactly the kind of signal that gets surfaced through the exception queue and can feed a separate fraud detection system as an input signal.

Q: How would you test the Matching Engine’s correctness before deploying a change?

Three layers: unit tests against known matching edge cases (exact match, tolerance-boundary match, no match), a replay test suite that re-runs the new matching logic against a large sample of historical production data and diffs the results against the previous version’s known-good output and a canary deployment that routes a small percentage of live traffic through the new version while comparing match-rate metrics against the baseline before full rollout.

Q: What is the difference between this and a plain accounting reconciliation tool like QuickBooks-style software?

Traditional accounting reconciliation tools are built for human-paced, relatively low-volume matching (hundreds to thousands of records) with a UI-driven workflow. This system is built for machine-paced, extremely high-volume matching (millions per day, spikes to a million per minute) where the vast majority of matching must happen with zero human involvement and humans only touch the small residual exception queue.

Q: How do you handle a banking partner that changes their settlement file format without notice?

The File Parser and Validator Service is built with a per-partner, versioned schema / adapter (a strategy pattern — one parser implementation per bank format version). Unexpected schema drift trips the file’s validation step (unexpected column count, missing expected header fields), routing the file to quarantine and alerting engineering, rather than attempting a best-effort parse that could silently corrupt data.