Every time someone pays with a card, there is a small, invisible promise being made. The cardholder promises the charge is legitimate. The merchant promises to deliver what was paid for. The bank promises to move the money correctly. Most of the time, this promise holds. But sometimes it breaks — a package never arrives, a subscription keeps charging after cancellation, or a stolen card number gets used to buy something. When that happens, the cardholder can ask their bank to reverse the payment. This reversal is called a chargeback, and the whole structured argument that follows between the cardholder’s bank, the merchant’s bank, the card network, and the merchant is called dispute processing.

To understand why this needs a serious piece of engineering, imagine a simpler world first: a lemonade stand. If a customer pays with cash and later feels cheated, they can only complain to the stand owner directly — there is no bank in the middle to force a refund. Card payments are different because a bank sits between the buyer and seller, and that bank has agreed, as part of a network like Visa or Mastercard, to protect cardholders from fraud and bad service. This protection is what gives people the confidence to type their card number into a website they have never used before. Without dispute rights, online commerce as we know it would not exist.

Chargebacks are not new. They started in the 1970s in the United States as a consumer-protection mechanism under laws like the Fair Credit Billing Act, originally designed for paper statements and phone-based bank disputes. Back then, a dispute might take a bank clerk weeks to resolve by mail. As card payments moved online and transaction volumes exploded — modern card networks process tens of thousands of transactions per second globally — the old manual process could not keep up. A payment platform processing millions of transactions a day might see tens of thousands of disputes a month, each with strict, network-mandated deadlines measured in days, not weeks. This is why chargeback and dispute processing evolved from a back-office paperwork function into a real-time, rules-driven distributed system.

For a payment platform — think of a company that processes payments on behalf of thousands of merchants, similar in spirit to Stripe, Adyen, or a bank’s merchant-acquiring division — dispute processing is not optional. Card network rules (from Visa, Mastercard, American Express, Discover, and regional networks like RuPay or UnionPay) mandate specific timeframes, specific evidence formats, and specific reason codes. Missing a deadline by even a few hours can mean an automatic loss for the merchant, regardless of whether the merchant was actually right. This is why the system we are about to design has to be fast, precise, auditable, and resilient — because money, trust, and regulatory obligations all depend on it.

1.1 The problem, stated precisely

If we strip away the jargon, the engineering problem is this: build a system that can receive a claim from an external, authoritative party (the card network), attach a strict, legally meaningful deadline to it, collect a structured response from another party (the merchant), evaluate that response against a rulebook that changes over time, and report a binding decision back — all while never losing track of a single case, never double-charging or double-refunding anyone, and never missing a deadline due to an internal outage. It sounds simple as a sentence, but each clause hides real distributed-systems difficulty: “never lose track” implies durability guarantees; “never double” implies idempotency and exactly-once semantics; “never miss a deadline due to an outage” implies the system’s reliability bar is tied directly to a wall clock it does not control.

It also helps to understand why this problem cannot simply be delegated to a human support team, even though a human support team is very much part of the real system. At scale, a payment platform might have anywhere from a few thousand to several hundred thousand open cases at any given moment, each ticking down on its own independent clock, each requiring a different evidence checklist depending on its reason code, and each carrying real financial consequences. No team of humans, however large, can reliably track that many independent deadlines by hand — the software has to do the tracking, and the humans (merchants, support agents, risk analysts) act on what the software surfaces to them.

1.2 Why this system matters for an architect

This is not a run-of-the-mill CRUD problem. A chargeback and dispute processing platform sits at the intersection of five hard sub-problems that architects are expected to reason about together:

• Time as a first-class trigger — unlike most systems, state can change simply because a clock ran out, so durable timers are load-bearing infrastructure.
• Strict, network-mandated deadlines — missing one is not a bug you patch tomorrow; it is a direct financial loss for a real merchant.
• Long-lived workflows — a single case can stay open for weeks or months, spanning many service deployments, config changes, and node failures.
• Regulatory audit and immutability — every state change must be traceable to a specific policy version and a specific piece of evidence, for years.
• Multi-party integration — the platform must correctly speak the language of at least three parties (card networks, merchants, and its own internal ledger) that each evolve independently.

Before drawing boxes and arrows, it helps to name the actors so the diagram makes sense. There are five parties in almost every dispute: the cardholder (the person who paid), the issuing bank (the cardholder’s bank, which issued the card), the card network (Visa, Mastercard, etc., which sets the rules and routes messages), the acquiring bank (the bank that lets the merchant accept cards, often represented by our payment platform), and the merchant (who sold the goods or service). Our payment platform typically sits on the acquiring side, acting as the merchant’s representative and the bridge to the card network.

A chargeback and dispute processing system is best thought of as an event-driven, workflow-orchestration platform layered on top of a strict state machine. Every dispute is a “case” that moves through well-defined states, and every state transition has a deadline, an owner, and required evidence. The architecture below shows the major building blocks a platform needs at scale.

2.1 Component-by-component breakdown

Internally, the system behaves like an orchestrated state machine where each dispute case moves through a fixed set of states, and transitions are triggered either by external events (a card network message, a merchant action) or by internal timers (a deadline expiring). This is fundamentally different from a typical CRUD app because time itself is a first-class trigger — a case can change state even if nobody clicks anything, simply because a clock ran out.

Each state transition writes an immutable event to an append-only ledger before any other side effect happens. This is important: if the notification service fails to email the merchant, or the analytics pipeline is temporarily down, the case’s state is still correctly recorded, and those side effects can be safely retried. This pattern — writing the fact first, then reacting to it — is the foundation of event sourcing, and it gives the system a built-in audit trail, which regulators and card networks actually require.

3.1 Deadline tracking

Because so much of the state machine is timer-driven, the system needs a reliable way to fire an action at a specific future time, even across service restarts, deployments, and node failures. A naive in-memory timer would lose all pending deadlines on a crash. Production systems instead use a durable scheduling mechanism: each case writes a “due at” timestamp to the database, and a background sweeper service continuously scans for cases whose deadline has passed, or better, uses a purpose-built delayed-message mechanism (like a Kafka topic with per-message delivery delay, or a distributed scheduler service) so that deadline firing is decoupled from any single service instance staying alive.

3.2 Idempotency and exactly-once effects

Card networks retry messages. Merchants double-click submit buttons. Network calls time out and get retried by client libraries. Every single operation in this system — creating a case, applying a state transition, sending a notification — must be idempotent, meaning doing it twice has the same effect as doing it once. This is usually achieved by assigning a unique idempotency key to every inbound message (often the network’s own dispute reference ID) and checking a “already processed” table before applying any effect.

public CaseCreationResult createCase(NormalizedDispute dispute) {
    String idempotencyKey = dispute.getNetworkReferenceId();

    Optional<CaseCreationResult> existing = idempotencyStore.find(idempotencyKey);
    if (existing.isPresent()) {
        // Same dispute seen before - return the original outcome unchanged
        return existing.get();
    }

    Case created = caseRepository.insertNew(dispute);
    outbox.publish(new CaseCreatedEvent(created.getId(), dispute));
    CaseCreationResult result = CaseCreationResult.of(created);
    idempotencyStore.save(idempotencyKey, result, Duration.ofDays(90));
    return result;
}

3.3 Concurrency control on a single case

It is entirely possible for two different triggers to try to act on the same case at nearly the same instant — for example, a deadline sweeper firing “mark as deadline missed” at the exact moment a merchant’s evidence upload finally completes and tries to mark the case “evidence submitted.” Without protection, this is a classic race condition, and the outcome would depend on which write happened to land last, which is not an acceptable way to decide a financial outcome. The Case Management Service protects against this with optimistic concurrency control: every case row carries a version number, and every update includes the version it expects to be updating from. If the version has already moved (because another process updated it first), the write is rejected and the service re-reads the latest state before deciding what to do next, rather than blindly overwriting a decision that already happened.

This matters because the two competing outcomes are not equivalent — one path results in the merchant losing money, the other does not — so the system needs a clear, deterministic tie-breaking rule, not just a mechanism to avoid corruption. A common rule is “if evidence was submitted before the deadline sweeper’s read of the current time, evidence wins,” which is enforced by comparing timestamps captured at the moment of the actual event, not at the moment the asynchronous worker happens to process it.

3.4 Consensus for leader election

The redundant deadline sweepers mentioned earlier need to agree on which instance is currently “the leader” allowed to fire actions, so the same expiry is not processed twice. This is a textbook distributed consensus problem, typically solved using a coordination service (like one built on the Raft consensus algorithm) that provides distributed locks with automatic expiry, so that if the current leader crashes, its lock lease expires and another sweeper instance can safely take over within a bounded amount of time.

3.5 CAP theorem in this context

The CAP theorem states that a distributed system facing a network partition must choose between consistency and availability, since it cannot fully guarantee both at the same instant. For the ledger and core case state, this system deliberately leans toward consistency: if a shard’s replicas cannot agree during a partition, it is safer to briefly reject writes to that shard than to risk two conflicting versions of a financial decision being accepted independently and needing to be reconciled later. For less critical paths, like the analytics pipeline or dashboard read cache, the system leans toward availability instead, accepting temporarily stale reads rather than blocking the merchant dashboard entirely during a transient network issue. This is a good example of why CAP trade-offs are rarely a single, system-wide decision — they are made independently for each component based on what that component actually protects.

3.6 Data structures behind the deadline sweeper

At the core of the deadline sweeper sits a conceptually simple but performance-critical data structure: a priority queue ordered by deadline timestamp, allowing the system to always efficiently ask “what is the next thing that needs attention” without scanning every open case on each pass. In practice this is implemented not as an in-memory heap, which would not survive a restart, but as a database index that behaves like a persistent priority queue, letting the sweeper query only the small slice of cases near the front of the queue on every pass, regardless of how many millions of far-future cases exist behind it.

Let’s trace one dispute end-to-end, the way an interviewer will often ask you to “walk through a request.” A cardholder does not recognize a $200 charge from an online electronics store and calls their bank. The issuing bank opens a dispute and sends a message through the card network to the merchant’s acquiring platform — our system.

Notice how many independent services are involved, and how each handoff is asynchronous. This matters because the total round trip — from dispute filing to final resolution — can take anywhere from a few days (if auto-resolved) to over 60 days (if it escalates to arbitration). A synchronous, request-response design simply cannot model a process that spans weeks; it has to be event-driven, with each service reacting to events and persisting its own durable state.

4.1 Evidence lifecycle in detail

The evidence step deserves its own walk-through because it is where most operational complexity lives. When a merchant is notified, they typically have between 7 and 20 days (network- and reason-code-dependent) to respond. The system needs to:

• Present the merchant with a checklist of exactly what evidence the specific reason code requires (proof of delivery looks different for “item not received” versus “item not as described”).
• Validate uploaded files for type, size, and completeness before accepting them, so a malformed PDF does not silently fail during representment three weeks later.
• Allow partial saves — a merchant might upload evidence over several sessions before final submission.
• Send reminder notifications as the deadline approaches (typically at the halfway point and 24–48 hours before expiry).
• Lock the case from further edits once submitted, to preserve a clean audit trail.

4.2 Reason codes drive everything

It is worth pausing on reason codes because they are the single most important piece of data in the entire system — almost every branching decision depends on them. A reason code is the card network’s classification of why the cardholder disputed the charge. Broadly, reason codes fall into a few families:

The Rules Engine is essentially a large, versioned decision table keyed on reason code, merchant category, transaction amount, and evidence availability. Because network rules change periodically (networks publish rule updates a few times a year), this decision table must be data-driven and hot-reloadable, not hardcoded into application logic that requires a redeploy to change.

4.3 The ledger’s view of the same timeline

It is worth separately tracing what happens to money during the same lifecycle, because the financial data flow does not move in perfect lockstep with the case’s workflow state. The moment a dispute is filed, the issuing bank typically debits the acquiring side immediately — this is a provisional debit, applied before anyone has determined who is actually right. The Ledger Service records this as a hold against the merchant’s balance, distinct from a final settlement. If the merchant later wins the case, the Ledger Service posts a compensating credit; if the merchant loses, the provisional debit simply becomes final, and an additional dispute fee is usually posted as a separate line item, since card networks and acquirers charge processing fees for handling a dispute regardless of outcome.

This two-track timeline — workflow state in the Case Management Service, financial state in the Ledger Service — is intentional. It means a merchant’s account balance always reflects the network’s actual movement of funds in near real time, even while the underlying case might still be sitting in a “pending evidence” state for another two weeks. Conflating these two into a single service would create a component that has to be simultaneously optimized for workflow flexibility and airtight financial correctness, which is a difficult combination to get right in one place.

4.4 A data-partitioning view of the same flow

It helps to see how a single dispute’s data physically spreads across the sharded store as it moves through its lifecycle, since this is a natural follow-up question in an interview about scaling.

5.1 Auto-resolution aggressiveness vs. merchant revenue protection

A key trade-off is between auto-resolution aggressiveness and merchant revenue protection. Auto-accepting every low-value dispute saves operational cost but silently gives away money the merchant might have won. Fighting every dispute protects revenue but burns engineering and support time on cases that were never winnable. Mature platforms tune this threshold using historical win-rate data per reason code and merchant category — effectively turning it into a data science problem layered on top of a workflow engine.

5.2 Consistency vs. latency in the case store

Another trade-off worth naming explicitly is consistency versus latency in the case store. A stricter, fully synchronous write path guarantees that every reader always sees the absolute latest state, but adds latency to every write and can reduce availability if any replica in the consistency group is slow or unreachable. A more relaxed approach, where reads can occasionally lag writes by a very small amount, improves latency and availability but forces careful thought about which specific operations can tolerate that lag and which absolutely cannot, such as the deadline sweeper’s core query, which must never operate on stale data that could cause it to miss an about-to-expire case.

5.3 Automation depth vs. human judgment

There is also a trade-off in how much intelligence to push into the Rules Engine versus how much to leave for human judgment. A highly automated system resolves cases faster and cheaper, but a rules engine that is too aggressive can systematically make the same mistake across thousands of cases before anyone notices, whereas a human reviewer, while slower, naturally catches unusual, one-off situations that a static rule set was never designed to anticipate. The healthiest systems treat automation as the default path for well-understood, high-confidence cases, while explicitly routing ambiguous or unusually high-value cases to a human review queue, rather than trying to automate every single case regardless of its confidence level.

A platform processing millions of card transactions per minute during peak events (holiday sales, major ticket releases) will see dispute volume spike proportionally, often with a lag of a few days to a few weeks after the transaction spike, since cardholders take time to notice and report issues. The system therefore needs to absorb bursty, delayed-correlation load rather than assume disputes arrive evenly.

6.1 Horizontal scaling of ingestion

The ingestion path should be stateless and horizontally scalable. Each incoming dispute file or webhook is independent, so ingestion workers can be scaled out behind a load balancer with no shared in-memory state. The event bus (commonly Kafka) absorbs bursts by buffering messages, decoupling the rate at which disputes arrive from the rate at which downstream services can process them.

6.2 Partitioning the case store

The dispute database is partitioned (sharded) by merchant ID or by a hash of the case ID, so that a spike in one merchant’s dispute volume does not create a hotspot that slows down every other merchant’s cases. Read-heavy operations, like a merchant checking their open cases, are served from read replicas or a cache layer, while writes to case state go through the primary shard owner to preserve strict consistency for financial state.

6.3 Batch vs. real-time trade-off

Some card networks still deliver dispute data as nightly batch files rather than real-time webhooks. The ingestion layer must support both without becoming two entirely separate systems: batch files are chunked and streamed onto the same internal event bus as real-time webhooks, so every downstream service only ever deals with one uniform stream, regardless of how the data physically arrived.

6.4 Connection pooling and backpressure

Even with a sharded, horizontally scaled store, an unbounded flood of writes can still overwhelm a single shard’s connection capacity during a burst. Every service maintains a bounded connection pool to the database rather than opening a new connection per request, and when the pool is exhausted, incoming work queues up in the message bus instead of piling up as open, half-finished database connections. This is a deliberate application of backpressure: it is far better for a burst of disputes to sit safely in a durable queue for a few extra minutes than for the database to be driven into a degraded state that slows down every other operation, including deadline checks for completely unrelated cases.

6.5 Batching for efficiency

Certain operations naturally batch well. Deadline reminder notifications, for instance, do not need to be sent the instant a case crosses a threshold; the sweeper can group all cases crossing a reminder threshold within the same few-minute window into a single batch write and a single batch of outbound notification jobs, dramatically reducing the number of individual database round trips compared to processing each case as a fully separate transaction.

6.6 Read/write ratio considerations

Dispute systems are read-heavy relative to writes: a single case might be written to a handful of times over its lifecycle (created, evidence submitted, resolved) but read dozens of times, by merchant dashboards, support agents, and internal risk tooling. This ratio justifies investing in read replicas and a caching layer ahead of over-optimizing the write path, though the write path still needs to be correct and durable even if it is not the primary scaling bottleneck.

6.7 Capacity math walkthrough

Concrete numbers help calibrate intuition. Suppose a platform processes $30$ million transactions a day with a dispute rate of $0.5%$, that gives roughly $150{,}000$ new disputes a day, or an average of about $1.7$ new disputes per second. Peak factors of $10times$ to $20times$ around post-breach or post-holiday windows push the ingestion path to a design target closer to $30$ to $50$ disputes per second sustained — small in absolute terms, but every one of those cases then lives in the store for weeks, so the open-case working set can easily exceed $2$ million rows at any given moment, and it is that number, not the arrival rate, that dominates deadline-sweeper query cost.

Because missed deadlines cause automatic financial losses for merchants, the deadline-tracking and notification path is arguably the single most reliability-critical component in the whole system — more so even than the ingestion path, because a delayed ingestion can often still catch up, while a missed deadline is frequently unrecoverable.

7.1 Multi-region deployment

The platform typically runs across multiple availability zones within a region, and often across multiple regions for disaster recovery. The case database uses synchronous replication within a region for strong consistency, and asynchronous replication cross-region for disaster recovery, accepting a small recovery-point gap in exchange for avoiding cross-region write latency on every transaction.

7.2 Graceful degradation

If the Rules Engine becomes unavailable, the system should not simply stop processing disputes — it should fail safe by routing all incoming cases to manual review rather than blocking ingestion entirely. Similarly, if the Notification Service is degraded, the Case Management Service should still record the state transition and queue the notification for retry, rather than letting a downstream outage corrupt the state machine itself.

7.3 Deadline redundancy

Given how costly a missed deadline is, production systems typically run the deadline sweeper with redundancy: multiple independent processes evaluate upcoming deadlines, with distributed locks (or a leader-election mechanism) ensuring only one process actually fires the action, while the others act purely as a safety net that would kick in if the primary sweeper stalls.

public void tick() {
    if (!leaderLock.isHeldByThisInstance("dispute-deadline-sweeper")) {
        return; // standby - do nothing this tick
    }
    List<Case> imminent = caseRepository.findOpenCasesDueBefore(
        Instant.now().plus(SWEEP_WINDOW));
    for (Case c : imminent) {
        // Optimistic concurrency guard - transitionIfVersion returns false
        // if evidence submission raced ahead and already advanced the case.
        boolean advanced = caseService.transitionIfVersion(
            c.getId(), c.getVersion(), CaseState.DEADLINE_MISSED);
        if (advanced) {
            outbox.publish(new DeadlineMissedEvent(c.getId()));
        }
    }
}

7.4 Backup and disaster recovery

Beyond live replication, the system takes regular point-in-time snapshots of the case and ledger stores, retained long enough to satisfy both operational recovery needs and regulatory record-keeping requirements, which for financial dispute records can extend to several years. Recovery drills are run on a schedule, not just documented in a runbook, because a disaster recovery plan that has never actually been executed tends to fail in exactly the moment it is needed, when assumptions about tooling or access turn out to be stale.

7.5 Chaos engineering for deadlines specifically

Because the deadline-tracking path is uniquely unforgiving of downtime, mature teams intentionally inject failure into staging or shadow environments — killing a sweeper mid-run, partitioning it from the database, delaying the message bus — and verify that no deadline is ever silently dropped as a result, only ever delayed and still eventually and correctly processed. This targeted chaos testing catches subtle bugs, like an in-progress deadline check that is not properly resumed after a crash, that would otherwise only surface in production during a real incident.

7.6 Defining “available” for this system

Availability for a dispute platform is not simply “can the API respond to a health check.” A more meaningful definition is “can every case that has a deadline in the next N minutes still be correctly evaluated and acted upon.” A system can technically be “up” by infrastructure metrics while still silently failing its actual reliability promise if, for example, the deadline sweeper’s database queries are timing out under load. Defining availability around the business outcome, not just infrastructure uptime, is what separates a system that merely looks healthy on a dashboard from one that is actually protecting merchants from missed deadlines.

This system sits at the intersection of two highly sensitive domains: payment data and legal evidence. It must comply with PCI DSS (Payment Card Industry Data Security Standard) for anything touching card data, and it must protect evidence files that can include personal information about both cardholders and merchant customers.

8.1 Data minimization and tokenization

The dispute system rarely needs the full card number (PAN). Instead, it works with a tokenized reference to the original transaction, resolving the token back to sensitive data only within a tightly scoped, audited service when absolutely necessary (for example, to verify AVS/CVV match results as fraud evidence). This limits the “blast radius” if any one service is compromised.

8.2 Access control on evidence

Evidence files can contain a customer’s home address, signature, or partial financial details. Access to the Evidence Service and its object storage must be governed by strict role-based access control, with every read logged, since evidence is effectively legal-grade material that may later be reviewed by auditors, card networks, or in rare cases, courts.

8.3 Authentication between parties

Communication with card networks typically uses mutual TLS and signed messages, since a forged dispute message could trigger an incorrect fund reversal. Merchant-facing APIs use OAuth2 with scoped tokens, so a compromised merchant integration key cannot be used to view or act on another merchant’s disputes.

8.4 Key management

The master keys used in the envelope encryption scheme described below are managed by a dedicated key management service with strict separation of duties: no single engineer can both approve a key rotation and independently execute it without a second authorized approver, and every key operation is itself logged as an auditable event. Keys are rotated on a fixed schedule as well as immediately upon any suspected compromise, and old key versions are retained only long enough to decrypt legitimately old evidence, never used for encrypting anything new.

8.5 Fraud on the dispute process itself

An often-overlooked risk is friendly fraud — a cardholder who did receive the goods but disputes the charge anyway to get a free refund. The system needs to support fraud-signal correlation (device fingerprinting, delivery confirmation, past dispute history for that cardholder) to help merchants build a stronger evidence case against pattern abusers, rather than assuming every dispute is a legitimate mistake.

8.6 Encryption at rest and in transit

All evidence files and dispute records are encrypted at rest using envelope encryption, where a per-file data key encrypts the actual content and is itself encrypted by a master key managed in a dedicated key management service. This layered approach means rotating the master key does not require re-encrypting every historical evidence file, which matters enormously at the scale of millions of retained cases. Every network hop, whether between internal services or across the public internet to a merchant or card network, is encrypted in transit using TLS, with certificate rotation automated rather than manually tracked, since an expired certificate on an internal service can silently take down an entire processing path.

8.7 Compliance considerations

Beyond PCI DSS, dispute records often fall under broader data protection regulations because they contain personal information about cardholders who are not the platform’s direct customers. This means the system needs clear data retention and deletion policies, and needs to support data subject access or deletion requests without breaking the immutable audit trail required for financial compliance — typically solved by retaining a redacted or tokenized version of records for audit purposes while purging directly identifying fields once legally permissible.

Because so much of this system’s correctness is about timing, observability has to be built around deadlines and stage transitions, not just generic request latency.

9.1 Key metrics

9.2 Distributed tracing

Since a single dispute touches half a dozen services over potentially weeks, correlating logs by a simple request ID is not enough — the system needs a persistent case ID that is stamped on every log line, event, and trace span from creation to closure, so an engineer can reconstruct the full timeline of a specific case instantly when a merchant disputes the platform’s own handling of their dispute.

9.3 Alerting philosophy

Alerts should be tiered by financial impact. A backlog in the analytics pipeline is a low-priority alert; a stalled deadline sweeper is a page-the-on-call-engineer-immediately alert, because every minute of delay there can translate directly into missed deadlines and real monetary loss for merchants.

• P1 (page immediately): deadline sweeper stalled, ledger write errors, representment submission failures approaching the network cutoff.
• P2 (urgent, business hours): single card-network webhook feed degraded, elevated auto-resolution error rate.
• P3 (informational): minor ingestion lag within SLA, analytics pipeline lag.

9.4 Service level objectives and error budgets

Rather than chasing 100 percent uptime, which is both unachievable and not the right goal, the platform defines explicit service level objectives for the components that matter most: for example, “99.99 percent of deadline evaluations complete within one minute of the deadline passing.” An error budget derived from this objective gives engineering teams a data-driven way to balance shipping new features against tightening reliability — if the budget is being consumed too fast, deployment velocity is deliberately slowed until reliability work catches up, rather than that trade-off being made informally or too late.

9.5 Health checks that reflect reality

A shallow health check that only confirms a service process is running can report “healthy” even while that service is failing every real request due to a misconfigured downstream dependency. Deeper health checks instead verify that a service can actually complete a representative operation — for the Case Management Service, this might mean confirming it can read and write a synthetic test case end to end — so that load balancers and orchestration systems route traffic away from an instance that is technically alive but functionally broken.

9.6 Structured logging

Every log line is emitted as structured data, not free-form text, with consistent fields for case ID, merchant ID, reason code, and current state. This is what makes the earlier-described tracing by case ID actually practical at scale — a structured query across the log store can pull every event for one case in milliseconds, whereas free-text log searching over months of high-volume logs would be far too slow to be useful during an active incident.

Each service in this architecture is independently deployable, packaged as a container and orchestrated by a system like Kubernetes, which allows the ingestion path (often bursty) to scale independently from the case management path (steadier, but latency sensitive) and the evidence/object-storage path (large payloads, different resource profile).

10.1 Progressive delivery

Because a bug in the Rules Engine could incorrectly auto-accept or auto-fight thousands of live cases, deployments to the rules and case-management services use canary releases — routing a small percentage of new cases to the new version first, comparing outcomes against the previous version, and only rolling forward once confidence is established. A blue-green approach is often layered on top for the ability to instantly roll back the entire fleet if something is wrong.

10.2 Configuration as data

Reason-code policy tables, deadline durations per network, and evidence checklists are treated as versioned configuration data stored separately from application code and deployed through their own reviewed, audited pipeline. This lets compliance or risk teams update policy in response to a network rule change without waiting for an engineering release cycle.

10.3 Infrastructure as code

The full topology — service definitions, network policies, database clusters, scaling rules — is defined declaratively and version-controlled, rather than manually configured through a cloud console. This gives the platform two important properties: the exact infrastructure state at any point in history can be reconstructed for an audit, and a full regional failover environment can be reliably recreated from the same definitions used for the primary region, avoiding the common failure mode where a disaster recovery environment has silently drifted out of sync with production over time.

10.4 Cost optimization

Object storage holding years of evidence files can become a significant, steadily growing cost line if left unmanaged. A tiered storage lifecycle policy automatically moves older, closed-case evidence to cheaper, lower-access-frequency storage classes after a defined window, while keeping recently active cases on fast storage, striking a balance between cost and the occasional need to retrieve older evidence for an audit or a reopened arbitration case.

10.5 Environment isolation

Staging and sandbox environments use synthetic or heavily anonymized dispute data rather than real merchant and cardholder information, since evidence files in particular can carry sensitive personal data that has no legitimate reason to exist outside the production environment’s tightly controlled access boundary.

11.1 Choosing the database for each store

11.2 Where NoSQL fits

Not everything needs relational guarantees. Evidence metadata, audit logs, and analytics aggregates are good fits for a wide-column or document store, since they are written once, read frequently, and do not need multi-row transactions. This is a classic polyglot persistence approach — using the right storage engine for each access pattern rather than forcing one database to do everything.

11.3 Caching strategy

Merchant dashboards frequently query “all my open cases” and “cases nearing deadline.” These read patterns are cached aggressively with short time-to-live values, since a few seconds of staleness on a dashboard view is acceptable, but the underlying write path for state transitions always bypasses the cache and goes straight to the primary store. Cache invalidation is event-driven — whenever a case transitions state, an event fires that invalidates the relevant cache entries, rather than relying purely on time-based expiry.

11.4 Load balancing

• Layer 4 (network) load balancing at the outermost edge for raw connection distribution across regions.
• Layer 7 (application) load balancing at the API Gateway, routing merchant traffic and network webhooks separately using health-check-aware routing so traffic never lands on an instance that is mid-restart or unhealthy.
• Client-side load balancing between internal services via a service mesh so that a single overloaded Rules Engine instance does not become a slow point for every case evaluation in the system.

11.5 Replication strategy in detail

Within a region, the primary shard for each partition replicates synchronously to at least one standby, so a failover promotes a standby with zero data loss for already-acknowledged writes. Across regions, replication is asynchronous, trading a small, bounded replication lag for the ability to avoid adding cross-region network latency to every single write, which would otherwise slow down every case transition in the system, including ones with no cross-region relevance at all.

11.6 Handling hot merchants

Occasionally, a single merchant’s dispute volume grows so large that even its own shard becomes a bottleneck relative to every other merchant sharing that shard. Production systems handle this with a secondary, finer-grained splitting strategy reserved for these outlier merchants: instead of trying to re-shard the entire system, a single oversized merchant can be given a dedicated shard or partition of its own, isolating its load without disrupting the simpler, uniform hashing scheme used for the overwhelming majority of merchants who never approach that scale.

11.7 Indexing for the access patterns that matter

The dispute store is indexed around the actual questions the system needs to answer quickly: “all open cases for merchant X,” “all cases with a deadline before time Y,” and “the full history of case Z.” These translate into composite indexes on merchant ID plus state, and on deadline timestamp plus state, so the deadline sweeper’s core query — find everything due soon that has not yet been actioned — remains fast even as the total number of historical cases grows into the tens of millions, most of which are closed and no longer relevant to that particular query.

The system exposes at least two very different API surfaces, and conflating them is a common design mistake. The network-facing API speaks in the rigid, standardized language of card network protocols and file formats, versioned and changed rarely, with heavy validation and strict SLAs. The merchant-facing API is a modern REST or GraphQL interface, designed for developer ergonomics, with webhooks for real-time case updates so merchants can build their own automated dispute-response tools rather than relying solely on a dashboard.

12.1 Core API contract

GET /v1/disputes/8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11
Response: {
  "caseId": "8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11",
  "merchantId": "acct_abc123",
  "state": "PENDING_EVIDENCE",
  "reasonCode": "13.1",
  "disputedAmount": { "value": 200.00, "currency": "USD" },
  "evidenceDueAt": "2026-08-18T23:59:00Z",
  "evidenceChecklist": ["shipping_proof","tracking_number","customer_comms"],
  "originalTransactionId": "txn_9f8a...",
  "version": 3
}

POST /v1/disputes/8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11/evidence
Headers:  Idempotency-Key: <merchant-generated-uuid>
Request:  {
  "expectedVersion": 3,
  "files": [
    { "type": "shipping_proof", "objectRef": "obj://ev/2026/08/abc.pdf" },
    { "type": "tracking_number", "value": "1Z9999W99999999999" }
  ],
  "notes": "Signed for by cardholder on 2026-08-05"
}
Response: {
  "caseId": "8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11",
  "state": "EVIDENCE_SUBMITTED",
  "acceptedAt": "2026-08-08T14:20:11Z",
  "version": 4
}

12.2 Service boundaries

Each microservice described in the architecture section owns its own data and exposes a narrow, well-defined interface. The Case Management Service, for example, never directly writes to the Ledger’s tables — it publishes a “case resolved, release funds” event, and the Ledger Service is solely responsible for its own consistency guarantees. This separation means a bug or slowdown in analytics can never accidentally corrupt financial ledger state, because there is no shared mutable data between them.

12.3 Webhooks and reliability

Merchant-facing webhooks must be designed assuming the merchant’s endpoint will sometimes be down. The Notification Service retries with exponential backoff, and merchants can always reconcile via a pull-based API as a fallback, so a temporarily unreachable webhook endpoint never causes permanent data loss on the merchant’s side.

12.4 Rate limiting and fairness

The merchant-facing API applies rate limits per merchant, not just globally, so a single merchant running an aggressive polling loop or a misbehaving integration cannot degrade response times for every other merchant sharing the same gateway fleet. Rate limit thresholds are tiered by merchant size and contract, and limit responses include clear guidance on retry timing, encouraging well-behaved backoff rather than tighter retry loops that would make the problem worse.

12.5 Versioning strategy

The merchant-facing API is explicitly versioned, and old versions are supported for a defined deprecation window rather than broken without notice, since merchants build real production systems against this API and an unannounced breaking change would directly damage their ability to respond to time-sensitive disputes. The network-facing API, by contrast, follows whatever versioning cadence each card network itself dictates, and the Ingestion Service’s anti-corruption layer is exactly what absorbs that external versioning churn without forcing a matching version bump on the merchant-facing side.

13.1 Patterns that fit well

13.2 Anti-patterns to avoid

The Saga pattern deserves a closer look because it directly answers a very common interview question: “how do you handle a multi-step process across services without a giant distributed transaction?” Instead of trying to lock the Case, Ledger, and Notification services together in one atomic operation, each step publishes an event when it completes, and the next service reacts to that event. If a later step fails, a compensating action (like reversing a provisional debit) is triggered rather than trying to roll back a transaction that was never atomic to begin with.

13.3 The Outbox pattern, explained simply

A subtle bug that trips up many first attempts at this kind of system is the gap between “we saved the case state change to the database” and “we published the event telling other services about it.” If these are two separate operations, a crash between them leaves the database correct but the rest of the system unaware that anything happened. The outbox pattern solves this by writing the event to an “outbox” table in the very same database transaction as the state change itself, so both succeed or both fail together, atomically. A separate, simple background process then reads new outbox rows and publishes them to the event bus, retrying safely if needed, since the source of truth (the outbox table) is already durably committed.

Best practices in a system like this tend to come from painful experience rather than theory, because the failure modes are rarely dramatic outages — they are quiet, compounding drifts that only become visible weeks later in a financial reconciliation report. The practices below are the ones that consistently separate platforms with a healthy dispute operation from those constantly firefighting avoidable losses.

14.1 Best practices

• Always design the deadline as the source of truth, and treat merchant reminders as a side effect of it, not the other way around.
• Version policy and reason-code mappings, and log which version was applied to every case, so a policy bug can be traced back precisely.
• Build reconciliation jobs that periodically compare internal case state against the card network’s own records, since network-side systems occasionally desync from any acquirer’s view.
• Design for partial merchant responses; do not force an all-or-nothing evidence submission experience.
• Keep the ledger append-only; never overwrite a financial record, only add compensating entries.
• Treat the merchant evidence UX as a product surface, not an internal tool — evidence quality directly determines financial outcomes.

14.2 Common mistakes

• Assuming dispute volume scales linearly and evenly with transaction volume, when in reality it is bursty and delayed by days or weeks.
• Under-investing in the merchant evidence UX, leading to low-quality evidence submissions and avoidable losses.
• Not building alerting around “cases with no activity approaching deadline,” which silently bleeds money until someone notices in a monthly report.
• Coupling the internal case ID to any card network’s dispute reference format, making a future network integration painful.
• Optimizing the automated rules engine purely for cost savings without periodically auditing its decisions against actual outcomes, letting a subtly wrong policy compound quietly for months.
• Treating the merchant evidence portal as an afterthought UI project rather than a core product surface, even though it directly determines how much revenue a merchant recovers.

One useful discipline that catches many of these mistakes early is a regular “loss review,” where a sample of lost cases is manually re-examined, independent of the automated pipeline, to check whether the loss was truly unavoidable or whether a process gap — a missing evidence type, a slow notification, a misconfigured rule — contributed to it. Over time, this review process becomes one of the richest sources of product and engineering improvements, because it directly surfaces the gap between how the system is designed to behave and how it actually behaves for real merchants.

14.3 Operational playbooks

Because deadline risk is the platform’s sharpest edge, mature operations teams maintain a clear, tested playbook for the specific failure mode of “the deadline sweeper has been unhealthy for N minutes,” including a manual fallback query that on-call engineers can run immediately to identify at-risk cases and notify affected merchants directly while the automated path is being restored. Having this playbook written and rehearsed before an incident, rather than improvised during one, consistently makes the difference between a contained delay and cases lost purely due to internal tooling failure rather than any merchant or cardholder action.

14.4 Testing strategy

Given how consequential state transitions are, the state machine itself is covered by exhaustive property-based tests that assert invariants like “a case can never move from Closed back to any open state” and “every path into Lost is either an explicit network decision or an expired deadline, never anything else,” rather than relying solely on example-based unit tests that only check a handful of hand-picked scenarios.

Large payment processors and acquiring banks all operate systems that closely resemble what we have described here, even though implementation details differ. The examples below show how the same patterns manifest across different business models.

A broader lesson emerges from these examples: the best dispute defense is frequently built upstream of the dispute system entirely, in how clearly the original transaction and any cancellation or delivery event was recorded in the first place. A subscription platform’s crisp cancellation confirmation, an e-commerce platform’s signed delivery record, an airline’s clear itinerary-change communication — each of these becomes the single most compelling piece of evidence long before the Rules Engine ever needs to evaluate a case.

If you remember nothing else from this tutorial, remember this: in most systems, losing a request means retrying it. In a dispute system, losing track of a deadline means losing real money for a real merchant, permanently. That single difference is what shapes almost every architectural choice described above — from durable scheduling, to redundant deadline sweepers, to fail-safe defaults when a downstream service degrades. Build for the deadline first, and the rest of the architecture follows naturally.

17.1 Key takeaways to carry into an interview

• Model the dispute as a durable, event-sourced state machine, not a mutable row that gets overwritten in place.
• Treat deadlines as first-class data with their own redundancy and monitoring, since they are the system’s single biggest source of irreversible financial risk.
• Separate policy (reason-code rules) from mechanism (the state machine engine) so that policy can change without a code deployment.
• Isolate the ledger’s strong consistency needs from the more relaxed consistency acceptable for analytics and reporting.
• Design the merchant evidence experience as carefully as any consumer-facing product, since evidence quality directly determines financial outcomes.
• Assume drift will happen between your internal state and the card network’s state, and build reconciliation in from day one rather than bolting it on after the first costly mismatch.
• Every operation must be idempotent — networks retry, merchants double-click, and mobile networks drop connections, so the same action must be safe to apply more than once.

17.2 The one idea to remember

Taken together, these principles describe a system that looks, on the surface, like a workflow tool, but underneath is a careful balance of distributed systems fundamentals — consistency, idempotency, durable scheduling, and graceful degradation — all in service of a very human goal: making sure that when something goes wrong with a payment, both the cardholder and the merchant get a fair, timely, and correctly recorded resolution.