Why one of the most feared bugs in all of software engineering — the accidental double charge — is really a very old problem in distributed systems wearing a new outfit.

Picture yourself paying for something online. You tap “Pay Now,” your phone’s internet flickers for a moment, and the screen just spins. Frustrated, you tap “Pay Now” again. A few seconds later, both attempts go through, and you check your bank statement to find you have been charged twice for the same order. This is one of the most feared bugs in all of software engineering, because it involves real money, real customer trust, and often, real legal and regulatory consequences.

The root of this problem is something every network engineer learns early: a network request can fail in a way where you, the sender, genuinely cannot tell whether the other side actually received and processed it, or not. The response telling you “yes, it worked” might have been lost on the way back to you, even though the payment itself went through perfectly on the server. This uncertain middle ground — called “at least once versus exactly once” delivery — has existed since the earliest distributed computing systems, long before the internet or online shopping existed.

Early payment systems in the 1970s and 1980s, built for bank mainframes and point-of-sale terminals, already grappled with this exact issue, using techniques like unique transaction reference numbers and manual reconciliation reports run overnight. As online payments exploded in the 2000s with the rise of e-commerce, and then again in the 2010s with mobile payments and instant transfers, the same fundamental problem returned at a vastly larger scale, and the industry converged on a now-standard solution: the idempotency key. This document walks through, piece by piece, how to design a payment system that uses this idea, and the surrounding architecture, to guarantee a transaction is processed exactly once, no matter how many times a client retries, even under the load of millions of requests per minute.

1.1 Why This Problem Never Fully Goes Away

It is tempting to think this is solvable once and forgotten, but the truth is that network timeouts are a permanent, unavoidable fact of distributed systems, not a bug to be fixed. Mobile networks drop packets, data centres experience brief partition events, load balancers occasionally kill slow connections, and client apps have their own retry logic that engineers do not always control. A well-designed payment system does not try to prevent timeouts from happening, since that is impossible; instead, it is designed so that a retry after a timeout is always, provably, safe.

1.2 A Very Short History of the Idempotency Key

Why “just retry on timeout” is one of the most quietly dangerous defaults in backend engineering, and why the fix has to be deliberate.

Let us break down precisely why “just retry on timeout” is dangerous without extra design, and why this is genuinely one of the harder problems in backend engineering.

2.1 The Core Problem

• Ambiguous timeouts: When a client does not receive a response within its timeout window, it has no way to know whether the server never received the request, received it but has not finished processing, or finished processing successfully and the response was simply lost on the way back.
• Client retries are necessary and expected: Mobile apps, browsers, and payment SDKs are built to retry on timeout, because most of the time a timeout really does mean the request was lost, and refusing to retry would make the product feel broken on flaky networks.
• Double charging is unacceptable: Unlike most retryable operations, charging a customer’s card twice for one order causes direct financial harm, damages trust, and can trigger chargebacks, regulatory scrutiny, and support costs far larger than the value of engineering effort saved by skipping this design work.
• Massive concurrent volume: At scale, millions of payment requests per minute means millions of opportunities for a network hiccup, so even a tiny per-request probability of a timeout translates into a large absolute number of retries every single minute.
• Multiple systems in the chain: A single payment touches the marketplace’s own services, an external payment gateway, and often a card network and issuing bank behind that, and a retry can occur at any layer of this chain, not just between the customer’s device and the marketplace.

2.2 Why Naive Approaches Fail

The only robust solution is to give every payment attempt an identity — the idempotency key — generated once by the client and reused on every retry of that same logical attempt, so the server can definitively recognise a retry as a retry, not as a new request.

2.3 The Scale of the Challenge in Numbers

A target of one million requests per minute works out to roughly 16,700 requests per second sustained, with realistic peaks during a major sale event reaching 50,000 to 80,000 requests per second for short bursts. Even a very small, well-engineered network failure rate — say one in every one thousand requests genuinely timing out on the way back to the client — would still mean 50 to 80 retries every single second at peak, each one requiring the system to correctly recognise it as a duplicate rather than processing it as new.

Now consider a worse, but entirely realistic, scenario: a brief regional network disruption, perhaps a mobile carrier outage affecting a large city, causing a much larger spike in timeouts across thousands of concurrent customers within the same few seconds. Without the idempotency guarantees described in this document, this single external event could translate directly into thousands of duplicate charges within moments — each one a real customer complaint, chargeback, and support ticket. This is precisely the kind of correlated failure scenario that makes idempotency not an optional refinement but a mandatory, load-bearing part of the architecture for any payment system operating at this scale.

A layer-by-layer walkthrough of the system, from the customer’s device all the way to the external bank, with every box explicitly labelled by what kind of component it is.

3.1 Component-by-Component Explanation

3.2 Why a Dedicated Idempotency Store Instead of Just Logging Requests

A reasonable early question is why we need a dedicated Redis-based Idempotency Key Store at all, rather than simply logging every incoming request and checking the log before processing a new one. The problem is speed and atomicity: a traditional log is designed for durability and sequential writing, not for the kind of instantaneous, atomic “claim this key only if nobody else has” check that must happen on the hot path of every single payment request, often within a few milliseconds, and often with many concurrent requests racing to claim the same key at nearly the same instant. Redis, with its single-threaded command execution model and native atomic operations, is purpose-built for exactly this kind of check, which is why it sits directly in the critical path rather than a general-purpose log or the main ledger database itself, which is comparatively much slower to query at this frequency and this level of concurrency.

Zooming in on exactly what happens inside the system when a client’s request times out and it retries — the moment of truth for every guarantee we’re promising.

4.1 The Atomic Claim Operation

The very first thing the Payment Orchestrator does with an incoming request is attempt to atomically claim the idempotency key in Redis, using a command such as SET key value NX, which only succeeds if the key does not already exist. This single atomic operation is what safely handles the case where two retries of the same request arrive at almost the exact same millisecond — perhaps because a client’s retry logic fired just as the original request’s slow response was finally arriving: only one of the two concurrent attempts will win the claim, and the other will immediately see that the key is already taken.

public class IdempotencyKeyStore {

    private final RedisTemplate<String, String> redisTemplate;
    private static final Duration LOCK_TTL = Duration.ofMinutes(10);

    public IdempotencyKeyStore(RedisTemplate<String, String> redisTemplate) {
        this.redisTemplate = redisTemplate;
    }

    public ClaimResult claim(String idempotencyKey) {
        Boolean claimed = redisTemplate.opsForValue()
            .setIfAbsent(key(idempotencyKey), "IN_PROGRESS", LOCK_TTL);

        if (Boolean.TRUE.equals(claimed)) {
            return ClaimResult.NEWLY_CLAIMED;
        }

        String status = redisTemplate.opsForValue().get(key(idempotencyKey));
        if ("IN_PROGRESS".equals(status)) {
            return ClaimResult.ALREADY_IN_PROGRESS;
        }
        return ClaimResult.ALREADY_COMPLETED;
    }

    private String key(String idempotencyKey) {
        return "idempotency:" + idempotencyKey;
    }
}

4.2 Handling the “Already In Progress” Case

A tricky, important case is when a retry arrives while the original attempt is still actively being processed, not yet finished. Simply rejecting the retry outright would be wrong, since the client legitimately needs an answer. Instead, the Orchestrator holds the retry request briefly — polling the key’s status for a short window, typically a few seconds — and once the original attempt completes, returns that same final result to the retry, rather than starting a second, parallel execution.

4.3 The Database as the Final Safety Net

Even with the Redis-based claim working correctly, good engineering practice never relies on a single layer of protection for something this important. The Ledger Database table has a unique constraint on the idempotency key column, so even in the rare case of a Redis failure, a bug, or a claimed key expiring too early, a genuine duplicate insert attempt will be rejected by the database itself with a constraint violation — which the Payment Execution Service catches and treats as “this was already processed,” rather than as an error.

CREATE TABLE payment_transactions (
    id BIGSERIAL PRIMARY KEY,
    idempotency_key VARCHAR(64) NOT NULL UNIQUE,
    customer_id     BIGINT      NOT NULL,
    amount_cents    BIGINT      NOT NULL,
    currency        CHAR(3)     NOT NULL,
    status          VARCHAR(20) NOT NULL,
    created_at      TIMESTAMP   NOT NULL DEFAULT now(),
    updated_at      TIMESTAMP   NOT NULL DEFAULT now()
);

public class PaymentExecutionService {

    private final PaymentTransactionRepository repository;
    private final PaymentGatewayAdapter gatewayAdapter;

    public PaymentResult execute(PaymentRequest request) {
        try {
            repository.insertPending(request.getIdempotencyKey(),
                request.getCustomerId(), request.getAmountCents(),
                request.getCurrency());
        } catch (DuplicateKeyException e) {
            // Another concurrent attempt already inserted this row
            return repository.findByIdempotencyKey(request.getIdempotencyKey())
                .toResult();
        }

        GatewayResponse response = gatewayAdapter.charge(request);
        repository.updateStatus(request.getIdempotencyKey(), response.getStatus());
        return new PaymentResult(response.getStatus(), response.getTransactionId());
    }
}

4.4 Choosing a Safe Lock Time-to-Live

The time-to-live set on a freshly claimed, still-in-progress idempotency key deserves careful thought. If it is set too short, a slow but otherwise legitimate payment attempt — perhaps delayed by a genuinely slow external gateway response — could have its lock expire while still processing, opening a dangerous window where a retry might slip through and start a second, parallel execution. If it is set too long, a request that crashed or failed in an unusual way partway through, without ever reaching a final state, could leave a key stuck as falsely IN_PROGRESS for an unnecessarily long time, blocking any legitimate retry from proceeding.

A common, well-tested approach uses a moderate lock time-to-live, generously longer than the ninety-ninth percentile expected processing time, combined with a background sweep process that detects and safely resolves any key that has remained IN_PROGRESS far longer than should ever legitimately happen — treating it as a signal to trigger the same reconciliation path used for the UNKNOWN gateway state.

The complete life of a payment attempt — from the customer’s tap, through a network timeout and a retry, to a final, safely-once confirmation.

1. Key Generation: The client generates a unique idempotency key once, at the moment the customer taps “Pay,” and stores it locally so it survives across retries of that same logical payment attempt.
2. Initial Request: The request, carrying the idempotency key in a header, travels through the CDN, Global Load Balancer, API Gateway, and Regional Load Balancer to a Payment Orchestrator instance.
3. Atomic Claim: The Orchestrator attempts to atomically claim the key in Redis. If successful, this is a genuinely new attempt, and processing continues.
4. Risk Scoring: The Fraud and Risk Service evaluates the transaction, itself keyed by the idempotency key so it is not scored twice on a retry.
5. Ledger Insert: The Payment Execution Service inserts a PENDING row into the Ledger Database, protected by the unique constraint on the idempotency key.
6. External Charge: The Payment Gateway Adapter calls the external payment processor, forwarding the idempotency key so the external system has its own duplicate protection too.
7. Network Timeout: Suppose the response from either our own system or the external gateway is lost in transit back to the client. The client’s timeout fires, and it has no confirmation.
8. Client Retry: The client automatically retries, sending the identical request with the identical idempotency key.
9. Fast Lookup, Not Re-Execution: The Orchestrator checks the key’s status in Redis, finds it already COMPLETED (or still IN_PROGRESS, handled as described earlier), and returns the already-computed final result directly, without re-running any payment logic or contacting the external gateway again.
10. Asynchronous Side Effects: In parallel, once the transaction reaches a final state, an event flows through Kafka to trigger order fulfilment and a customer notification — exactly once, regardless of how many times the client retried the underlying request.
11. Reconciliation Safety Net: Periodically, the Reconciliation Service compares our Ledger Database against the external gateway’s own transaction records, catching and resolving any transactions left in an UNKNOWN state due to a timeout on the gateway call itself.

Every design choice in this system pays for something and costs something else — here they are, side-by-side, so nothing is smuggled in.

Millions of requests per minute, tens of thousands per second at peak, a heavy external dependency — the numbers force every layer to be scaled and shaped deliberately.

The target for this system is millions of requests per minute — roughly 16,700 requests per second sustained, with peaks potentially reaching 50,000 to 80,000 requests per second during high-traffic events such as a major sale. Let us look at how this design holds up.

7.1 Horizontal Scaling of Stateless Services

The Payment Orchestrator, Fraud and Risk Service, and Payment Execution Service are all stateless, holding no important data in local memory between requests. This means we can run as many replicas as needed behind the Regional Load Balancer, scaling horizontally with an auto-scaler that watches CPU usage and request queue depth — exactly the same core scaling strategy used across almost every high-throughput backend system.

7.2 The Idempotency Store as a Critical Hot Path

Every single payment request, whether new or a retry, must pass through the Redis Idempotency Key Store, making it one of the most heavily hit components in the entire system. It is deployed as a sharded, replicated cluster, splitting the key space across many nodes so no single node becomes a bottleneck, with keys automatically expiring after a reasonable window (typically ten minutes to a few hours) since a key only needs to remain useful for as long as a client might realistically still be retrying.

7.3 Keeping the Common Path Fast

The overwhelming majority of requests are genuinely new attempts, not retries, so the atomic claim in Redis succeeds immediately and processing proceeds without any extra delay. For the smaller fraction that are retries of an already-completed transaction, the Orchestrator returns the stored final response directly from Redis, which is dramatically faster than re-running the full payment workflow — meaning retries are actually cheaper for the system to handle than fresh requests.

7.4 Connection Pooling to the External Gateway

Calls to the external Payment Gateway Adapter are the slowest step in the entire pipeline, often 200 to 600 milliseconds, and external gateways impose their own rate limits. The Payment Gateway Adapter maintains carefully sized connection pools and applies its own internal queuing and backpressure, so a burst of incoming traffic does not simply forward an unmanageable flood of concurrent requests to the external gateway, which could get the marketplace’s integration throttled or temporarily blocked.

7.5 Capacity Planning With Real Numbers

At a peak of 80,000 requests per second, if a single Payment Execution Service instance can safely handle around 500 requests per second — accounting for the relatively heavier work of a database insert and an external call compared to a simple read — we would need roughly 160 instances at peak, scaled up gradually as load climbs rather than provisioned permanently. The Redis Idempotency Key Store, holding keys for a bounded recent window (perhaps 50 million active keys at any given moment, each a small entry of well under a kilobyte), comfortably fits within a modestly sized cluster with room to spare. The Ledger Database’s write throughput, since every request results in at most one insert, is the primary scaling constraint on that layer, which is why the ledger is typically sharded by customer ID or transaction ID range across multiple database clusters once a single cluster’s write capacity is approached.

7.6 Backpressure Toward the External Gateway

It is important to recognise that our own system can often scale its internal compute layer far more elastically than the external payment gateway can absorb sudden bursts of traffic. If our Payment Orchestrator fleet auto-scales up to handle 80,000 requests per second internally, but the external gateway’s contractually agreed rate limit is only 20,000 requests per second, blindly forwarding every request as fast as it arrives would simply overwhelm the gateway and cause a cascade of failures that then, ironically, generate exactly the kind of client-side timeouts and retries this whole system is built to survive.

The Payment Gateway Adapter therefore applies explicit backpressure, using a bounded queue and controlled concurrency limits that respect the external gateway’s known capacity, returning a clear, honest “please retry shortly” response to requests that cannot yet be forwarded, rather than accepting unlimited work it cannot actually deliver on.

A payment system that is frequently unavailable is a business emergency, not just an inconvenience — so redundancy and safe failure handling are designed into every layer.

8.1 Redundancy at Every Layer

Every service runs multiple replicas spread across multiple availability zones, and the Redis Idempotency Key Store and Ledger Database both run in clustered, replicated configurations, so the loss of any single machine, rack, or even an entire data centre does not take the payment system down.

8.2 Handling Ambiguous External Gateway Failures

The hardest reliability case in this entire system is a timeout on the call to the external payment gateway itself. Blindly retrying that call risks the exact duplicate charge this design exists to prevent, since we cannot be certain the first attempt did not already succeed on the gateway’s side. Instead, the transaction is marked UNKNOWN, and the Payment Gateway Adapter uses the gateway’s own transaction status lookup API (if available, using the same idempotency key) to definitively determine the true outcome before either confirming success or safely retrying.

8.3 Circuit Breakers on the External Gateway

If the external payment gateway starts failing or timing out repeatedly, a circuit breaker (using a library such as Resilience4j) opens and stops sending new charge attempts to it for a cooldown period, immediately marking new incoming requests as temporarily unavailable, with clear messaging back to the client, rather than letting every request pile up waiting on a failing dependency.

@CircuitBreaker(name = "paymentGateway", fallbackMethod = "handleGatewayUnavailable")
public GatewayResponse charge(PaymentRequest request) {
    return gatewayClient.charge(request);
}

public GatewayResponse handleGatewayUnavailable(PaymentRequest request, Throwable t) {
    return GatewayResponse.temporarilyUnavailable();
}

8.4 Idempotency Key Expiry Policy

Idempotency keys cannot be kept forever, both for storage efficiency and because customers do not retry indefinitely. A carefully chosen expiry window — typically 24 hours — balances giving genuinely slow or delayed retries enough time to be recognised, against not accumulating unbounded storage. This expiry window is documented clearly for client developers, so they understand the guarantee only holds within that window.

8.5 Disaster Recovery

The Ledger Database is replicated across regions with a well-tested failover process, and the entire payment stack can run in an active-active or active-passive multi-region configuration, so that a full regional outage does not stop the marketplace from processing payments, with the Global Load Balancer rerouting traffic automatically to a healthy region.

8.6 Testing Reliability on Purpose

Mature payment teams regularly run controlled chaos engineering exercises — deliberately injecting timeouts into the simulated external gateway call, killing a Redis node, or simulating a database failover during a test transaction — specifically to verify that the idempotency guarantees described in this document hold up in practice, not just on paper. Automated tests that fire the exact same payment request twice in rapid succession, simulating a real client retry, are a standard, mandatory part of the test suite for any payment system.

Payment systems are one of the highest-value targets on the internet — security here has to be layered, thorough, and boring to attackers.

9.1 Rate Limiting and Abuse Prevention

The API Gateway enforces per-customer and per-IP rate limits using a token bucket algorithm, preventing both accidental retry storms from a buggy client and deliberate abuse — such as an attacker probing the payment endpoint rapidly to test stolen card numbers, a pattern known as card testing fraud.

9.2 Idempotency Key Ownership Validation

An idempotency key must be scoped to the authenticated customer who created it. The system must reject any attempt to reuse or guess another customer’s idempotency key, since without this check, an attacker could potentially interfere with or read the cached result of someone else’s transaction.

9.3 PCI DSS Compliance

Handling card data brings the system into scope for the Payment Card Industry Data Security Standard (PCI DSS). In practice, most marketplaces avoid ever touching raw card numbers directly by using tokenisation: the external Payment Gateway Adapter exchanges sensitive card details for an opaque token during checkout, and only that token — never the raw card number — flows through our own Payment Execution Service and Ledger Database, dramatically shrinking the amount of infrastructure that needs to meet the strictest compliance requirements.

In practice, this means the checkout page’s card entry form typically submits card details directly to the payment gateway’s own hosted fields or SDK, bypassing our servers entirely, and our system only ever receives and stores the resulting token, which is useless to an attacker outside the context of our specific merchant account and cannot be reversed back into the original card number.

9.4 Encryption in Transit and at Rest

Every network hop in the diagram, from the client all the way to the external gateway, uses TLS encryption. The Ledger Database encrypts sensitive fields at rest, and access to raw transaction data is tightly scoped through role-based access control, with all access logged for audit purposes.

9.5 Fraud and Risk Scoring

The Fraud and Risk Service evaluates signals such as unusual transaction velocity, mismatched billing and shipping locations, and device fingerprinting, before a charge is attempted — and, importantly, this scoring itself respects the idempotency key so a retried request is never re-scored and potentially flagged inconsistently from its original attempt.

9.6 Audit Logging

Every state transition of a payment, from RECEIVED through to COMPLETED or FAILED, is recorded in an immutable audit log, separate from the operational database, since regulators and internal fraud investigations both require a complete, tamper-evident history of exactly what happened to every transaction and when.

Given the financial stakes, observability for this system needs to be both broad and unusually precise — every duplicate hit, every UNKNOWN transaction, is a story worth reading.

10.1 Key Metrics to Track

10.2 Distributed Tracing

A unique trace ID, generated at the API Gateway, follows a request through every service it touches. This is especially valuable for payments, because when investigating a customer complaint about a specific transaction, engineers can pull up the complete trace — including both the original attempt and any retries — and see exactly what happened at each step.

10.3 Structured, Immutable Logging

Every service emits structured JSON logs including the idempotency key and trace ID, sent to a centralised logging system. For payment-specific events, logs are also written to an append-only, tamper-evident audit trail, since payment logs often need to be retained far longer than typical application logs for compliance reasons.

10.4 Alerting and Service Level Objectives

A reasonable SLO for this system might state that 99.9 percent of payment requests resolve, successfully or with a clear failure, within 3 seconds, and that the count of transactions stuck in the UNKNOWN state for more than 5 minutes stays at zero under normal operation. Alerts fire immediately, paging the on-call engineer, if the UNKNOWN count rises above zero for a sustained period — this specific metric is one of the most direct early warnings of a real problem in the payment pipeline.

10.5 Dashboards Built for Different Audiences

A mature monitoring setup for a payment system typically maintains at least two distinct dashboards, because engineers and business stakeholders need different views of the same underlying data. An engineering-facing dashboard focuses on latency percentiles, error rates, circuit breaker state, and infrastructure health — the kind of detail needed to diagnose a technical incident quickly. A business and finance-facing dashboard, by contrast, focuses on transaction success rate, total processed volume, and the reconciliation mismatch count over time, giving non-engineering stakeholders visibility into the health of the payment system without needing to interpret raw infrastructure metrics, and building shared trust that the guarantees described throughout this document are holding up in production, not just in design documents.

All services are packaged as containers and orchestrated with Kubernetes, with deployment practices tuned for the extra caution a payment system demands.

11.1 Careful, Gradual Rollouts

New versions of the Payment Execution Service or Payment Gateway Adapter are rolled out using a canary strategy, receiving a very small slice of real traffic first — often as low as 1 percent — with close monitoring of the UNKNOWN state count and error rate before gradually increasing traffic to the new version, since a subtle bug in payment logic can be far more costly than in most other systems.

11.2 Auto-Scaling

A Horizontal Pod Autoscaler watches CPU usage and request queue depth for the Payment Orchestrator and Payment Execution Service, adding replicas automatically as load climbs, which matters both for handling normal daily traffic patterns and for absorbing sudden spikes during major sale events.

11.3 Multi-Region Deployment

The stack is deployed across multiple geographic regions for both latency and resilience, with the Ledger Database using a carefully chosen replication strategy — often a single writable primary region per customer’s data with read replicas elsewhere — since payment data typically has strict consistency and regulatory data residency requirements that are more restrictive than the shipping cost or catalogue data seen in other parts of a marketplace.

11.4 Infrastructure as Code

All infrastructure, including the Redis cluster, database cluster, Kubernetes configuration, and networking rules, is defined in code using tools such as Terraform, ensuring environments are reproducible and every change goes through code review — especially important for a system where a misconfiguration could have direct financial consequences.

Different data has different physics — the ledger, the idempotency store, and the reporting workload each need a different storage shape.

12.1 Choosing the Ledger Database

We chose PostgreSQL, a strongly consistent relational database, for the Ledger, specifically because payment records require strict ACID guarantees — meaning a transaction is either fully recorded or not recorded at all, with no partial or inconsistent state ever visible — and because the unique constraint feature is exactly the tool needed to enforce the idempotency guarantee at the storage layer.

12.2 Why Not a NoSQL Store for the Ledger

Many other parts of a marketplace, such as the shipping rate table discussed in other guides in this series, benefit from NoSQL stores optimised for massive read throughput. The Ledger is different: its defining requirement is strong consistency and correctness under concurrent writes, not primarily read throughput, so a relational database with proper transactions and constraints is the better fit here, even though it generally scales horizontally with more operational effort than a NoSQL store.

12.3 Sharding the Ledger for Scale

As transaction volume grows, a single PostgreSQL cluster’s write throughput eventually becomes the limiting factor. The Ledger is sharded, typically by customer ID or a hash of the idempotency key, across multiple independent database clusters, so write load is spread horizontally, while each individual shard still enforces its own strict unique constraint and ACID guarantees within its own scope.

12.4 Redis Cluster Design for Idempotency Keys

The Idempotency Key Store runs as a sharded, replicated Redis cluster. Keys use a bounded time-to-live (typically 24 hours, matching the documented client retry window), and replication ensures that the loss of a single Redis node does not silently lose a key’s claimed status, which could otherwise risk a duplicate execution slipping through.

12.5 Load Balancing Strategy

12.6 Read Replicas for Reporting and Reconciliation

Since the Ledger Database’s primary role is to safely accept writes with strict consistency, read-heavy workloads that do not need up-to-the-millisecond freshness — such as the Reconciliation Service’s periodic comparison job, business reporting dashboards, and customer support lookups — are routed to read replicas rather than the primary write node. This keeps the primary free to focus on what it does best, accepting and safely committing new transaction writes at the throughput the business demands, while still giving every other part of the organisation fast, reliable access to transaction data without risking any impact on payment processing latency itself.

The contract each service exposes — and why splitting the payment flow into distinct services beats one large monolith at this scale.

13.1 The Public Payment API

POST /api/v1/payments
Headers:
  Idempotency-Key: a1b2c3d4-5678-90ef-ghij-klmnopqrstuv

Request Body:
{
  "orderId":            "ORD-93211",
  "amountCents":        250000,
  "currency":           "INR",
  "paymentMethodToken": "tok_9f8e7d6c"
}

Response Body:
{
  "transactionId":  "TXN-77123",
  "status":         "COMPLETED",
  "idempotencyKey": "a1b2c3d4-5678-90ef-ghij-klmnopqrstuv"
}

13.2 Internal Service Contracts

Internal services communicate over gRPC, chosen for its speed and strongly typed contracts — particularly valuable here since a mismatched field type or an ambiguous optional field in a payment message could cause exactly the kind of subtle bug this whole system is designed to prevent. The idempotency key is threaded through every internal call in the chain, not just the initial client-facing request.

13.3 Why Microservices Fit This Problem

Separating the Payment Orchestrator, Fraud and Risk Service, Payment Execution Service, and Payment Gateway Adapter allows each to scale and evolve independently. The Fraud and Risk Service, for example, might need to integrate new machine learning models frequently, while the Payment Execution Service — handling the most safety-critical logic — changes far more conservatively and infrequently, and this separation lets each team move at the pace appropriate to their component’s risk profile.

13.4 Error Handling and Idempotency at the API Level

The public API always returns the same response shape for a given idempotency key, whether this is the very first call or the tenth retry, which is the concrete, observable guarantee the entire architecture is built to deliver. A missing idempotency key header is rejected immediately at the API Gateway with a clear error, rather than being silently accepted and treated as if a key were auto-generated, since that would quietly defeat the entire protection this design provides.

{
  "error": {
    "code":      "MISSING_IDEMPOTENCY_KEY",
    "message":   "An Idempotency-Key header is required for payment requests",
    "retryable": false
  }
}

The reusable ideas this architecture leans on, and the tempting shortcuts that will quietly wreck it.

14.1 Patterns Used

14.2 Anti-patterns to Avoid

The habits mature payment teams share — and the recurring, expensive failures that catch newer ones.

15.1 Best Practices

• Generate the idempotency key once per logical user action, on the client, and reuse it consistently across every retry of that same action.
• Enforce a unique database constraint on the idempotency key as the final, unbreakable safety net — never rely on the cache layer alone.
• Model payment state explicitly, including an honest UNKNOWN state for genuinely ambiguous outcomes, rather than forcing every path into success or failure.
• Thread the idempotency key through every internal service call and every external gateway call in the chain, not just the initial client-facing request.
• Build and continuously run an automated reconciliation process comparing internal records against the external gateway’s records.
• Write automated tests that deliberately fire duplicate concurrent requests and assert exactly one side effect resulted.

15.2 Common Mistakes

• Treating idempotency as purely a client-side concern and not enforcing it on the server, trusting clients to simply “not retry too much.”
• Forgetting that Kafka consumers, cron jobs, and internal retries can also produce duplicates, and only protecting the outermost client-facing API.
• Setting the idempotency key expiry window too short, so a legitimately slow retry — perhaps after a long mobile network outage — arrives after the key has already expired and is incorrectly treated as brand new.
• Blindly retrying an external gateway call after a timeout instead of checking its status first, risking a real duplicate charge on the external side.
• Not testing the concurrent duplicate scenario specifically, since normal functional tests naturally send one request at a time and never exercise the exact race condition this system is built to handle.

15.3 A Pre-Launch Readiness Checklist

Before launching or significantly changing this system, experienced teams verify a short list of critical items:

• The database unique constraint actually exists and is enforced — not just planned in a design doc.
• A load test specifically fires concurrent duplicate requests with the same idempotency key and confirms exactly one charge results.
• The Reconciliation Service is running and its mismatch count is being actively monitored, not just built and forgotten.
• Circuit breaker thresholds for the external gateway are tuned from real observed latency and error rates, not guessed.
• The on-call team knows exactly what the UNKNOWN transaction state means and has a clear, documented runbook for investigating it.

15.4 Documenting the Contract for Client Developers

A subtle but important best practice is treating the idempotency behaviour as a clearly documented contract for whoever builds the client applications, not just an internal backend implementation detail. This documentation should explicitly state when to generate a new key versus reuse an existing one, how long a key remains valid, and exactly what the client should do if it receives a response indicating the original request is still in progress. Without this clarity, well-meaning client developers on a mobile team or a third-party integration partner can easily undermine the entire design — for example by accidentally generating a fresh key on every retry attempt, which silently reopens the exact vulnerability this whole architecture was built to close.

Every serious payment platform independently converges on the same core discipline — strong evidence the pattern reflects the problem, not fashion.

A rapid-fire tour of the questions that come up most often, followed by the durable lessons worth carrying away from this entire walkthrough.

17.1 Frequently Asked Questions

17.2 Key Takeaways

17.3 Closing Thoughts

This design gives a marketplace the ability to process payments at massive scale while providing one of the strongest guarantees a financial system can offer a customer: that a single tap of the “Pay” button, no matter how many times the network forces it to be retried, results in exactly one charge. Achieving this is less about any single clever trick, and more about consistently applying the same discipline — an identity for every attempt, layered protection, and honest handling of ambiguity — at every single layer of the system.

If you are approaching this as a system design interview question, the strongest signal you can give is not simply naming “idempotency key” as a buzzword, but walking through, layer by layer, exactly where duplicates could sneak in — from the client’s own retry logic, through internal service calls, all the way to the external gateway integration — and showing a concrete, layered defence at each one, including an honest plan for the genuinely ambiguous cases that no amount of clever request-response logic alone can fully resolve. That combination of breadth (from the client to the bank) and depth (understanding exactly why each layer of protection is necessary rather than redundant) is what separates a surface-level answer from a truly production-ready design.

Finally, it is worth remembering that this kind of system is never truly “finished” on day one. Real production payment systems evolve continuously as new failure modes are discovered in practice, as external gateway behaviour changes, and as transaction volume grows well beyond initial estimates. The teams that operate these systems successfully over the long term are the ones who treat monitoring, reconciliation, and chaos testing not as one-time launch tasks, but as an ongoing, permanent part of how the system is run — because the cost of complacency in a payment system is measured directly in real customers’ money and real customers’ trust. Building this discipline in from the very first version of the system, rather than retrofitting it after a costly incident, is consistently the difference between a payment platform customers quietly trust and one that makes headlines for the wrong reasons. It is, in the end, a small amount of extra engineering discipline in exchange for a guarantee that customers will never even notice is there — which, for a payment system, is exactly the point.