Every month, billions of small transactions happen quietly in the background of modern life. Your Netflix subscription renews. Your gym membership charges your card. Your cloud hosting bill gets deducted. Your favorite meal-kit service ships a new box and bills you for it. None of these require you to lift a finger — the money simply moves, on schedule, every single time.

This entire category of software is called recurring subscription billing, and it is one of the most deceptively difficult problems in system design. It looks simple on paper: “charge the customer’s card every 30 days.” But underneath that one sentence hides a minefield of distributed systems problems — network failures, duplicate charges, expired cards, currency conversion, retries, timezones, and money that must never be lost, never be double-counted, and always be traceable.

Subscription billing as a business model is not new. Newspapers and magazines have billed subscribers on a recurring basis for over a century. What changed in the last two decades is the scale and the automation. In the 1990s, a magazine’s billing team might process a few thousand renewals a month, largely by mail or manual card entry. Today, a company like Netflix, Spotify, or Amazon Prime bills tens of millions of customers, often on similar dates (like the 1st of the month), which creates massive traffic spikes that a system must absorb without falling over.

The rise of Software-as-a-Service (SaaS) in the 2000s and 2010s made recurring billing a core piece of infrastructure for almost every tech company, not just media companies. Payment processors like Stripe, Braintree, Razorpay, and PayPal built dedicated “Billing” and “Subscriptions” APIs specifically because so many companies were solving the same hard problem — and getting it wrong in expensive ways, either by charging customers twice or by silently losing revenue when charges failed.

In this tutorial, we will design a recurring subscription billing system from scratch — the kind of system that could power a company like Netflix, Spotify, or a SaaS startup. We will focus heavily on two of the hardest requirements in this space: handling failed payment retries without double-charging, and never losing track of a customer’s billing cycle, even while the system is under extreme load — think a million requests landing in a single minute during a mass renewal event.

Let us break down exactly why this problem is hard. On the surface, “charge a customer every month” sounds like a single database update. In reality, it touches at least five separate concerns that can each fail independently:

Now layer on scale. Imagine a company with 50 million subscribers, and a large fraction of them signed up around the same promotional date, so their renewals cluster together. On the 1st of the month, the scheduler may need to trigger a million or more billing events within a single minute. If our system architecture cannot absorb that burst without falling over, we get cascading failures: timeouts to the payment gateway, duplicate charge attempts because the customer’s request retried before the first one finished, and potentially very angry customers charged twice for the same subscription.

2.1 The two core guarantees we must design for

Everything in this tutorial ultimately serves two non-negotiable guarantees:

• Exactly-once effective charging. A given billing cycle for a given subscription must result in at most one successful charge, even if the request to charge it is sent, retried, times out, or is processed by ten different service instances simultaneously.
• Durable billing cycle state. At any point in time, we must be able to answer, with certainty, “has this customer’s current cycle been paid, is it pending, or has it failed and how many times has it been retried?” — and that answer must never be lost, even during a crash, a deployment, or a network partition.

2.2 Framing the problem against CAP

It helps to place this problem on the consistency-availability spectrum before drawing any boxes. The write path — the actual movement of money and the update of billing cycle state — is deliberately biased toward consistency: we would rather delay a charge safely in a queue than risk writing conflicting records from two nodes that both think they are the primary. The read path — a customer browsing their past invoices, a support agent looking up account history — is deliberately biased toward availability, served from read replicas that may lag the primary by a second or two. Recognizing that one system can (and should) sit in different places on the CAP spectrum for different flows is the single most important framing decision in this design.

2.3 Scale numbers we need to plan for

To ground the discussion, a realistic mid-to-large scenario looks like this: 50 million active subscriptions across a mix of monthly, yearly, and weekly plans; roughly 1–3% of those subscriptions renew on any given calendar day, clustered heavily around calendar boundaries; peak burst of about 1 million billing events per minute, roughly 16,700 events per second; a first-attempt success rate typically in the 85–95% range depending on catalog and geography, meaning 5–15% of cycles enter the retry pipeline; and a support-visible tail of retries and dunning communications that can persist for up to two weeks after the original due date. Every architectural choice from this point on is measured against these numbers.

Before we draw any architecture diagrams, let us build a shared vocabulary. Each of these terms will come up repeatedly.

3.1 Subscription

A subscription is a record that links a customer to a plan (e.g., “Pro Monthly Plan, $15/month”) and tracks its own lifecycle: active, past due, suspended, or cancelled. Think of it as a standing agreement — “keep billing me this amount, this often, until I cancel.”

3.2 Billing Cycle

A billing cycle is one specific period of time within a subscription — for example, “March 1 to March 31” — and it has a due date, an amount owed, and a status (scheduled, paid, past due). A subscription generates a new billing cycle every time the previous one completes. This is the unit we must never double-charge and never lose.

3.3 Idempotency and idempotency keys

An operation is idempotent if performing it multiple times has the same effect as performing it once. In payments, we achieve this using an idempotency key — a unique identifier generated for each intended charge (usually a combination of subscription ID and billing cycle ID) that is sent to the payment gateway. The gateway (and our own internal services) remember which keys have already been processed, and refuse to process the same key twice, simply returning the original result instead.

3.4 Retry with exponential backoff and dunning

When a charge fails — say, due to insufficient funds — we do not retry immediately in a loop. That would hammer the payment gateway and annoy banks. Instead, we use exponential backoff: retry after 1 day, then 3 days, then 7 days, giving the customer time to fix their payment method or their bank balance to recover. This structured retry process, combined with customer communication (emails, in-app banners) asking them to update their card, is called dunning — an old accounting term for the process of pursuing overdue payments.

3.5 Ledger

A ledger is an append-only record of every financial event: every attempted charge, every success, every failure, every refund. Critically, ledger entries are never edited or deleted — if a mistake happens, you add a new correcting entry, you never rewrite history. This is the same principle accountants have used for centuries with physical ledger books, and it is what allows us to reconstruct exactly what happened to any dollar at any time.

3.6 Payment Service Provider (PSP)

A PSP (like Stripe, Razorpay, Braintree, or Adyen) is the external service that actually talks to card networks and banks on our behalf. We never store raw card numbers ourselves (this is a huge security and compliance simplification); instead we store a token or reference that the PSP gives us, and we ask the PSP to charge that token.

Now let us design the actual system. We will build it as a set of independently deployable services, connected through an event backbone, sitting behind a load balancer and API gateway. Every request — whether it comes from a scheduler firing a billing event or a customer updating their card — passes through the same edge layer, so we get consistent authentication, rate limiting, and traffic shaping no matter the entry point.

4.1 Component breakdown

4.2 What happens if each component fails, in isolation

A useful way to stress-test any architecture is to remove one box at a time and ask what actually breaks. If the Load Balancer fails, no external traffic reaches the platform at all — this is why load balancers are always deployed as redundant pairs or as a managed cloud service, not as a single instance. If the API Gateway tier fails, backend services become unreachable, so gateway instances themselves are horizontally scaled and load-balanced. If the Scheduler is briefly down, no new billing events are emitted, but nothing already in-flight is lost — the moment it recovers, it picks up the cycles it missed based on their still-Scheduled state in the primary database. If the Payment Service is fully down, queued events accumulate safely in Kafka and are processed once it recovers, and no charge is ever silently lost because nothing is dequeued and forgotten. If Redis goes down, idempotency checks fall back to a database-backed lookup path with strictly worse latency but identical correctness. If the PSP is down, the circuit breaker discussed later trips and holds new charges in the queue rather than firing them into a failing dependency — retries automatically resume when the PSP recovers.

Let us trace exactly what happens, step by step, when a billing cycle comes due. This is the sequence that must be bulletproof, because this is where money changes hands.

5.1 Why the idempotency check comes first

Notice that the very first thing the Payment Service does — before touching the PSP at all — is check the cache for the idempotency key. This ordering is deliberate. The idempotency key is deterministically generated from the subscription ID and the billing cycle ID (for example, sub_8842_cycle_2026_08), so no matter how many times this exact charge request is retried — by the customer’s client, by a queue redelivery, or by our own Retry Orchestrator misfiring — it always produces the same key, and the cache will recognize it.

5.2 The “reserve then confirm” pattern

A subtle but critical detail: the Payment Service does not just check the key and then charge — it reserves the key (marks it as “processing”) before calling the PSP, and only writes the final result after the PSP responds. This two-step reserve-then-confirm pattern prevents a dangerous race condition: if two instances of the Payment Service somehow received the same charge request at nearly the same moment (for example, due to an at-least-once delivery guarantee from the message queue), only the first one to successfully reserve the key proceeds to call the PSP. The second one sees the key is already reserved and waits for, or defers to, the first attempt’s result.

Java example: atomic idempotency reservation

public ChargeResult processCharge(ChargeRequest request) {
    String idempotencyKey = request.getIdempotencyKey();

    // Atomic claim: only one caller can succeed here
    boolean claimed = redisClient.setIfNotExists(
        idempotencyKey, "PROCESSING", Duration.ofMinutes(10));

    if (!claimed) {
        // Someone else already owns this key; poll for the final result
        return waitForFinalResult(idempotencyKey);
    }

    try {
        ChargeResult result = paymentGatewayClient.charge(
            request.getPaymentMethodToken(),
            request.getAmount(),
            idempotencyKey);

        redisClient.set(idempotencyKey, serialize(result), Duration.ofDays(7));
        ledgerService.recordTransaction(request, result);
        return result;

    } catch (GatewayTimeoutException e) {
        // Do not assume success or failure; enqueue for reconciliation
        redisClient.set(idempotencyKey, "UNKNOWN_NEEDS_RECONCILIATION",
            Duration.ofDays(7));
        reconciliationQueue.enqueue(idempotencyKey);
        throw e;
    }
}

5.3 Algorithms, data structures, and concurrency control underneath the flow

Behind the simple-looking sequence diagram in Figure 2, several classic computer science ideas are doing real work.

Consistent hashing for queue partitioning

When the Scheduler publishes a ChargeRequested event, it does not pick a queue partition at random — it hashes the customer ID (typically with an algorithm like Murmur3) and maps that hash onto one of the queue’s partitions using consistent hashing. This guarantees that all events for a given customer always land on the same partition, which in turn guarantees they are consumed in order by a single consumer. Without this, two events for the same subscription could be processed by two different consumer threads simultaneously, reintroducing the very race condition idempotency keys are meant to prevent at the network layer, but now at the ordering layer.

Compare-and-swap for state transitions

The database update that moves a billing cycle from Scheduled to Processing is a compare-and-swap (CAS) operation in spirit: UPDATE ... SET status = 'PROCESSING' WHERE id = ? AND status = 'SCHEDULED'. This is the same fundamental primitive used in lock-free concurrent data structures — it either succeeds atomically because the expected old value matched, or fails atomically because someone else changed it first, with no in-between state ever observable to other transactions. Relational databases give us this for free through their row-level locking and MVCC (Multi-Version Concurrency Control) implementations.

Priority queues for retry scheduling

The Retry Orchestrator does not scan the entire billing_cycle table every second looking for cycles whose retry time has arrived. Instead, it maintains (or queries an index equivalent to) a min-heap ordered by “next retry timestamp,” so it can always cheaply find the next batch of cycles due for a retry attempt without an expensive full scan. In practice this is usually implemented as a well-indexed database column combined with a delayed-message feature of the queue (many queue systems, including Kafka via a delay-topic pattern, or purpose-built schedulers like temporal workflows, support this natively).

Token bucket for rate limiting

The outbound rate limiter toward the PSP is a classic token bucket: tokens refill at a fixed rate (say, 2,000 per second, matching the PSP’s contracted limit), and each outbound charge call consumes one token. If the bucket is empty, the call waits or is deferred back onto the queue rather than being sent and rejected. This smooths bursty internal traffic into a steady stream the PSP can reliably accept.

A billing cycle moves through a well-defined set of states from the moment it is scheduled to the moment it is either paid or the subscription is cancelled. Modeling this explicitly as a state machine — rather than scattering booleans and flags across the code — is what keeps the system predictable under retries and failures.

6.1 Walking through the lifecycle

• Scheduled — the cycle exists with a known due date and amount, but nothing has happened yet.
• Processing — the Payment Service currently has an in-flight or reserved idempotency key for this cycle. No other charge attempt can be made while in this state.
• Failed_Attempt_N — the charge was declined; the system now waits according to the backoff schedule before trying again.
• Dunning — all automated retries have been exhausted; the customer is actively notified and given a grace period to fix their payment method.
• Suspended — access to the product is paused (but the subscription is not yet deleted), giving the business one more chance to recover the customer.
• Paid — the cycle is closed successfully, and a new Scheduled cycle is created for the next period.
• Cancelled — terminal state; the subscription ends.

6.2 Why we never let two states be true at once

Every billing cycle row in the database has exactly one status column, and transitions between statuses happen through a single, guarded update — usually a conditional SQL update like UPDATE billing_cycle SET status = 'PROCESSING' WHERE id = ? AND status = 'SCHEDULED'. If the row was already moved out of “Scheduled” by another process, this update affects zero rows, and the caller knows immediately that someone else got there first. This is the database-level equivalent of the atomic idempotency reservation we saw earlier, and it is what prevents two concurrent scheduler runs from both trying to charge the same cycle.

7.1 Key trade-off: synchronous vs asynchronous charging

We chose an asynchronous, queue-based flow for the main billing cycle path. The alternative — charging synchronously inside a single API call — is simpler to reason about but does not scale to a million-events-per-minute burst, because it ties up a request thread for the entire duration of the (sometimes slow) call to the PSP. The trade-off is that asynchronous flows introduce latency between “cycle due” and “customer notified” — usually seconds, sometimes longer under load — which is an acceptable cost for the throughput and resilience gained.

7.2 A quick summary of the deliberate compromises

This is where the “million requests in a minute” requirement lives. Let us design for it explicitly rather than hoping the architecture happens to hold up.

8.1 Where the burst actually comes from

The burst is not customer-initiated traffic — it is our own scheduler generating events because many subscriptions happen to renew on the same date. This is actually good news: because we control the producer, we can smooth the burst ourselves rather than relying on downstream systems to absorb it unpredictably.

8.2 Techniques that make a million events a minute survivable

Jittered scheduling

Instead of firing all due billing events at exactly midnight, the Scheduler Service spreads them across a window (say, a 2-hour band) using deterministic jitter based on customer ID. Customers still get billed “on their date,” but not all in the same 60-second window. This alone can turn a million-events-in-a-minute problem into a much gentler few-thousand-events-per-minute problem.

Partitioned, backpressure-aware queueing

The message queue (e.g., Kafka) is partitioned by customer ID, so charge events for different customers can be processed fully in parallel, while events for the same customer stay strictly ordered (important so we never process two cycles for one subscription out of order). Consumers pull at a rate they can sustain rather than events being pushed faster than they can be handled — this is backpressure, and it protects the Payment Service and the PSP from being overwhelmed.

Horizontal auto-scaling

The Payment Service, Subscription Service, and Retry Orchestrator are all stateless and horizontally scalable — state lives in the database, cache, and queue, not in service memory. This means Kubernetes (or an equivalent orchestrator) can scale Payment Service pods from a baseline of 50 up to 500+ automatically as queue depth grows, and scale back down afterward to save cost.

Rate limiting toward the PSP

Even with all our own scaling, the PSP itself has rate limits. The Payment Service enforces a token-bucket rate limiter on outbound calls to the PSP, so a burst on our side degrades into a longer (but steady) processing queue rather than a flood of failed requests and PSP-imposed throttling.

8.3 Back-of-the-envelope capacity estimate

8.4 Request coalescing on the read path

Popular subscription-holder screens like “my invoices” and “my plan” occasionally see coordinated bursts — for example, when a product-wide email is sent linking to that page. A coalescing layer in front of the read replicas ensures that if thousands of requests miss the cache for the same page fragment at once, only one actually queries the database, while the rest wait on the shared result. This turns a potential thundering-herd on the read path into a single origin call, at the cost of a small amount of added latency for the waiting requests.

9.1 No single point of failure

Every layer in the architecture is deployed across multiple availability zones at minimum, and often multiple regions for the largest deployments. The Load Balancer itself is typically a managed, redundant service (like an AWS Application Load Balancer or a cloud provider’s equivalent) that has no single instance to fail.

9.2 Handling partial failures gracefully

9.3 Circuit breaker pattern in practice

public class PaymentGatewayCircuitBreaker {
    private final AtomicInteger consecutiveFailures = new AtomicInteger(0);
    private volatile long openedAtMillis = 0;
    private static final int FAILURE_THRESHOLD = 20;
    private static final long COOLDOWN_MS = 30_000;

    public boolean allowRequest() {
        if (openedAtMillis == 0) return true;
        if (System.currentTimeMillis() - openedAtMillis > COOLDOWN_MS) {
            openedAtMillis = 0; // half-open: allow a trial request
            return true;
        }
        return false;
    }

    public void recordSuccess() {
        consecutiveFailures.set(0);
        openedAtMillis = 0;
    }

    public void recordFailure() {
        if (consecutiveFailures.incrementAndGet() >= FAILURE_THRESHOLD) {
            openedAtMillis = System.currentTimeMillis();
        }
    }
}

9.4 CAP theorem in the context of billing

The CAP theorem states that a distributed system can only guarantee two of three properties during a network partition: Consistency, Availability, and Partition tolerance. Since partitions are a fact of life in any real network, the practical choice is really between consistency and availability when a partition occurs.

For the Ledger and billing cycle state, we deliberately favor consistency over availability. If the primary database is unreachable, we would rather delay processing a charge (and let it queue up safely) than risk writing an inconsistent or duplicate financial record from a stale replica. This is why the Payment Service’s core write path talks to a single-writer primary database rather than accepting writes on multiple replicas.

For read-heavy, less critical paths — like a customer viewing their past invoice history in the app — we lean toward availability over strict consistency, serving from read replicas that might lag the primary by a second or two. A customer seeing yesterday’s invoice list refresh a moment late is a minor inconvenience; a customer being charged twice because we chose availability for the write path would not be.

9.5 Consensus and leader election

Several components in this architecture rely on distributed consensus under the hood, even though we rarely implement it ourselves. Kafka brokers use a consensus protocol (via a controller elected through KRaft or historically ZooKeeper) to agree on partition leadership, ensuring exactly one broker is authoritative for a given partition at a time — this is what gives us the “one consumer group member processes a given partition” guarantee we rely on for ordered, non-duplicated event handling. Similarly, our database’s primary-replica setup uses a consensus or quorum mechanism to agree on which node is the current primary during failover, preventing a dangerous “split-brain” scenario where two nodes both believe they are the primary and accept conflicting writes.

9.6 Disaster recovery

The Ledger Database and Primary Database are continuously replicated to a secondary region with a defined Recovery Point Objective (RPO) of under one minute and a Recovery Time Objective (RTO) of a few minutes, using automated failover. Because billing correctness matters more than almost anything else in the business, backups are tested regularly with real restore drills, not just taken and forgotten.

9.7 Graceful degradation from the customer’s point of view

It is worth explicitly designing what a customer experiences when parts of this pipeline are degraded, rather than leaving it as an accident of implementation. If the Notification Service is temporarily behind, receipts arrive a few minutes late but no billing action is affected. If the Retry Orchestrator is paused, past-due cycles simply stay past-due until it resumes; no cycle is prematurely cancelled just because retries are delayed. If read replicas lag, past-invoice screens may briefly show a slightly older list, but the current cycle’s payment status — the one that matters — is always served from the primary. The one thing that must never degrade, under any combination of failures, is the correctness of the ledger itself, which is why it is deliberately the simplest and most carefully guarded write path in the entire system.

10.1 Never touch raw card data

The single most important security decision in this design is that our services never store or even see raw card numbers. All card entry happens through the PSP’s own hosted fields or SDK, which returns us an opaque token. This dramatically shrinks our PCI-DSS (Payment Card Industry Data Security Standard) compliance scope — we only ever handle tokens, not primary account numbers.

10.2 API Gateway as a security boundary

Every request to any internal service passes through the API Gateway, which enforces authentication (via signed JWTs or OAuth 2.0 tokens), authorization (does this caller have permission to touch this subscription), and rate limiting (protecting against both abuse and accidental self-inflicted traffic storms).

10.3 Idempotency keys must not be guessable

Because idempotency keys are deterministic, we must ensure they cannot be guessed and abused by an attacker to probe for information about other customers’ transactions. Keys are namespaced with an internal, non-public subscription identifier and are never accepted directly from untrusted client input for server-triggered billing cycles — only the Subscription Service, an internal trusted component, generates them.

10.4 Encryption and secrets

• All data in transit uses TLS 1.2 or higher, terminated at the Load Balancer and re-encrypted internally between services (mutual TLS in stricter environments).
• All data at rest — especially the Ledger Database — is encrypted using provider-managed or customer-managed keys.
• PSP API keys and database credentials live in a secrets manager (like AWS Secrets Manager or HashiCorp Vault), never in code or environment files committed to source control.

10.5 Fraud and abuse prevention

The Payment Service integrates basic fraud signals — velocity checks (has this payment method been used across many unrelated accounts recently), geographic anomaly detection, and PSP-provided risk scores — before submitting high-risk charges, and can route suspicious transactions through 3D Secure (an additional bank-side authentication step) rather than blindly attempting to charge.

10.6 Least-privilege and audit logging

Only a narrow set of internal services and a small, well-defined administrator role are ever authorized to initiate refunds or manual charge overrides through the Payment Service. Every such privileged action is written to an audit log (backed by the same append-only discipline as the ledger itself) capturing who performed it, when, from which IP, and against which subscription. Combined with mutual TLS between internal services, this makes both accidental misconfiguration and deliberate insider abuse considerably harder to hide.

11.1 The three pillars

We instrument the system with metrics, logs, and distributed traces, following the standard observability model:

11.2 Correlation IDs

Every billing event carries a correlation ID generated at the Scheduler and propagated through every hop — the queue message, the Payment Service’s PSP call, the Ledger entry, and the Notification Service’s email. This is what turns “we had an incident” into “here is the exact sequence of 14 log lines across 5 services for this one customer’s failed charge.”

11.3 Critical alerts

11.4 Tiered alerting and dashboards

Good alerting is tiered rather than binary. Soft warnings fire when a metric drifts a meaningful amount off baseline — for example, charge success rate falling by two standard deviations from a rolling 24-hour mean — giving on-call engineers time to investigate before customers notice. Hard pages fire only for signals that mean money is at risk right now: ledger write failure, sustained unbounded queue lag, or reconciliation backlog exceeding a business-defined threshold. A good operational dashboard groups panels into three tiers: a headline row (overall charge success, queue lag, reconciliation depth) visible at a glance; a middle row breaking down per-PSP and per-region success rates so localized issues do not hide inside a healthy global average; and a bottom row surfacing business-facing signals like recovered revenue from dunning and total value stuck in the retry pipeline.

12.1 Containerized microservices on Kubernetes

Each service (Subscription, Payment, Retry Orchestrator, Ledger, Notification, Scheduler) is packaged as a Docker container and deployed on Kubernetes, which handles scheduling, health checks, and horizontal pod auto-scaling based on CPU, memory, or custom metrics like queue depth.

12.2 Blue-green and canary deployments

Because this system touches money, deployments are rolled out cautiously. A new version of the Payment Service is first deployed as a canary receiving a small percentage (e.g., 5 percent) of traffic; only after its error rates and latency match or beat the current version does the rollout proceed to 100 percent. Blue-green deployment (maintaining two full environments and switching traffic between them) provides an instant rollback path if something goes wrong.

12.3 Infrastructure as code

All infrastructure — Kubernetes clusters, database instances, Kafka clusters, load balancer configuration, IAM roles — is defined declaratively using tools like Terraform, so environments are reproducible, reviewable through pull requests, and auditable.

12.4 Multi-region deployment for the highest tiers

For companies operating at true global scale, the entire stack (except perhaps the PSP integration, which may be region-specific for regulatory reasons) is deployed across multiple cloud regions, with the Global Load Balancer routing customers to their nearest healthy region, as shown earlier in Figure 4.

12.5 Cost optimization

A billing system’s compute needs are extremely spiky — a huge burst around common renewal dates, and comparatively idle time in between. This makes it a strong candidate for aggressive auto-scaling policies, including scaling the Payment Service tier down to a small baseline overnight and during low-traffic windows, rather than provisioning for peak capacity around the clock. Spot or preemptible instances are a reasonable fit for the Notification Service (which can tolerate brief interruptions and retries) but are deliberately avoided for the Payment Service tier itself, where mid-request interruption risk is not worth the cost savings on a workload this sensitive. Database and cache costs are managed by right-sizing read replica counts to actual read traffic and using storage tiering (hot ledger data on fast storage, older archived ledger data moved to cheaper cold storage after a defined retention period, while remaining queryable for audits).

12.6 A quick capacity planning worksheet

When sizing this system for a new deployment, work through these questions in order: what is the peak billing-events-per-minute you expect on the busiest calendar day of the year (drives Kafka partition count and Payment Service pod ceiling)? what is the typical first-attempt failure rate for your customer geography (drives Retry Orchestrator throughput and dunning email volume)? what is the average duration a cycle spends in the retry pipeline (drives Redis idempotency-key TTL and cache size)? and finally, what is the business-tolerated Recovery Point Objective for the ledger (drives replication strategy and backup cadence)?

13.1 Data model

13.2 Why relational, not NoSQL, for the core billing tables

Billing data has strong relational integrity requirements (a payment attempt must belong to exactly one billing cycle; amounts must be exact decimals, never floating point) and benefits enormously from ACID transactions when updating cycle status and writing ledger entries together. A relational database like PostgreSQL, with row-level locking and conditional updates, is the right foundation here — even though other parts of the broader product (like caching or activity logs) might legitimately use NoSQL stores.

13.3 Sharding strategy

As the customer base grows past what a single database instance can handle, we shard by customer ID (or a hash of it), so a given customer’s subscription, cycles, and payment attempts always live on the same shard — this keeps the common query patterns (like “get this customer’s current cycle”) a single-shard operation, avoiding expensive cross-shard joins.

13.4 Caching strategy

• Idempotency keys — stored in Redis with a time-to-live of about 7 days, long enough to cover any realistic retry window, short enough to bound memory usage.
• Subscription plan details — rarely change, so they are cached aggressively (minutes to hours) to avoid repeated database lookups during the scheduler’s scan.
• Cache invalidation — plan updates explicitly invalidate the relevant cache keys on write, rather than relying purely on TTL expiry, since stale pricing is a real business risk.

13.5 Load balancing algorithms

The Load Balancer uses least-connections routing rather than simple round robin for the Payment Service tier, since individual charge requests can vary significantly in duration depending on PSP response times — least-connections keeps work more evenly spread across instances under uneven latency.

13.6 Consistent hashing at the cache tier

The Redis cluster used for idempotency keys is fronted by consistent hashing so that when nodes are added or removed to handle growing load, only a small slice of keys need to move to a different node — not the majority of them, as would happen with naive modulo hashing. This matters specifically because reshuffling a large fraction of live idempotency keys during a rescaling event is exactly when the system is most vulnerable to duplicate-charge windows, and consistent hashing keeps that risk small and bounded.

14.1 Key internal APIs

POST /internal/v1/charges
Headers: Idempotency-Key: sub_8842_cycle_2026_08
Body: {
  "subscriptionId": "sub_8842",
  "billingCycleId": "cycle_2026_08",
  "amount": "15.00",
  "currency": "USD",
  "paymentMethodToken": "pm_tok_9f3a"
}

Response 200 OK
{
  "attemptId": "att_772f",
  "outcome": "SUCCEEDED",
  "ledgerEntryId": "ledg_00931"
}

Notice the idempotency key is passed as a header, following the convention used by Stripe and most major PSPs, so retried HTTP requests (due to client timeouts) are inherently safe even before our own internal reservation logic runs.

14.2 Why microservices instead of a monolith here

Splitting Subscription, Payment, Retry, Ledger, and Notification into separate services allows each to scale, deploy, and fail independently. A surge in notification volume (say, a mass dunning email campaign) should never be able to slow down the Payment Service’s ability to process charges. This isolation is worth the added operational complexity at this scale.

14.3 Synchronous vs asynchronous API boundaries

14.4 The transactional outbox pattern

To guarantee that a database update (marking a cycle paid) and a queue publish (notifying downstream systems) either both happen or neither happens, we use the transactional outbox pattern: the event to be published is written to an “outbox” table in the same database transaction as the state change, and a separate relay process reads the outbox table and publishes to the queue, marking rows as sent. This avoids the classic dual-write problem where a database commit succeeds but the queue publish fails (or vice versa), which would otherwise create exactly the kind of inconsistency we are trying to eliminate.

14.5 API contract discipline

Because these services evolve independently, their contracts have to evolve carefully. Every internal endpoint carries an explicit version prefix (like /internal/v1/charges), breaking changes are shipped by adding a new version alongside the old rather than mutating the existing one, and contract tests run on every deploy to verify that changes to a producer’s schema do not silently break its consumers. Field additions are always safe (consumers ignore unknown fields), field removals are treated as breaking changes, and enum value additions require checking that every consumer’s switch statements have a sensible default branch.

15.1 Patterns used in this design

15.2 Anti-patterns to avoid

15.3 Compensating actions for partial failures

Occasionally a charge that already succeeded needs to be reversed — for example, a customer disputes a duplicate signup, or an automated fraud check catches something after the fact. Rather than trying to “undo” a ledger entry, the correct pattern is to append a new refund entry that offsets the original. This is the same forward-only correction philosophy accountants have used for centuries: history is never rewritten, only extended. It is also a lightweight relative of the saga pattern, applied to a single financial operation rather than an entire distributed workflow.

16.1 Best practices

16.2 Common mistakes

16.3 Testing strategy

Because correctness matters more here than in almost any other subsystem, the testing pyramid for a billing system leans unusually heavily on integration and chaos-style tests rather than relying on unit tests alone.

• Unit tests cover the pure logic — proration math, backoff interval calculation, state machine transition rules — with no network calls involved.
• Contract tests against a PSP sandbox environment verify that our idempotency key usage actually behaves as the PSP’s real API expects, since assumptions about vendor behavior are a common source of production surprises.
• Chaos and fault-injection tests, as described in the FAQ section below, deliberately introduce timeouts, duplicate deliveries, and mid-transaction crashes to prove the system self-heals correctly rather than merely hoping it does.
• Load tests simulate the million-events-per-minute burst scenario in a staging environment sized proportionally, to validate that auto-scaling policies and rate limiters engage correctly before a real traffic spike ever hits production.
• Reconciliation drills periodically run the reconciliation job against a controlled set of intentionally ambiguous transactions to confirm it correctly resolves “unknown outcome” states without either duplicating or dropping a charge.

17.1 The common thread

Across every one of these companies, the same underlying pattern repeats: a durable record of what is owed and when, a strictly idempotent execution path for actually moving money, a graduated retry and communication strategy before punishing the customer, and an immutable audit trail. The specific technology choices vary — some companies run their own PSP integrations, others lean entirely on a vendor like Stripe Billing — but the architectural shape converges because the underlying correctness problem is the same one we solved in this tutorial.

17.2 Alternative designs considered, and why they were rejected

A strong system design answer does not just present one solution — it shows you evaluated and consciously rejected reasonable alternatives.

17.2.1 A single nightly batch job on a monolith

The simplest possible design is a single cron job that queries every due subscription and charges them one at a time from a monolithic application. This is genuinely fine at very small scale (say, a few thousand subscribers), but it collapses under a million-events-per-minute burst because one process becomes a serial bottleneck, and any single failure (a slow PSP response, a database blip) can stall the entire nightly run. It was rejected here because the whole point is designing for the burst case, not the average case.

17.2.2 Fully synchronous charging directly from the API

Another option is having every charge happen in a synchronous API call, with the caller (the Scheduler) waiting on the response before moving to the next customer. This is simpler to reason about but ties up a thread per in-flight charge for the entire duration of the PSP call, which does not scale to the throughput this system needs. Async queueing with idempotent consumers gives us the same correctness with far higher throughput.

17.2.3 A vendor-only solution (Stripe Billing or similar)

For many teams, outsourcing the entire billing pipeline to a vendor like Stripe Billing or Chargebee is the right call: it eliminates a huge category of engineering work, and vendor engineering teams have already solved most of the problems we discussed. This tutorial designs the system from scratch specifically because interviewers ask about it and because larger companies eventually build their own for cost, control, and multi-PSP flexibility reasons — but building it in-house is genuinely a choice with real trade-offs, not the only defensible answer.

18.1 Frequently asked questions

18.2 Putting it all together

Stepping back from the individual components, this design ultimately rests on a small number of ideas working together rather than any single clever trick. A single authoritative write path through the Payment Service, gated by a deterministic idempotency key, guarantees that no cycle is ever charged twice regardless of what happens elsewhere in the pipeline. An explicit billing-cycle state machine, backed by conditional database updates, guarantees that no two processes can ever proceed on the same cycle simultaneously. An event backbone with per-customer partitioning decouples bursty producers from steady, sustainable consumers while preserving the ordering guarantees we depend on for correctness. And a deliberate split between eventually-consistent read paths and strongly-consistent write paths means the system can be fast where staleness is harmless, and strict where it is not. None of these ideas is unique to billing — the same pattern shows up across inventory, seat availability, ride matching, and account balances — which is exactly why this problem is such a popular and durable interview question.

18.3 Closing thought

Recurring subscription billing is a great system design interview topic precisely because it rewards depth over breadth. A candidate who draws a clean architecture diagram with an API Gateway and Load Balancer in front of a few services is showing surface familiarity with modern infrastructure. A candidate who can explain why an idempotency key must be deterministic, why it must be reserved atomically before an external call, why an “unknown outcome” must never be assumed to be a failure, and why a jittered schedule turns a million-request spike into a manageable stream, is showing the kind of judgment that actually keeps a real payments system trustworthy in production. The architecture diagrams in this tutorial are a starting point, not a finished blueprint — the reasoning behind each decision is the part worth carrying into your own designs.