A neighbourhood coffee-shop stamp card, scaled up to tens of millions of customers and millions of transactions per day — that is what a marketplace loyalty points program really is under the hood.

Think about the last time you collected stamps on a small paper card at a neighbourhood coffee shop — buy nine coffees, get the tenth free. That paper card is, in essence, a loyalty points system: a way of tracking value a customer has earned through repeat behaviour, and a promise that this value can later be exchanged for something worth having. A loyalty points program on a large online marketplace is the same idea, scaled up by many orders of magnitude — instead of one shopkeeper stamping one card, it is a distributed software system tracking earned and spent points for tens of millions of customers, across millions of transactions per day, redeemable not just for a free coffee but for cashback, vouchers, merchandise, airline miles, or donations to charity.

Loyalty programs are not a new idea in commerce. Airlines pioneered large-scale frequent-flyer programs in the 1980s, and retail chains followed with punch cards and stamp books long before that. What has fundamentally changed with the move to digital, marketplace-scale loyalty programs is that points are no longer a side ledger kept loosely in a single store’s cash register — they are now a first-class, real-time, auditable form of internal currency that must be as reliable and tamper-resistant as the money moving through the platform’s actual payment systems. A customer who earns 500 points from a purchase expects to see that balance update within moments, and a customer who redeems 2,000 points for a gift voucher expects that redemption to be irreversible, correctly deducted, and never double-spent.

This is precisely what makes designing a loyalty points system a genuinely interesting distributed-systems problem, not just a simple counter that goes up and down. It combines the correctness requirements of a financial ledger (points must never be lost, duplicated, or double-spent), the scale requirements of a high-traffic e-commerce platform (millions of earn events during a flash sale), and the product complexity of a rewards marketplace (hundreds of redemption options, each with different partners, inventory constraints, and fulfilment mechanics).

1.1 How the Architecture of Loyalty Systems Has Evolved

It is worth tracing how this kind of system has evolved alongside broader shifts in software architecture.

Before designing any system, we need to be precise about the four questions a points program must answer for every customer, and why a marketplace invests real engineering effort into getting them right.

A loyalty points system exists to answer four continuously repeating questions for every customer on the platform:

1. Earning — When a customer completes a purchase (or another qualifying action, like writing a review or referring a friend), how many points should they earn, and when should that be credited?
2. Balance — At any moment, what is the customer’s true, trustworthy, spendable points balance?
3. Redemption — When a customer wants to spend points on one of many reward types, how do we reserve, deduct, and fulfil that redemption correctly, exactly once?
4. Lifecycle — How do points expire, get reversed on refunds, or get adjusted for fraud, and how is all of this kept perfectly auditable?

From a business point of view, loyalty programs exist because acquiring a new customer is expensive, while retaining an existing one is comparatively cheap — industry estimates commonly cite new customer acquisition as costing five to seven times more than retaining an existing customer. A well-designed points program increases purchase frequency, average order value, and platform stickiness, because customers who have accumulated a meaningful points balance are reluctant to switch to a competing marketplace and lose that value.

2.1 Why This Is a Hard System Design Problem

2.2 Quantifying the Business Opportunity

Consider a marketplace with 20 million monthly active shoppers, an average order value of ₹2,000, and a loyalty program that awards 2% of order value as points (so ₹100 worth of points per average order), redeemable at roughly ₹1 of value per 10 points. If loyalty membership increases repeat purchase frequency by even 15% among enrolled customers — a figure well within what published retail loyalty case studies report — the incremental revenue from that single behavioural shift can run into hundreds of crores annually for a platform of this size, comfortably justifying a dedicated engineering investment in a correct, scalable points ledger.

2.3 The Width of the Redemption Catalog Is Itself a Design Constraint

Unlike a simple cashback-only program, a marketplace loyalty program is explicitly asked to support “a wide range of reward options,” and each category of reward carries genuinely different engineering requirements:

The full production architecture, box by box — including the entry-point components like the API Gateway and Load Balancer — and the two-topic event backbone at the centre of it all.

Let’s design the full system now, box by box, including the entry-point components like the API Gateway and Load Balancer, exactly as they would appear in a real production architecture diagram.

3.1 Component-by-Component Breakdown

The single most important decision in this whole system: treat points the way accountants treat money — through an append-only, double-entry ledger.

4.1 Why a Double-Entry Ledger, Not a Single Balance Column

The single most important architectural decision in this entire system is treating points the way accountants treat money: through double-entry bookkeeping. Instead of a mutable “balance” field that gets incremented or decremented in place, every points event is recorded as an immutable row in an append-only ledger table, with a debit and a credit that always net to zero across the whole system. A customer’s current balance is never stored directly — it is always the sum of all ledger entries for that customer.

This design gives three enormous benefits: full auditability (you can always reconstruct exactly how a balance was reached), safety under concurrency (appending new rows is far easier to make correct under concurrent writes than mutating a shared counter), and easy reconciliation (the sum of all customer balances should always match the platform’s total outstanding points liability, a number finance teams track closely).

-- Simplified ledger table shape
CREATE TABLE points_ledger (
    entry_id        BIGSERIAL PRIMARY KEY,
    user_id         BIGINT NOT NULL,
    entry_type      VARCHAR(20) NOT NULL, -- EARN, REDEEM, EXPIRE, REVERSE
    points_delta    INTEGER NOT NULL,      -- positive for earn, negative for redeem/expire
    reference_id    VARCHAR(64) NOT NULL,  -- order id, redemption id, expiry batch id
    idempotency_key VARCHAR(64) NOT NULL UNIQUE,
    batch_id        BIGINT,                -- links redemption/expiry back to the earn batch
    created_at      TIMESTAMP NOT NULL DEFAULT now()
);
CREATE INDEX idx_ledger_user_time ON points_ledger(user_id, created_at);

4.2 Idempotent Crediting of Earned Points

Because the Earn Rules Engine consumes events from Kafka, which offers at-least-once delivery by default, the same OrderCompleted event might be processed more than once after a consumer restart. The idempotency_key column (a deterministic key derived from the order ID, such as "earn:" + orderId) combined with a unique constraint means a duplicate credit attempt simply fails the insert rather than double-crediting the customer.

public class LedgerService {

    public LedgerEntry creditPoints(String userId, int points, String orderId) {
        String idempotencyKey = "earn:" + orderId;
        try {
            return ledgerRepository.insert(new LedgerEntry(
                userId, "EARN", points, orderId, idempotencyKey));
        } catch (DuplicateKeyException e) {
            // Already credited for this order, safe to ignore
            return ledgerRepository.findByIdempotencyKey(idempotencyKey);
        }
    }
}

4.3 Atomic Redemption Using Conditional Balance Checks

The trickiest correctness problem in the whole system is preventing two concurrent redemption requests from both succeeding when only one has sufficient balance to cover. This is solved with a conditional, single-statement debit that checks the computed balance and inserts the debit row within the same database transaction, relying on the database’s own concurrency control (row-level locking or a serializable isolation level) rather than application-level locking.

public class RedemptionService {

    @Transactional(isolation = Isolation.SERIALIZABLE)
    public RedemptionResult redeem(String userId, int pointsRequired, String rewardId) {
        int currentBalance = ledgerRepository.computeBalance(userId); // SELECT ... FOR UPDATE
        if (currentBalance < pointsRequired) {
            return RedemptionResult.insufficientBalance();
        }

        boolean stockReserved = inventoryService.reserve(rewardId, userId);
        if (!stockReserved) {
            return RedemptionResult.outOfStock();
        }

        String idempotencyKey = "redeem:" + userId + ":" + rewardId + ":" + requestId();
        ledgerRepository.insert(new LedgerEntry(
            userId, "REDEEM", -pointsRequired, rewardId, idempotencyKey));

        return RedemptionResult.success();
    }
}

Using SERIALIZABLE isolation (or, more commonly in production, an explicit SELECT ... FOR UPDATE row lock on a per-user balance summary row) ensures that if two redemption requests for the same user arrive concurrently, the database itself serialises them, so the second request always sees the already-decremented balance from the first and correctly fails if funds are insufficient.

4.4 Materialized Balance for Fast Reads

Recomputing a customer’s balance by summing every ledger entry they have ever had would become slower over time as the ledger grows. To keep balance reads fast, the system maintains a materialized balance — a periodically or event-driven updated summary row per user, cached in Redis, that represents “balance as of ledger entry N.” Any read that needs absolute correctness (like the redemption check above) still recomputes from the authoritative ledger with a lock, but the vast majority of simple “show my points” reads on the app’s homepage can be served from this fast, eventually-consistent materialized value.

4.5 Points Expiry Using Time-Bucketed Batches

Points earned on different dates often expire on different dates (a common policy is “points expire 365 days after being earned”). Scanning the entire ledger table for expired batches on every run does not scale. Instead, the Expiry Service groups earn entries into time buckets (for example, one bucket per calendar month) and maintains a small index of “buckets due to expire soon,” so it only ever needs to examine a bounded, relevant slice of the ledger on each run rather than the whole history.

public class ExpiryService {

    // Runs daily; only scans buckets whose expiry date has just passed
    public void processExpiringBatches(LocalDate today) {
        List<EarnBatch> dueBatches = batchRepository.findExpiringOn(today);
        for (EarnBatch batch : dueBatches) {
            int remaining = batch.getRemainingPoints(); // earned minus already redeemed from this batch
            if (remaining > 0) {
                String idempotencyKey = "expire:" + batch.getBatchId();
                ledgerService.debit(batch.getUserId(), remaining, batch.getBatchId(), idempotencyKey, "EXPIRE");
            }
        }
    }
}

Note the FIFO (first-in-first-out) assumption embedded here: when a customer redeems points, the system should conceptually consume the oldest, soonest-to-expire batches first, so that a customer with both old and new points does not lose their older points to expiry while their newer points sit untouched. This “which batch does this redemption draw from” mapping is what the batch_id column on redemption ledger entries is for.

4.6 FIFO Batch Consumption Algorithm

When a redemption debits points, the system needs to decide exactly which earn batches those points are drawn from, so that expiry tracking stays accurate. This is a classic FIFO consumption problem, conceptually similar to how inventory accounting systems consume the oldest stock first.

public class BatchConsumptionService {

    public List<BatchAllocation> consumeFifo(String userId, int pointsToRedeem) {
        List<EarnBatch> activeBatches = batchRepository
            .findActiveBatchesOrderedByExpiry(userId); // oldest expiry first

        List<BatchAllocation> allocations = new ArrayList<>();
        int remaining = pointsToRedeem;

        for (EarnBatch batch : activeBatches) {
            if (remaining <= 0) break;
            int available = batch.getRemainingPoints();
            int consumed = Math.min(available, remaining);
            allocations.add(new BatchAllocation(batch.getBatchId(), consumed));
            remaining -= consumed;
        }

        if (remaining > 0) {
            throw new InsufficientBalanceException(userId, pointsToRedeem);
        }
        return allocations;
    }
}

4.7 Applying the CAP Theorem to This System

Different parts of this architecture make deliberately different CAP trade-offs, and being able to articulate this clearly is a strong signal in a system design interview.

4.8 Concurrency Considerations

Because every ledger operation is scoped to a single user, and that user’s rows are protected by row-level locking (or serializable isolation) within a single database shard, there is no need for distributed locking across services. The one place true cross-service coordination matters is the saga-style redemption flow (debit, then reserve inventory, then fulfil), where consistency is achieved not through a shared lock but through the sequential, compensatable steps described later in this section.

Two sequence flows — earning and redeeming — plus the state machine every earn batch lives through, from the moment it is credited to the moment it is fully redeemed, expired, or reversed.

5.1 Earning Points — End-to-End Flow

5.2 Redeeming Points — End-to-End Flow

5.3 Lifecycle of a Points Batch as a State Machine

This state view highlights an important nuance: a batch can be reversed even after partial redemption, if the originating order is refunded. In that case, the system must debit whatever remains of that batch and, if the customer has already redeemed more than the refunded order would have earned, flag the account for a “negative adjustment” that is handled carefully rather than allowed to push the visible balance below zero silently.

Every disadvantage of this design is the direct cost of the correctness and auditability the advantages depend on — and knowing where to place that complexity is the whole game.

No system design decision comes for free, and a loyalty points ledger is a particularly instructive case study because it sits at the intersection of financial correctness, customer trust, and marketing flexibility. A team building this system is constantly balancing rigor (this is, after all, a form of internal currency) against the need to move quickly on catalog and promotional experimentation. The tables below summarise the most consequential trade-offs.

Reading across both tables together, a pattern emerges: nearly every disadvantage listed is the direct cost of achieving the correctness and auditability the advantages depend on. This is not a coincidence — it reflects a broader principle in system design that rigor is rarely free, and the job of a system architect is not to eliminate these trade-offs but to place them deliberately, spending complexity on the parts of the system (the ledger’s correctness guarantees) where getting it wrong is expensive, while staying simpler in the parts (the catalog’s flexibility) where getting it wrong is cheap to fix.

6.1 Build Versus Buy

Not every marketplace needs to build a loyalty ledger from scratch. Vendors like Talon.One, Loyalty Lion, and Antavo offer configurable loyalty-program platforms that handle earning rules, tiering, and a base reward catalog out of the box. Building in-house, as described throughout this article, becomes the right call once a marketplace needs tight, low-latency coupling with its own order and payment systems, wants to support a genuinely wide and fast-evolving redemption catalog with many bespoke partner integrations, operates at a scale where per-transaction platform fees from a third-party vendor become significant, or needs the kind of custom fraud detection that only comes from deep integration with the platform’s own risk signals. As with most build-versus-buy decisions in system design, the right starting point is usually the vendor platform, with an in-house migration considered only once a specific, measurable limitation of the vendor option is blocking the business.

Where the throughput actually goes, and why a single PostgreSQL primary would collapse the moment a marketplace runs a serious flash sale.

7.1 Capacity Planning Example

Assume a marketplace processing 5,000 completed orders per second at peak (a large flash sale), each generating one earn event. Each ledger insert, even a simple one, might take a few milliseconds under load; a single PostgreSQL primary handling both earns and redemptions serially would become a bottleneck well before reaching this throughput. This is why the ledger is sharded by user ID across many database instances — spreading 5,000 writes per second across, say, 20 shards brings the per-shard load down to roughly 250 writes per second, comfortably within what a well-tuned PostgreSQL instance can sustain.

7.2 Read/Write Asymmetry

Balance-check reads (a customer viewing their points on the homepage) vastly outnumber actual ledger writes, often by a factor of 50 to 100. This is precisely why the materialized, Redis-cached balance exists: it absorbs the overwhelming majority of read traffic, leaving the authoritative, lock-protected ledger reads for the comparatively rare moment a redemption is actually being processed.

7.3 Scaling the Earn Rules Engine

Like the abandonment detection pipeline in event-driven order processing, the Earn Rules Engine scales horizontally by increasing Kafka partition count on the order-completed topic, partitioned by user ID so that a single user’s events are always processed in order by the same consumer instance, allowing per-user earn logic (like daily earning caps) to be tracked safely without cross-instance coordination.

7.4 Load Testing Before Major Sale Events

Ahead of a large promotional event, teams typically replay a multiplied version of historical peak order volume against a staging environment that mirrors production, specifically validating three things: that ledger shard write throughput holds up under the expected earn-event surge, that Redis cache hit rates for materialized balances remain high enough to keep read latency low even as many new balances are being written concurrently, and that redemption throughput (a much lower-volume but higher-stakes path) is not starved of database connections by the much larger volume of concurrent earn writes hitting the same shards.

7.5 Redemption Throughput Versus Earn Throughput

It is worth explicitly noting that earn and redemption traffic have very different shapes. Earn events arrive continuously, proportional to order volume, and can tolerate a small amount of processing delay. Redemption requests are comparatively rare but latency-sensitive — a customer actively waiting on the “redeem” button expects an immediate result. This asymmetry is why many production systems give redemption requests a separate, prioritised connection pool to the ledger database shards, ensuring a burst of earn-event processing during a flash sale never starves a customer’s real-time redemption attempt.

Points are money. The reliability engineering here has to match the standard of a financial system, not that of a marketing feature.

8.1 Isolating the Ledger from the Shopping Critical Path

Earning points is intentionally designed as an asynchronous, eventually-consistent side effect of a completed order, not a synchronous part of the checkout flow. If the entire points pipeline goes down, customers can still shop and pay normally — points simply get credited a little later once the pipeline recovers, since Kafka’s durable log guarantees no earn events are lost.

8.2 Redemption Reliability and Compensation

Redemption, unlike earning, cannot simply be “delayed” from the customer’s point of view — a customer expects an immediate outcome. When a downstream step fails (for example, the Partner Integration Gateway cannot reach an airline’s API to complete a miles transfer after points have already been debited), the system needs a compensating action: refund the debited points back into the ledger as a REVERSE entry, and notify the customer clearly that the redemption did not complete rather than leaving them in limbo.

public RedemptionResult redeem(String userId, int points, String rewardId) {
    LedgerEntry debit = ledgerService.debitWithLock(userId, points, rewardId);
    try {
        partnerGateway.fulfill(rewardId, userId);
        return RedemptionResult.success();
    } catch (PartnerFulfillmentException e) {
        ledgerService.reverse(debit.getEntryId(), "fulfillment failed: " + e.getMessage());
        return RedemptionResult.failedAndRefunded();
    }
}

8.3 Replication and Failover

Each ledger database shard runs with synchronous replication to a standby replica, so a primary failure triggers an automatic failover with no data loss for committed transactions. Kafka topics carrying earn and ledger-change events are replicated across brokers (replication factor 3), ensuring the loss of a single broker never loses an event.

8.4 Disaster Recovery

Beyond single-node failover, the system plans for full regional outages. Ledger shard backups are taken continuously (via write-ahead log shipping) to a secondary region, allowing a bounded recovery-point objective — typically a few seconds to a couple of minutes of potential data loss in the worst case of a sudden total regional failure, which is disclosed and accepted as a documented trade-off rather than left ambiguous. Kafka topics can be mirrored cross-region using tools like MirrorMaker or a managed cluster-linking feature, so that even the event backlog itself survives a full regional loss. Recovery drills — deliberately failing over a shard to its replica in a controlled test — are run periodically specifically because a disaster recovery plan nobody has ever actually exercised is not a plan worth trusting.

A financial-grade ledger deserves financial-grade security discipline — on the wire, at the API boundary, and against the fraud patterns bad actors will absolutely try.

9.1 Authentication and Authorization

All customer-facing requests are authenticated via OAuth2/JWT at the API Gateway before reaching the Points or Redemption Service. Internal service-to-service calls use mutual TLS, and the Ledger Service specifically enforces that only an authenticated, allow-listed set of internal services (Earn Rules Engine, Redemption Service, Expiry Service, Fraud Detection Service) may ever write to it — no service outside this small set can create a ledger entry, regardless of what request it receives.

9.2 Preventing Redemption Fraud

9.3 Data Protection

Ledger entries reference order IDs and reward IDs but avoid storing unnecessary PII directly; contact and shipping details needed for physical merchandise rewards are fetched from the User Profile Service at fulfilment time rather than duplicated into ledger rows, minimising the sensitive data footprint of the ledger itself.

9.4 Testing Strategy for a Financial-Grade Ledger

Testing this system requires more rigor than a typical CRUD service. A mature test suite includes:

• Property-based tests that generate large numbers of random concurrent earn and redeem operations against a test ledger and assert the final balance always matches the sum of all entries.
• Concurrency stress tests that fire many simultaneous redemption requests for the same user against a shared balance to verify the locking strategy actually prevents double-spend under real contention, not just in theory.
• Contract tests against each external reward partner’s sandbox environment to catch breaking API changes before they reach production.
• A dedicated reconciliation test suite that deliberately introduces simulated bugs (a missed idempotency check, a race condition) to confirm the nightly reconciliation job actually catches them.

Because a subtle ledger bug can silently create or destroy real financial value at scale, teams typically hold this service to a materially higher testing bar than most other services in the platform.

You cannot run a financial ledger you can’t measure — observability here is a first-class part of the design, not a bolt-on.

10.1 Key Metrics to Track

10.2 Reconciliation Jobs

Beyond real-time metrics, a periodic (typically nightly) reconciliation job independently recomputes every customer’s balance from the raw ledger and compares it against the materialized, cached balance, alerting if any discrepancy is found. This acts as a safety net that catches subtle bugs in the caching or event-processing layers before they compound into customer-facing balance errors.

10.3 Dashboards for Different Audiences

10.4 Distributed Tracing

Every ledger entry and redemption request carries a correlation ID propagated from the originating order or redemption click through every downstream service, enabling engineers to use tools like Jaeger or Zipkin to trace exactly what happened to a specific points event end to end — invaluable when investigating a specific customer support escalation.

10.5 Service Level Objectives and Error Budgets

A mature team commits to explicit SLOs rather than a vague sense that “the ledger should be fast and correct.” Typical commitments include: 99.99% of ledger debit transactions complete correctly with no double-spend, ever (this one has effectively zero acceptable error budget, unlike most other SLOs in the system); 99.9% of redemption requests receive a definitive success or failure response within 3 seconds; and 99.5% of earned points are reflected in the customer-visible balance within 2 minutes of order completion. Framing correctness as a near-zero-budget SLO, distinct from the more typical 99.9%-style availability SLOs used elsewhere in the system, is a useful way to communicate to stakeholders that not all reliability targets in this system are equally negotiable.

How each service actually gets shipped, sharded, and cost-controlled — with extra caution reserved for the Ledger Service itself.

11.1 Typical Cloud-Native Deployment

• Compute: Each service (Points, Redemption, Ledger, Earn Rules Engine, Expiry, Fraud Detection) runs as a containerised deployment on Kubernetes, with Horizontal Pod Autoscaling driven by CPU usage and Kafka consumer lag.
• Ledger database: Managed PostgreSQL (Amazon RDS/Aurora or Cloud SQL), sharded by user ID, with synchronous replicas for zero-data-loss failover.
• Event streaming: A managed Kafka service (Amazon MSK or Confluent Cloud) removes the operational burden of running brokers directly.
• Cache: Managed Redis (Amazon ElastiCache or Google Memorystore) for materialized balances and idempotency-key lookups.
• CI/CD: Each service has an independent pipeline with automated tests and a progressive rollout strategy; changes to the Ledger Service specifically go through additional manual review given its financial sensitivity.

11.2 Deploying Ledger Changes Safely

Because the Ledger Service is the most sensitive component in the system, changes to it are deployed with extra caution: schema migrations are always additive and backward-compatible, logic changes go through a shadow-mode period (running the new logic alongside the old and comparing outputs without acting on the new result) before being fully cut over, and any change affecting balance computation requires a corresponding update to the nightly reconciliation job’s expectations.

11.3 Multi-Region Considerations

For a marketplace operating across multiple countries, points programs are frequently kept region-scoped rather than globally pooled, both because reward catalogs and tax treatment of loyalty value differ by jurisdiction, and because keeping ledger shards region-local reduces cross-region latency for the redemption path’s locking operations, which are highly latency-sensitive. Only aggregated, non-transactional reporting data typically needs to flow into a single global analytics warehouse.

11.4 Cost Optimisation

The two largest cost levers in this system are database read/write capacity on the ledger shards and partner-integration fees for external reward fulfilment (gift card face-value markups, airline mile-transfer costs). Teams control the first by aggressively caching materialized balances so the vast majority of reads never touch the database, and control the second by negotiating volume-based partner pricing and steering customers toward internally-fulfilled rewards (cashback, internal vouchers) through catalog placement and default ordering, since these carry no external fulfilment cost at all.

Which store owns which data, how sharding by user ID keeps writes fast, and which caches absorb the vast majority of read traffic before it ever touches the database.

12.1 Why PostgreSQL for the Ledger

The Ledger Service needs strong transactional guarantees — the ability to atomically check a balance and insert a debit within a single transaction, with proper isolation to prevent race conditions. A relational database like PostgreSQL, with support for row-level locking and configurable isolation levels, is the natural fit here, in contrast to the Cart Service in a typical checkout system, which can more comfortably use a document store given carts don’t carry the same double-spend risk.

12.2 Sharding by User ID with Consistent Hashing

As with other high-scale user-keyed systems, the ledger is sharded across many PostgreSQL instances using consistent hashing on user ID, so that adding new shard capacity only requires moving a small fraction of users rather than re-sharding the entire dataset. Every ledger operation is inherently scoped to a single user, which makes this a very clean sharding key — nearly all ledger queries naturally stay within a single shard, avoiding the cross-shard query complexity that plagues systems needing to join data across many users at once.

public class ConsistentHashRing {

    private final SortedMap<Long, String> ring = new TreeMap<>();
    private final int virtualNodesPerShard;

    public ConsistentHashRing(List<String> shardIds, int virtualNodesPerShard) {
        this.virtualNodesPerShard = virtualNodesPerShard;
        for (String shardId : shardIds) {
            addShard(shardId);
        }
    }

    public void addShard(String shardId) {
        for (int i = 0; i < virtualNodesPerShard; i++) {
            long hash = hash(shardId + "#" + i);
            ring.put(hash, shardId);
        }
    }

    public String getShardForUser(String userId) {
        long hash = hash(userId);
        SortedMap<Long, String> tailMap = ring.tailMap(hash);
        Long nodeHash = tailMap.isEmpty() ? ring.firstKey() : tailMap.firstKey();
        return ring.get(nodeHash);
    }

    private long hash(String input) {
        return Hashing.murmur3_128().hashString(input, StandardCharsets.UTF_8).asLong();
    }
}

The “virtual nodes” concept in this code, placing each shard at multiple points around the ring rather than just one, exists to smooth out load distribution — without it, a single unlucky hash placement could give one physical shard a disproportionate share of user accounts, and therefore a disproportionate share of ledger write load.

12.3 Caching Layers

Two distinct caches serve different purposes here. First, the materialized balance cache in Redis serves the overwhelming majority of “what are my points” reads without touching PostgreSQL. Second, an idempotency-key cache (also in Redis, with a TTL matched to the realistic window during which a duplicate request might arrive) allows the Redemption Service to quickly reject a duplicate request without needing a full database round-trip in the common case.

12.4 Load Balancing Strategy

The Load Balancer in front of the Points and Redemption services uses least-connections routing, since redemption requests can vary significantly in processing time depending on which reward type and partner is involved, and least-connections avoids sending a disproportionate share of new requests to an instance already busy handling a slow partner call.

A small, well-versioned REST surface at the edge, and a strictly-versioned event contract on the inside — each service moves at the pace its own risk profile allows.

13.1 Example REST API Surface

GET    /api/v1/points/balance              -> Get current points balance
GET    /api/v1/points/history               -> Get ledger history, paginated
GET    /api/v1/rewards/catalog              -> List available reward options
POST   /api/v1/rewards/redeem               -> Redeem points for a specific reward
GET    /api/v1/rewards/redemption/{id}      -> Check status of a redemption
POST   /api/v1/points/earn-preview          -> Preview points to be earned before purchase

13.2 Error Handling

{
  "error": {
    "code": "INSUFFICIENT_BALANCE",
    "message": "You do not have enough points for this redemption",
    "currentBalance": 1450,
    "requiredPoints": 2000
  }
}

Precise error codes like INSUFFICIENT_BALANCE, REWARD_OUT_OF_STOCK, REDEMPTION_LIMIT_EXCEEDED, and ACCOUNT_UNDER_REVIEW (used when the Fraud Detection Service has flagged an account) let the frontend show the customer an accurate, actionable message rather than a generic failure. It is worth deliberately distinguishing retryable errors from non-retryable ones in this error contract: a client encountering REWARD_OUT_OF_STOCK should not automatically retry the same request, since the outcome will not change, whereas a client encountering a generic SERVICE_UNAVAILABLE response can safely retry with backoff, since the underlying condition may well have cleared by the next attempt.

13.3 Why Microservices Instead of a Monolith Here

Splitting the Ledger, Points, Redemption, Catalog, and Partner Integration into independently deployable services means each can evolve at its own pace. The Rewards Catalog, in particular, changes far more frequently than the Ledger — new seasonal rewards, partner onboarding, pricing changes — and decoupling it means marketing and partnerships teams can ship catalog changes without any risk to the sensitive, rarely-changing ledger code.

13.4 Event Schema Versioning

Ledger-change events published to Kafka use a schema registry (commonly Avro with Confluent Schema Registry) to enforce backward-compatible evolution — new optional fields with defaults are fine, but removing or retyping an existing field is not — so that downstream consumers like the Analytics Warehouse and Fraud Detection Service, built and deployed independently, never break when the Ledger Service team adds new event fields.

Every choice above traces back to a well-known pattern — and being able to name the ones you’re avoiding is often as important as the ones you’re using.

14.1 Patterns Used Well

• Event Sourcing: The append-only ledger is a textbook event-sourcing pattern — the current balance is a derived projection of the full history of events, not a stored fact in itself.
• Saga Pattern: Redemption’s debit-then-fulfil-then-confirm-or-reverse flow is a saga, coordinating a multi-step operation across services with an explicit compensating action if a later step fails.
• CQRS: Writes go through the strongly consistent Ledger Service, while the vast majority of reads are served from a separately optimised, eventually-consistent materialized balance — a clean separation of command and query responsibilities.
• Idempotent Receiver: Every write operation (earn credit, redemption debit, expiry) is keyed by a deterministic idempotency key, making the system safe against Kafka’s at-least-once delivery and safe against client retries.
• Outbox Pattern: To guarantee a ledger write and its corresponding published event never drift apart, the Ledger Service can write both the ledger entry and the outgoing event to the same local database transaction, in an outbox table, with a separate relay process publishing to Kafka — ensuring an event is eventually published if and only if the underlying ledger write actually succeeded.
• Dead Letter Queue: A redemption that permanently fails (invalid reward ID, permanently unreachable partner) is routed to a dead letter queue for manual investigation rather than being silently dropped or endlessly retried against a partner that will never succeed.

14.2 Anti-Patterns to Avoid

14.3 Why the Anti-Patterns Matter More Than They First Appear

Each of these anti-patterns tends to survive initial code review because it works fine under light load and low concurrency — the bugs only surface once real traffic creates genuine contention. A synchronous partner call inside a locked transaction, for example, might pass every test written against a fast sandbox environment, but the very first time a partner’s production API responds slowly under its own load, every redemption attempt for every customer queues up behind that single slow lock, turning an isolated partner slowdown into a platform-wide redemption outage. This is exactly why the architecture in this article insists on separating the debit from the fulfilment call, with compensation handling the rare failure case explicitly rather than hoping the fast path is always fast.

The habits that keep a loyalty ledger honest at scale — and the ordering in which real teams typically discover and adopt them.

15.1 How Mature Teams Evolve This System Over Time

Most loyalty programs start far simpler than the architecture described here — often a single service with a mutable balance column and one or two redemption options (usually just cashback or a simple voucher). As the program grows in scale and the reward catalog widens, teams typically migrate to a proper ledger design first (since correctness bugs become costly quickly), then decouple the catalog and partner integrations as the number of reward types grows, and finally build out dedicated fraud detection and reconciliation tooling once the program’s financial liability becomes large enough that finance and risk teams demand tighter guarantees. Recognising this evolutionary path is a useful thing to bring up in a system design interview — it signals an understanding that this level of architecture is earned through real scale, not assumed from day one.

The same architectural shape shows up across every serious loyalty program in the industry, from marketplaces to airlines to your neighbourhood coffee chain.

16.1 Lessons Learned Across the Industry

A consistent lesson across companies that have discussed their loyalty infrastructure publicly is that the ledger’s correctness properties matter more than almost any other part of the system — a slow redemption is a minor annoyance, but a double-spent or lost point is a trust and financial issue that erodes customer confidence in the whole program. Another consistent lesson is that reward catalog breadth is a major driver of program engagement: customers redeem more actively, and therefore re-engage with the platform more often, when they have many small, achievable redemption options rather than only a small number of high-value, hard-to-reach ones.

16.2 Comparing Loyalty Program Archetypes

The marketplace loyalty program described in this article sits at the more complex end of this spectrum, precisely because it combines a very high-volume, multi-category earn trigger with a deliberately wide and heterogeneous redemption catalog — a combination that neither a single-brand punch card system nor even a typical single-purpose airline program needs to fully solve.

16.3 Glossary of Key Terms

The questions that come up most often when engineers — and interviewers — probe the reasoning behind each choice in this system.