Designing a Real-Time Multi-Party Payment Splitting System

Designing a Real-Time Multi-Party Payment Splitting System

Designing a Real-Time Multi-Party Payment Splitting System

A complete, interview-grade walkthrough of building a system like Splitwise-meets-Venmo: one expense, divided among several people, where each person pays their own share independently, at their own time, while the group sees balances update in real time — the independent-Payable state machines, sharded aggregate counters, append-only ledger, and event-driven fan-out that make the whole thing scale to millions of split-payment events per minute without ever losing a cent.

01

Introduction & History

Imagine six friends go on a trip. One person pays the hotel bill upfront. Another pays for the rental car. A third books the safari tour. By the end of the trip, everyone owes everyone else a little money, and nobody wants to do the mental math. This is the exact problem that group expense-splitting apps solve, and it has quietly grown into one of the most interesting distributed systems problems in consumer fintech.

The idea itself is old — people have split bills with pen and paper, or with a calculator passed around a dinner table, for as long as group spending has existed. What changed is the shift from “who owes whom” being a piece of social bookkeeping to being an actual, regulated, real-money movement problem. Early tools like Splitwise (launched around 2011) solved the bookkeeping half of the problem extremely well: they tracked who owed what, computed the minimum number of transactions needed to settle a group, and left the actual money movement to bank transfers, cash, or third-party apps. What we are designing here goes one step further — a system where the payment itself is split, tracked, and collected in real time, with each participant paying their own share independently, and the group organizer (or the merchant) seeing the payment fill up like a shared pool.

This is the same category of problem faced by apps that support “Group Pay” for event tickets, food delivery, hotel bookings, or shared subscriptions — where one total needs to be broken into several independent payment obligations, collected asynchronously, and reconciled against a single logical transaction. It combines three worlds that don’t normally sit together: the social/collaborative nature of group apps, the strict correctness guarantees of payment systems, and the real-time expectations of modern mobile UX.

By the end of this tutorial, you will understand how to design such a system from the ground up: how a bill gets split, how each participant’s share is tracked as an independent payable unit, how partial payments are collected without ever double-charging or losing money, how the group sees live updates, and how the system scales to handle millions of these split transactions per minute while staying financially consistent to the last cent.

Everyday analogy

Think of it like a shared kitty jar at a group dinner: each person drops in exactly their own share when they’re ready, everyone can see the jar filling up in real time, and nobody at the table is on the hook for anyone else’s money. What we’re designing here is that same jar — only wired into card networks and mobile wallets, running at millions of transactions per minute, with an auditor sitting inside guaranteeing every cent is accounted for.

💬
What an interviewer may ask

“Why not just let one person pay the full amount and settle later?” — Because many products (event ticketing, group food orders, shared subscriptions) require each participant to pay with their own payment instrument for compliance, liability, and refund-attribution reasons. The organizer should not bear the financial or fraud risk of the whole group.

02

Requirements

2.1 Functional Requirements

  • A user (the “organizer”) can create a split payment request for a total amount, tied to an expense (a bill, an order, an event, a subscription).
  • The total can be divided equally, by fixed custom amounts, by percentage, or by shares/weights (for example, 2 shares for a couple, 1 share for a single person).
  • Each participant sees their own owed share and can pay it independently, at any time, using their own payment method (card, wallet, bank transfer, UPI, etc.).
  • The group (and the organizer) can see a real-time view of how much of the total has been collected, and who has and hasn’t paid.
  • The system must support partial completion — a split can be “60% collected” for hours or days without failing.
  • Support reminders, nudges, and expiry of unpaid shares.
  • Support refunds — for the whole split, or for an individual participant’s share, or proportionally if the underlying expense is cancelled or reduced.
  • Handle overpayment/underpayment edge cases such as a participant leaving the group after paying, or a total being edited after some shares are already paid.
  • Provide an audit trail: every cent must be traceable to a specific participant, payment instrument, and timestamp.

2.2 Non-Functional Requirements

  • Strong consistency for money movement — no double charges, no lost payments, no split that appears “fully paid” when it is not.
  • Eventual consistency with low latency is acceptable for the “live view” of group balances (a few hundred milliseconds of lag is fine), but the underlying ledger must be strongly consistent.
  • High availability — payment collection must keep working even if the “social” layer (chat, notifications, live view) degrades.
  • Idempotency for every payment attempt, retried network call, and webhook from a payment processor.
  • Scale: design for millions of split-payment events per minute at peak (think a large ticketing platform on a concert on-sale day, or a food delivery app during dinner rush).
  • Auditability and regulatory compliance (PCI-DSS for card data, KYC/AML where applicable, data residency).
  • Low latency for the pay action itself — users should get payment confirmation in under 2–3 seconds.
📌
Back-of-envelope estimation

Assume 50 million daily active users, of whom 5% create or participate in a split each day → 2.5 million splits/day, averaging 3.5 participants each → roughly 8.75 million individual share-payment events per day. At a realistic peak-to-average ratio of 8x (dinner-time and weekend spikes), peak throughput is roughly 800–900 payment events per second, or comfortably in the “millions of requests per minute” range once you include read traffic (balance checks, live view polling/streaming, notification fan-out), which typically runs 20–50x higher than write traffic in social-payment apps.

03

Architecture & Components

The system is best thought of as three cooperating layers: a social/collaboration layer (who is in the group, how the bill is split), a payment collection layer (each participant paying their share through a payment gateway), and a ledger and reconciliation layer (the single source of truth for money, immune to double-counting). Splitting these concerns is the single most important architectural decision in this system, because it lets you scale and secure the money-handling part independently from the more experimental, fast-moving social features.

flowchart TB subgraph Client [Client Layer] MOBILE[Mobile App] WEB[Web App] end subgraph Edge [Edge Gateway] LB[Load Balancer] APIGW[API Gateway Authn Rate Limit Routing] WS[Realtime Gateway WebSocket SSE Fan-out] end subgraph Core [Core Services] SPLIT[Split Service Creates Split Computes Shares] GROUP[Group Service Membership Roles] PAY[Payment Orchestrator Service] LEDGER[Ledger Service Double-Entry Source of Truth] WALLET[Wallet Balance Service] NOTIF[Notification Service] RECON[Reconciliation Service] end subgraph Async [Event Backbone] BROKER[Event Streaming Platform Kafka] end subgraph External [External Systems] PSP[Payment Service Provider Gateway] BANK[Bank Card Network] KYC[KYC Fraud Service] end subgraph Data [Data Stores] SPLITDB[Split Metadata DB] LEDGERDB[Ledger DB Append-Only] CACHE[Distributed Cache] SEARCH[Search Analytics Store] end MOBILE –> LB WEB –> LB LB –> APIGW APIGW –> SPLIT APIGW –> GROUP APIGW –> PAY MOBILE -.-> WS WS –> BROKER SPLIT –> SPLITDB SPLIT –> BROKER GROUP –> SPLITDB PAY –> KYC PAY –> PSP PSP –> BANK PAY –> LEDGER LEDGER –> LEDGERDB LEDGER –> BROKER BROKER –> WALLET BROKER –> NOTIF BROKER –> RECON BROKER –> SEARCH WALLET –> CACHE RECON –> LEDGERDB RECON –> PSP
Fig. 3.1 — End-to-end architecture: client, gateway, core services, event backbone, external systems, and data stores.

3.1 Component Responsibilities

ComponentResponsibility
API GatewayTLS termination, authentication, authorization, request routing, rate limiting, request/response schema validation.
Split ServiceCreates a split, computes each participant’s share (equal, custom, percentage, shares), stores the split definition, emits a “SplitCreated” event.
Group ServiceOwns group membership, invites, roles (organizer vs participant), so the Split Service doesn’t need to duplicate this logic.
Payment OrchestratorDrives the state machine for each individual participant’s payment: initiate, authorize, capture, handle failure/retry, handle refund.
Ledger ServiceThe financial source of truth. Every money movement is recorded as a double-entry transaction here. Nothing is considered “true” until the ledger says so.
Wallet / Balance ServiceMaintains fast-read aggregate views (“split is 60% funded,” “you owe $12.50”) derived from the ledger, optimized for the live UI.
Reconciliation ServicePeriodically compares the ledger against the payment gateway’s records to catch drift, delayed webhooks, or missed events.
Realtime GatewayMaintains WebSocket/SSE connections and fans out balance-update events to connected clients in a group.
Event Streaming PlatformDecouples services; every state change is published as an event so downstream consumers (notifications, analytics, realtime gateway) don’t block the payment path.
💬
What an interviewer may ask

“Why is the Ledger Service separate from the Payment Orchestrator?” — Because the orchestrator deals with the messy, retryable, externally-dependent world of talking to a payment gateway, while the ledger must be a clean, append-only, internally consistent record. Mixing them means a slow or flaky PSP call could block or corrupt your source of truth for money.

04

Internal Working

4.1 Splitting the Bill

When an organizer creates a split, the Split Service must first decide how to divide the total amount into individual “shares.” This sounds trivial but has a classic rounding problem: if $100 is split three ways equally, each person owes $33.33, but $33.33 × 3 = $99.99, one cent short. The system must deterministically assign the leftover cents so the sum of shares always equals the original total exactly. A common approach is the “largest remainder method”: compute each share, truncate to the smallest currency unit, then distribute the leftover cents one at a time to the participants with the largest fractional remainder, in a stable, reproducible order (for example, by participant ID).

For percentage or weighted (shares) splits, the same rounding-remainder logic applies after computing each participant’s raw share from their weight. All amounts inside the system are represented as integers in the smallest currency unit (for example, cents, paise) — never as floating point — to avoid the classic binary floating-point rounding bugs that have caused real financial discrepancies in production systems.

4.2 Independent Payment Collection

Once shares are computed, each participant gets their own Payable record — effectively a mini invoice tied back to the parent split. This is the key architectural insight: a split payment is not one transaction, it is a fan-out of N independent transactions that happen to share a parent context. Each Payable has its own state machine (pending → authorized → captured → settled, or failed/expired/refunded), and each transitions independently of the others. Participant A paying does not block, delay, or depend on Participant B paying.

This independence is what allows true “pay whenever you want” behavior. The Payment Orchestrator treats every Payable exactly like a standalone payment: it never needs to know or care whether it is 1 of 1 or 1 of 12 participants. The aggregation — “how much of the total is collected” — is a derived, read-side concern computed by the Wallet/Balance Service, not something the payment path needs to reason about synchronously.

4.3 Aggregation Without Locking

A naive design would maintain a single mutable “amount collected so far” counter on the parent split row, and increment it inside the same database transaction as each payment capture. This works at small scale but becomes a hot-row contention problem at scale — if a split has 500 participants (for example, a large event ticket group-buy) all paying within the same minute, every payment would be fighting to lock and update the same row.

The better approach: each payment capture is written as its own immutable ledger entry (an event, not an update to shared state). The “amount collected” is computed either (a) on read, by summing captured Payables for that split — fine for low-traffic splits, or (b) maintained as an eventually-consistent materialized counter, updated asynchronously by a stream consumer reading the ledger’s append-only event log and applying atomic increments (for example, a Redis INCR or a database counter shard) rather than inside the payment’s own transaction. This trades a few hundred milliseconds of “eventual” balance visibility for near-unlimited write scalability — an acceptable trade for a live view, unreasonable for the ledger itself.

sequenceDiagram participant U as Participant App participant GW as API Gateway participant PO as Payment Orchestrator participant PSP as Payment Gateway participant LG as Ledger Service participant BUS as Event Bus participant WB as Wallet Balance Service participant RT as Realtime Gateway U->>GW: Pay my share GW->>PO: Initiate Payment with idempotency key PO->>PSP: Authorize charge PSP–>>PO: Authorized PO->>PSP: Capture PSP–>>PO: Captured with txn id PO->>LG: Record double-entry transaction LG–>>PO: Committed PO–>>U: Payment confirmed LG->>BUS: Publish PaymentCaptured event BUS->>WB: Update split aggregate async BUS->>RT: Push balance update RT–>>U: Live update to all group members
Fig. 4.1 — Sequence for a single participant paying their own share of a split.
💬
What an interviewer may ask

“How do you avoid hot-row contention when 500 people pay into the same split at once?” — Decouple the write path (each payment is an independent, immutable ledger entry) from the aggregate path (the “total collected” counter is updated asynchronously via the event stream, using sharded counters or CRDT-style commutative increments so no single row becomes a bottleneck).

4.4 Concurrency Control on the Payment Path

Even though individual Payables are independent of each other, a single Payable can still face internal concurrency hazards — for example, a user double-tapping “Pay” on a slow network, or a client retrying a request that actually succeeded server-side but timed out before the response arrived. Two broad strategies handle this:

  • Optimistic concurrency control on the Payable row: every update carries a version number (or a compare-and-swap on status), so a transition from Pending to Authorizing only succeeds if the row is still in the expected prior state. A second concurrent attempt simply fails the compare-and-swap and is told “already in progress,” rather than corrupting state.
  • Idempotency keys as the true concurrency guard: rather than relying purely on database-level locking, the client-generated idempotency key is the authoritative de-duplication mechanism. The orchestrator’s first rule on any incoming pay request is “have I seen this exact idempotency key before?” — if yes, it returns the previously recorded outcome instead of re-executing anything, which is far cheaper and safer than distributed locking under high concurrency.

Pessimistic row-level locking is deliberately avoided on the hot payment path wherever possible, because holding a lock across a network call to an external PSP (which can take anywhere from tens of milliseconds to several seconds) would tie up database connections and create cascading contention exactly when the system is under the most load. The combination of optimistic state transitions plus idempotency keys achieves correctness without ever holding a lock across an external network call.

4.5 Consistency Model: Why the Ledger is Strict and the Live View is Not

This system deliberately runs two different consistency models side by side, and understanding why is one of the most important conceptual pieces of the design. The Ledger Service behaves like a classic ACID system: every write is a strongly consistent, durable, isolated transaction, because money that appears to move but didn’t (or vice versa) is a direct financial and legal liability. The Wallet/Balance Service, by contrast, behaves like an eventually consistent, CRDT-friendly system: it is perfectly fine for two group members to see the “amount collected” figure update at slightly different times, as long as it converges to the correct value within a bounded, short window (typically under one second). This split mirrors the classic distinction between a system’s “system of record” and its “system of engagement” — the former must be boring, strict, and provably correct; the latter should be fast, responsive, and forgiving of minor staleness.

4.6 Consensus & Replication for the Ledger

Because the Ledger DB cannot tolerate split-brain writes (two replicas both accepting conflicting transactions during a network partition), its underlying storage engine relies on a consensus protocol — commonly Raft or Paxos-based replication in modern distributed SQL systems — to elect a single write leader per shard and guarantee that a transaction is only acknowledged as committed once a quorum of replicas has durably persisted it. This is what allows the system to survive the loss of a single node (or even an entire availability zone) without losing or duplicating a committed payment: any replica that missed the quorum simply catches up from the replication log, and the leader election process ensures at most one node believes it is authoritative for a given shard at any moment.

05

Data Flow & Lifecycle

Every split payment moves through a well-defined lifecycle. Modeling this explicitly as a state machine — rather than scattering boolean flags across tables — is what keeps the system correct as new requirements (partial refunds, expiry, group edits) get added over time.

5.1 Split Lifecycle

stateDiagram-v2 [*] –> Draft Draft –> Active: Organizer publishes split Active –> PartiallyFunded: At least one share captured PartiallyFunded –> FullyFunded: All shares captured Active –> Expired: Deadline passed zero captured PartiallyFunded –> Expired: Deadline passed some captured FullyFunded –> Settled: Funds disbursed to payee PartiallyFunded –> Cancelled: Organizer cancels Active –> Cancelled: Organizer cancels Settled –> Refunding: Refund requested Refunding –> Refunded: Refund completed Expired –> Refunding: Auto-refund captured shares
Fig. 5.1 — Split-level lifecycle: draft, active, partial/full funding, settlement, cancellation, and refund states.

5.2 Individual Payable (Share) Lifecycle

stateDiagram-v2 [*] –> Pending Pending –> Authorizing: Participant taps Pay Authorizing –> Authorized: Gateway approves Authorizing –> Failed: Gateway declines Authorized –> Captured: Capture confirmed Captured –> Refunded: Refund issued Failed –> Pending: Retry allowed Pending –> Expired: Deadline passed Captured –> [*] Refunded –> [*] Expired –> [*]
Fig. 5.2 — Per-participant Payable lifecycle: independent state machine per share.

5.3 Step-by-Step Walkthrough

  1. Creation: Organizer selects participants and split method. Split Service computes shares using integer-cent arithmetic and the largest-remainder rounding rule, persists the split and its Payables in Draft state, then transitions to Active once published. A SplitCreated event is emitted.
  2. Notification: The Notification Service consumes SplitCreated and pushes each participant a request-to-pay notification, with their exact share amount.
  3. Independent Payment: Each participant, on their own schedule, opens the Payable and pays via their preferred instrument. The Payment Orchestrator drives authorization and capture with the PSP, using a unique idempotency key per attempt so retried taps or flaky networks never cause duplicate charges.
  4. Ledger Write: On successful capture, the Ledger Service records a double-entry transaction: debit the participant’s payment instrument (external), credit an internal “Split Escrow” account for that split. This is the durable, immutable fact.
  5. Async Fan-Out: The ledger write publishes an event to the streaming platform. Consumers update the read-optimized balance aggregate, push a realtime update to all connected group members, and log the event for analytics.
  6. Completion Check: When the sum of captured shares equals the split total, the split transitions to FullyFunded, triggering disbursement — releasing the escrowed funds to the organizer, merchant, or event system.
  7. Expiry / Cleanup: A scheduled job sweeps splits past their deadline; unpaid Payables are marked Expired, and if the product requires “all or nothing” funding (like crowdfunding-style group buys), any already-captured shares are automatically refunded.
📌
Production example

Ticketing platforms such as those used for large concerts implement a very similar “group cart” flow: one person reserves seats, and each attendee pays their own portion within a fixed hold window (commonly 10–15 minutes), after which unpaid seats are released back into inventory. The hold window is functionally identical to the “Expired” transition in the Payable lifecycle above, and is essential to prevent inventory or funds from being stuck indefinitely on an incomplete group purchase.

06

Data Model

The data model separates three concerns: the split’s metadata (owned by the Split Service), the individual payables and their payment attempts (owned by the Payment Orchestrator), and the immutable ledger entries (owned by the Ledger Service). Keeping these as logically — and often physically — separate stores prevents an application bug in the social/UX layer from ever being able to corrupt financial records.

EntityKey FieldsNotes
Splitsplit_id, group_id, organizer_id, total_amount_cents, currency, split_method, status, deadline_at, created_atImmutable once Active except via explicit, audited edit events.
Payable (Share)payable_id, split_id, participant_id, owed_amount_cents, status, idempotency_key, updated_atOne row per participant per split. Independently stateful.
PaymentAttemptattempt_id, payable_id, psp_reference, method, amount_cents, result, created_atMultiple attempts can exist per payable (for example, a failed retry before success).
LedgerEntryentry_id, split_id, payable_id, debit_account, credit_account, amount_cents, entry_type, created_at (append-only)Never updated or deleted; corrections are new offsetting entries.
BalanceSnapshotsplit_id, amount_collected_cents, participants_paid, last_updated_atDerived/cached; rebuildable from LedgerEntry at any time.

Notice that LedgerEntry rows are append-only. This is a deliberate, non-negotiable design choice in payment systems: you never UPDATE a financial fact, you only add new facts (like a refund entry that offsets a capture entry). This gives you a full audit trail for free and makes reconciliation and dispute resolution tractable — you can always replay the ledger to reconstruct the true state at any point in time.

07

Advantages, Disadvantages & Trade-offs

Upside

Advantages

  • Each participant controls their own payment instrument and timing — no single point of financial liability on the organizer.
  • Independent Payables scale horizontally; there is no shared mutable state on the hot write path.
  • Append-only ledger gives strong auditability and simplifies dispute handling and regulatory reporting.
  • Decoupled async fan-out keeps the payment critical path fast, since notifications and realtime updates never block a capture.
Downside

Disadvantages

  • Significantly more complex than a single-payer model — more moving parts, more states, more failure modes.
  • Eventually-consistent live balances mean the UI can briefly show a stale “collected so far” figure.
  • Partial funding creates product and UX complexity: what happens if a split never completes? Escrow, timeouts, and refund policies all need careful design.
  • Multiple independent payment instruments multiply fraud surface area and PSP fee overhead compared to one large charge.

7.1 Key Trade-off Decisions

📌
Trade-off — Synchronous vs. Asynchronous Aggregation

Choice made: Asynchronous aggregation via event stream for the “live view,” synchronous, strongly consistent writes for the ledger itself.
Why: The ledger must never lie about whether money moved. The aggregate view can lag by a few hundred milliseconds without any real harm — users tolerate a brief delay before a progress bar updates, but they will never tolerate being charged twice or a payment vanishing.

📌
Trade-off — Escrow vs. Direct-to-Payee

Choice made: Route captured funds into an internal escrow account until the split is fully funded (or a partial-release policy is met), rather than paying the organizer/merchant immediately per share.
Why: Escrow makes refunds atomic and simple if the split never completes, and avoids the organizer receiving fragmented, hard-to-reconcile trickles of money.

08

Performance & Scalability

At the scale of millions of split-related requests per minute, three parts of the system need explicit scaling strategies: the payment write path, the balance read path, and the realtime fan-out path.

~900/sPeak payment captures
3.5k–4k/sPeak event bus messages
10k+/sPeak balance reads
< 1sLive view convergence

8.1 Write Path (Payment Capture)

  • Partitioning by split_id / payable_id: Since Payables are independent, they can be sharded across database partitions using a hash of participant_id or payable_id, so no single shard becomes a hotspot even for a viral, thousand-participant split.
  • Idempotency keys stored in a fast key-value store (Redis, DynamoDB) with a short TTL let the orchestrator safely retry PSP calls without risk of double capture — a client-generated key ensures retried taps map to the same underlying attempt.
  • Async PSP webhooks: many gateways confirm captures via asynchronous webhook rather than a synchronous response; the orchestrator must be built to accept “pending → confirmed via webhook” as a first-class flow, not an afterthought.

8.2 Read Path (Balance / Live View)

  • Sharded counters: instead of one row per split holding “amount collected,” use N sub-counters (for example, 16 shards per split) that are summed on read and incremented round-robin on write, eliminating contention for viral splits.
  • Cache-aside with cache invalidation via event stream: the Wallet Service keeps balances in a distributed cache (Redis/Memcached), invalidated or updated as ledger events arrive, so most reads never touch the primary database.
  • Read replicas for the Split Metadata DB serve the (much higher volume) “view my group’s splits” traffic, keeping that load off the primary used for writes.

8.3 Realtime Fan-Out

Pushing live balance updates to every group member is a classic fan-out problem. For small groups (a handful of friends) this is trivial. For large groups (hundreds of event attendees), a naive “push to every open connection synchronously” approach does not scale. The standard solution is a pub/sub topic per split or per group, with the Realtime Gateway layer subscribing connected clients to the relevant topic and the event bus handling fan-out delivery, decoupling the number of connected clients from the load on core payment services.

flowchart LR LEDGER[Ledger writes event] –> TOPIC[Kafka Topic per split id] TOPIC –> RT1[Realtime Gateway Node 1] TOPIC –> RT2[Realtime Gateway Node 2] TOPIC –> RT3[Realtime Gateway Node N] RT1 –> C1[Connected Clients Group A] RT2 –> C2[Connected Clients Group B] RT3 –> C3[Connected Clients Group C]
Fig. 8.1 — Realtime fan-out: per-split pub/sub topic decouples connected-client load from core services.
💬
What an interviewer may ask

“How would you handle a split with 10,000 participants (for example, a viral crowdfunded event)?” — Shard the aggregate counter, make every write asynchronous relative to the read-optimized view, use per-split pub/sub topics rather than broadcasting to a single global channel, and consider a polling fallback with exponential backoff for clients on unreliable connections rather than assuming persistent WebSockets scale infinitely.

8.4 Capacity Planning Walkthrough

Translating the earlier back-of-envelope estimate into concrete capacity numbers helps validate the architecture. At a sustained peak of roughly 900 payment-capture events per second, and assuming each capture triggers, on average, one ledger write, one cache update, one notification event, and one realtime push, the event bus needs to comfortably sustain on the order of 3,500–4,000 messages per second at peak, which is well within the range of a modestly sized Kafka cluster (a handful of brokers, tens of partitions per topic) rather than requiring exotic infrastructure. The read path is the more demanding side: if each of the roughly 3.5 average participants per split checks the live view 3–4 times during an active split, and splits stay “live” for an average of several hours, read QPS for balance checks alone can reach tens of thousands per second at peak — which is exactly why the design pushes reads onto a cache-backed, denormalized Wallet Service rather than ever hitting the Ledger DB directly for a balance check.

Storage growth is dominated by the append-only ledger: at roughly 9 million share-payment events per day, each ledger entry being a few hundred bytes, the system accumulates on the order of a few gigabytes of ledger data per day before compression — trivial for modern storage, but the append-only nature means the table (or its underlying log) grows monotonically, so partitioning by time (in addition to the participant-based sharding used for lookups) and a clear archival/cold-storage policy for entries older than the active regulatory retention window are necessary from day one.

8.5 CAP Theorem in Practice

The CAP theorem states that during a network partition, a distributed system must choose between consistency and availability for the affected data. This system does not make one global choice — it makes the choice per subsystem, deliberately. The Ledger Service is CP: during a partition that prevents a quorum from being reached, it will refuse new writes rather than risk two partitions each independently confirming a payment, because an unavailable ledger for a few seconds is recoverable, but a duplicated or lost payment is not. The Wallet/Balance Service and Realtime Gateway are AP: during a partition, they continue serving the last-known (possibly slightly stale) balance rather than failing outright, because showing a user a balance that is a few seconds out of date is a far better experience than showing an error screen.

09

High Availability & Reliability

9.1 Replication & Failover

The Ledger DB, being the financial source of truth, is typically deployed with synchronous replication within a region (to guarantee no committed transaction is ever lost on failover) and asynchronous replication cross-region (for disaster recovery). This reflects the CAP theorem trade-off directly: for the ledger, the system favors consistency over availability during a network partition — it is better to briefly reject a payment than to risk a split-brain scenario where two nodes both believe they captured the same payment.

The Split Metadata and social layers, by contrast, can favor availability — if the “view group members” feature is degraded for a few seconds during a partition, that is a far smaller cost than a ledger inconsistency.

9.2 Idempotency & Exactly-Once Effects

True exactly-once delivery does not exist in distributed systems, so the design instead achieves exactly-once effect through idempotency: every payment initiation carries a client-generated idempotency key, the orchestrator persists the outcome keyed by that value, and any retry (from a flaky mobile network, a timed-out request, or an app crash) is answered from the stored outcome rather than re-executed against the PSP.

9.3 Failure Recovery Scenarios

FailureMitigation
PSP times out after charging the cardOrchestrator treats the attempt as “Unknown,” does not retry blindly, and reconciles against the PSP’s transaction status API before deciding success/failure.
Ledger write fails after PSP capture succeedsOutbox pattern: PSP capture and ledger write are coordinated through a durable local transaction plus retryable outbox event, guaranteeing the ledger is eventually written even if the first attempt crashes mid-flight.
Event bus consumer lags or crashesBalance/notification consumers are idempotent and resume from their last committed offset; the ledger itself is unaffected, so worst case is a delayed live-view update, never a lost payment.
Region outageCross-region async replica is promoted; any transactions not yet replicated are reconciled from PSP records during recovery (see Reconciliation Service).
📌
Backup & disaster recovery

Ledger data is backed up continuously via write-ahead log shipping to durable object storage, enabling point-in-time recovery. Because the ledger is append-only, backups are simple to verify — you can always replay from a known-good snapshot plus the log and compare checksums against the live system.

9.4 Graceful Degradation

A well-designed payment system degrades in layers rather than failing all at once. If the fraud-scoring service becomes slow or unreachable, the orchestrator can fall back to a stricter, rules-based check with tighter transaction limits rather than blocking all payments outright. If the Realtime Gateway is overloaded or unavailable, clients silently fall back to periodic polling of the (still-functioning) balance endpoint, so the group can still see approximately correct progress even without live push updates. If the Notification Service queue backs up, payments continue to be captured normally — reminder emails and push notifications are allowed to be late, but a payment must never be blocked waiting on a notification to send. This layered degradation is only possible because of the architectural separation described in Section 3: the parts of the system that are allowed to be “best effort” are structurally incapable of blocking the parts that must be “always correct.”

10

Security

  • PCI-DSS scope minimization: raw card data never touches internal services — it goes directly from the client SDK to the PSP (tokenization), and internal systems only ever see a token/reference, never the PAN or CVV.
  • Idempotency + replay protection: idempotency keys prevent duplicate charges; all mutating APIs require short-lived, signed request tokens to prevent replay attacks.
  • Authorization boundaries: a participant can only view and pay their own Payable; only the organizer (or an authorized admin) can view aggregate group financials or cancel a split. This is enforced at the API Gateway and re-verified in the service layer (defense in depth).
  • Fraud & velocity checks: the Payment Orchestrator calls a fraud-scoring service before authorization — checking velocity (many splits created rapidly), device fingerprinting, and known bad actors — since a fan-out payment system is an attractive target for card-testing fraud (attackers using stolen cards to make many small “share” payments to test validity).
  • Encryption: TLS in transit everywhere; ledger and PII fields encrypted at rest, with field-level encryption for particularly sensitive data (bank account numbers, KYC documents).
  • Webhook verification: every inbound webhook from the PSP is verified via signature (HMAC) before being trusted, to prevent spoofed “payment succeeded” callbacks.
  • Audit logging: every state transition (split created, edited, cancelled; payment authorized, captured, refunded) is logged immutably with actor identity, supporting both security forensics and financial audits.
💬
What an interviewer may ask

“How would you prevent a malicious participant from paying someone else’s share to launder or test stolen card details?” — Bind each payment attempt to the authenticated participant_id of the logged-in user matching the Payable’s owner, apply the same fraud/velocity scoring as any standalone payment, and rate-limit payment attempts per user and per split.

10.1 Handling Disputes and Chargebacks

Because each participant pays with their own instrument, chargebacks (a cardholder disputing a charge with their bank) arrive per-Payable rather than per-split, which is actually a helpful property — the blast radius of a single dispute is naturally contained to one person’s share rather than the whole group’s payment. When a chargeback notification arrives from the PSP, the system records a new offsetting ledger entry (never mutating the original capture entry), transitions the affected Payable to a distinct Disputed sub-state, and — depending on product policy — either pauses disbursement of the overall split until resolved (if the split hasn’t yet reached FullyFunded) or treats it as a post-settlement adjustment requiring recovery from the organizer or an internal loss reserve. Surfacing this clearly to the organizer, rather than silently absorbing it, is both a compliance expectation and a trust-building product decision.

10.2 Data Privacy Considerations

Group financial apps sit at an unusual privacy intersection: participants can see that a friend owes money and roughly when they paid, but should never see another participant’s underlying payment instrument details, bank routing information, or full transaction metadata. The API layer enforces field-level authorization — a “split status” response includes a participant’s name and payment state, but PaymentAttempt and LedgerEntry details are only ever visible to the paying participant themselves (and to authorized internal systems), never to other group members or even the organizer.

11

Monitoring, Logging & Metrics

11.1 Key Metrics

MetricWhy it matters
Payment success rate (by PSP, method, region)Detects gateway degradation or a specific card network issue quickly.
Capture-to-ledger-write latencyTracks health of the outbox/event pipeline; a growing lag signals backpressure.
Ledger vs. PSP reconciliation drift (count & amount)The single most important financial-correctness metric — should be zero; any nonzero value pages an on-call engineer.
Split completion rate & time-to-fully-fundedProduct health signal — are splits actually getting paid, or stalling?
Realtime fan-out delivery latencyUX quality signal for the live view feature.
Idempotency key collision/reuse rateHelps catch client bugs or retry storms early.

11.2 Observability Stack

Distributed tracing (for example, using a standard like OpenTelemetry) is essential here because a single “user pays their share” action touches many services — gateway, orchestrator, PSP, ledger, event bus, wallet — and a trace ID propagated through all of them is the only practical way to debug a slow or failed payment in production. Structured logs (JSON, with split_id/payable_id/trace_id on every line) feed a centralized log store, and dashboards separate “business” metrics (completion rate, GMV collected) from “system” metrics (latency, error rate, queue depth), because these two audiences (product/finance vs. engineering on-call) need different views of the same system.

Alerting should be tiered: reconciliation drift and repeated capture failures page immediately (financial correctness), while elevated realtime-delivery latency might only warrant a ticket, since it degrades UX but never money-safety.

12

Deployment & Cloud Architecture

Each core service (Split, Group, Payment Orchestrator, Ledger, Wallet, Notification, Reconciliation) is deployed as an independently scalable microservice, typically as containers orchestrated by Kubernetes, with horizontal pod autoscaling driven by request rate and queue depth rather than CPU alone (since payment orchestration is often I/O-bound waiting on PSP responses).

  • Multi-region active-active for stateless services (API Gateway, Split Service reads, Realtime Gateway) to minimize latency for a global user base.
  • Ledger DB as active-passive with regional leader — writes go to a single regional primary per data-residency/regulatory boundary (important, since payment data often has legal residency constraints), with synchronous in-region replicas and async cross-region replicas for DR.
  • Blue-green or canary deployments for the Payment Orchestrator specifically, given its criticality — new versions serve a small percentage of traffic first, monitored against the reconciliation-drift metric before full rollout.
  • Infrastructure as code for reproducible environments, and strict separation of the PCI-scoped subsystem (anything touching card tokens) into its own network segment/VPC with tightly controlled ingress/egress.
  • Cost optimization: the event stream and analytics consumers can run on spot/preemptible compute since they are not on the critical payment path, while the Ledger and Payment Orchestrator run on reserved, highly available capacity.
13

Databases, Caching & Load Balancing

13.1 Database Choices

StoreGood fitReasoning
Ledger DBStrongly consistent relational DB (for example, PostgreSQL, or a distributed SQL system like Spanner or CockroachDB for global scale)ACID transactions and strong consistency are non-negotiable for double-entry accounting.
Split Metadata DBRelational DB with read replicas, or a document store if split configurations are highly variableRead-heavy, moderately consistent; benefits from flexible querying (by group, by user, by date).
Balance CacheRedis / in-memory KV storeSub-millisecond reads for the live view; sharded counters live here.
Event LogKafka (or equivalent log-based broker)Durable, ordered, replayable stream; the backbone for async fan-out and for rebuilding aggregates from scratch if needed.
Analytics / SearchColumnar warehouse (for example, for GMV reporting) plus a search index for “find my splits”Optimized for aggregate queries over time, decoupled from the transactional path.

13.2 Sharding Strategy

The Ledger and Payables tables are sharded by a hash of participant_id (or a combination of participant_id and split_id) so that a single user’s payment history stays co-located for fast lookups, while ensuring no single split — however large — can overload one shard, since its Payables are naturally distributed across many participants and therefore many shard keys.

flowchart TB ROUTER[Shard Router hash of participant id] ROUTER –> S1[Shard 1] ROUTER –> S2[Shard 2] ROUTER –> S3[Shard 3] ROUTER –> S4[Shard N] S1 –> R1[Replica] S2 –> R2[Replica] S3 –> R3[Replica] S4 –> R4[Replica]
Fig. 13.1 — Sharding: participant-id hash routes each Payable and its ledger entries to its own shard plus replica.

13.3 Caching Strategy

Balances use a cache-aside pattern: reads check Redis first, fall back to a recompute-from-ledger path on a miss, and writes update the cache asynchronously as ledger events stream in. A short TTL acts as a safety net, so even if an event is somehow missed, the cache self-heals within seconds rather than serving stale data indefinitely. Split metadata (rarely changing once Active) uses a longer TTL with explicit invalidation on edit/cancel events.

13.4 Load Balancing

Layer-7 (application) load balancing at the API Gateway routes by path and applies weighted routing for canary releases. Within the Payment Orchestrator tier, requests are load balanced with awareness of PSP connection pools, since PSP-side rate limits mean an even spread of outbound calls matters as much as inbound request distribution — a burst of traffic to a single orchestrator instance can otherwise exhaust its PSP connection pool and cause cascading timeouts.

14

APIs & Microservices

14.1 Core API Surface (Conceptual)

Endpoint (conceptual)Purpose
Create SplitOrganizer defines total, method, participants; returns split_id and each Payable.
Get Split StatusReturns current aggregate state — collected amount, per-participant status. Read-optimized, served from cache/materialized view.
Pay ShareParticipant-initiated; requires idempotency key; kicks off the Payment Orchestrator flow.
Cancel / Refund SplitOrganizer or admin-initiated; triggers refund workflow for all captured Payables.
Subscribe to Split UpdatesWebSocket/SSE subscription for realtime balance changes.
Reconciliation Report (internal)Used by the Reconciliation Service and finance teams to compare ledger vs. PSP state.
Illustrative “Pay Share” request and response
POST /v1/splits/{split_id}/payables/{payable_id}/pay
Idempotency-Key: pay_9b21a7ce-88de-4c1f-9a1a-c0f2
Authorization: Bearer <participant token>
{
  "amount_cents":     3334,
  "currency":         "USD",
  "payment_method":   { "type": "card_token", "token": "tok_visa_4242" }
}

Response 202 Accepted:
{
  "payable_id":       "pay_5f7d…",
  "attempt_id":       "att_2e14…",
  "status":           "AUTHORIZING",
  "poll_url":         "/v1/payables/pay_5f7d/status"
}

Later, PSP webhook (verified via HMAC signature):
{
  "event":            "payment.captured",
  "attempt_id":       "att_2e14…",
  "psp_reference":    "ch_1P8x…",
  "amount_cents":     3334,
  "captured_at":      "2026-08-11T18:24:11Z"
}

14.2 Microservices Boundaries

Service boundaries follow the single-responsibility-per-bounded-context principle from domain-driven design: Group and Split are “social” bounded contexts that can evolve quickly with product experiments; Payment Orchestrator, Ledger, and Reconciliation are “financial” bounded contexts that change slowly, deliberately, and under stricter review. This boundary is not just organizational — it is enforced technically, with the financial services exposing a narrow, versioned, backward-compatible API and owning their own data stores, so a schema change in the Split Service can never accidentally corrupt ledger data.

Communication between the two worlds is deliberately asynchronous and event-driven wherever possible (Split Service publishes events, Payment services react), with synchronous calls reserved for the few paths that genuinely need an immediate answer (for example, “pay my share” needs a synchronous response to the mobile client, even though its downstream effects fan out asynchronously).

💬
What an interviewer may ask

“Would you use REST, gRPC, or GraphQL here?” — Typically gRPC or REST with strict schemas for internal service-to-service calls (performance and strong typing matter for the payment path), REST/GraphQL for external-facing client APIs (flexibility for evolving mobile/web UIs), and an event schema registry (for example, Avro/Protobuf with a schema registry) for the Kafka-based event backbone to prevent breaking changes from silently corrupting downstream consumers.

15

Design Patterns & Anti-Patterns

15.1 Patterns Applied

Pattern

Saga

A split’s full lifecycle (authorize → capture → ledger write → escrow → eventual disbursement) is a long-running, multi-step process across service boundaries — modeled as a saga with compensating actions (for example, a refund saga if disbursement fails after funds were escrowed).

Pattern

Outbox

Guarantees that a local database write (for example, marking a Payable captured) and the corresponding event publish (to notify the ledger and downstream consumers) happen atomically, avoiding the classic “dual write” bug where a service crashes after updating its DB but before publishing the event.

Pattern

CQRS

Command Query Responsibility Segregation: writes go through the strongly consistent Ledger/Payable path; reads (the live balance view) go through a separate, denormalized, eventually-consistent read model optimized for speed.

Pattern

Idempotent Receiver

Every mutating endpoint and webhook handler is designed to safely process the same request or event more than once without side effects.

Pattern

Circuit Breaker

Calls to the PSP and fraud service are wrapped in circuit breakers so a degraded external dependency fails fast and predictably rather than cascading into orchestrator thread and connection exhaustion.

15.2 Anti-Patterns to Avoid

Anti-patternWhy it’s dangerous
Single mutable “amount_collected” column updated in-place per paymentCreates a hot-row bottleneck and is the single most common scaling mistake in this domain.
Using floating-point numbers for moneyLeads to rounding drift that compounds across millions of transactions; always use integer minor units.
Synchronous, blocking notification or realtime calls on the payment critical pathA slow push-notification provider should never be able to slow down or fail a payment capture.
Treating PSP webhooks as fire-and-forget with no signature verificationOpens the door to spoofed “payment succeeded” events.
Deleting or mutating ledger rows to “fix” a mistakeAlways correct with a new offsetting entry; never rewrite history in a financial ledger.
Coupling the Group/Social schema directly to the Ledger schemaA foreign key from ledger rows straight into mutable group tables creates fragile coupling between fast-moving product code and slow-moving financial code.
16

Best Practices & Common Mistakes

16.1 Best Practices

  • Always compute split shares using deterministic, testable rounding logic (largest-remainder method), and unit-test it exhaustively against edge cases (splits that don’t divide evenly, single-cent totals, very large participant counts).
  • Design every payment-related API to be idempotent by default, not as an afterthought bolted on later.
  • Separate “eventually consistent, fast” read paths from “strongly consistent, source-of-truth” write paths explicitly, and document which guarantee each API endpoint provides.
  • Build the Reconciliation Service on day one, not as a v2 feature — silent financial drift is one of the hardest bugs to catch retroactively.
  • Version your event schemas from the start; a Kafka topic with breaking, unversioned schema changes will eventually cause a painful outage.
  • Make refund and expiry policies explicit product decisions early (what happens to a partially-funded split that expires?), since retrofitting this logic after launch is painful and error-prone.

16.2 Common Mistakes

  • Assuming all participants will pay promptly — real-world splits often sit partially funded for days; the system must handle this gracefully, not treat it as an error state.
  • Forgetting that a participant can leave a group, be removed, or dispute a charge after paying — requiring careful handling of orphaned Payables and chargebacks.
  • Under-provisioning the realtime fan-out layer, assuming groups stay small, then being surprised when a product feature (for example, “public group buys”) creates thousand-person splits.
  • Neglecting currency and locale correctness — different currencies have different minor-unit precision (for example, some currencies have no minor unit at all), and naive “divide by 100” logic breaks for these.
17

Real-World / Industry Examples

Example

Group Expense Apps

Consumer bill-splitting apps popularized the “who owes whom” ledger model, using a debt-simplification algorithm to minimize the number of settling transactions in a group — a graph problem where each unpaid balance is an edge, and the goal is to find the minimum set of transfers that zeroes out the graph.

Example

Event Ticketing “Group Pay”

Large ticketing platforms let one buyer reserve a group of seats and invite others to pay their own portion within a time-boxed hold, mirroring the Payable expiry mechanism described in Section 5 — unpaid seats release back to inventory rather than blocking the whole group’s purchase.

Example

Food Delivery Group Orders

Group ordering features in food delivery apps let each person in an office or household add items and pay for their own portion of a shared cart, with the merchant receiving one consolidated order once all portions are paid or the organizer closes the cart — directly analogous to the escrow-and-disbursement model in Section 4.3.

Example

Crowdfunded / Pooled Payments

Pooled-fund and crowdfunding platforms use “all-or-nothing” or “keep-it-all” funding models, which map directly onto the split lifecycle’s expiry and auto-refund transitions shown in Section 5.1.

Example

Ride-Hailing Fare Splitting

Ride-hailing apps that let multiple riders in the same trip split the fare demonstrate the independent-Payable model at its simplest: each rider’s portion is charged to their own saved payment method at the end of the trip, and a failure to charge one rider does not reverse or block the successful charges already made to the others — precisely the independence principle described in Section 4.2.

Example

Shared Subscription Billing

Family or group subscription plans (streaming services, cloud storage) that let each member pay their own portion of a shared plan need the same recurring version of this system: instead of a one-time split, the Split Service creates a new set of Payables on each billing cycle, and the Payment Orchestrator’s retry and dunning logic (retrying failed recurring charges before removing a member’s access) becomes an extension of the failure-recovery flows described in Section 9.

18

FAQ, Summary & Key Takeaways

Q1

Why not just charge the organizer the full amount and let them collect from friends offline?

Because that reintroduces exactly the manual, error-prone, socially awkward process the product is trying to eliminate, and it concentrates all financial and fraud liability on one person. The independent-Payable model spreads both the payment action and its risk across each actual participant.

Q2

How do you guarantee the sum of individual shares always equals the original total, given rounding?

Deterministic integer-cent arithmetic plus the largest-remainder method: compute truncated shares, then distribute leftover minor units one by one, in a stable order, until the sum matches exactly.

Q3

What happens if a split never gets fully funded?

This must be an explicit, product-defined policy: either the split allows partial disbursement once a deadline passes (release what was collected), or it enforces all-or-nothing funding with automatic refunds of any captured shares on expiry — both are legitimate designs, but the system must implement whichever is chosen deterministically and transparently to users.

Q4

How is this different from a simple “split the bill” calculator?

A calculator only computes who-owes-what; this system additionally collects, tracks, and reconciles real money movement for each individual, independently, with the full rigor (idempotency, ledger integrity, fraud checks, HA) expected of any production payment system.

Q5

How would you extend this design to support multiple currencies within the same group?

Store the split’s total and every Payable in a single canonical currency chosen at creation time, and let each participant’s payment instrument charge in their local currency via the PSP’s built-in conversion, recording the exact conversion rate and both amounts (local and canonical) on the PaymentAttempt so the ledger entry itself always stays in one consistent currency — never let the ledger mix currencies within a single split’s accounting.

Q6

What if the organizer edits the total amount after some participants have already paid?

Treat this as a controlled state transition, not a silent update: recompute shares for unpaid participants only, leave already-captured Payables untouched, and if the new total is lower than what’s already collected, trigger a partial refund workflow for the overage — always through new offsetting ledger entries, never by editing history.

Q7

How do you test a system like this before it touches real money?

Run the full orchestrator and ledger logic against a sandboxed PSP environment that can simulate authorization failures, timeouts, and delayed webhooks on demand, and maintain a dedicated reconciliation test suite that intentionally injects drift (a missed webhook, a duplicate event) to verify the Reconciliation Service actually catches it — testing the unhappy paths is far more valuable here than testing the happy path.

18.1 Key Takeaways

  • Model a split as a fan-out of independent Payables, each with its own state machine — never as one payment with shared mutable state.
  • Separate the strongly consistent, append-only Ledger (source of truth) from the eventually consistent, fast Wallet/Balance read view.
  • Use integer minor-unit arithmetic and the largest-remainder rounding method to keep splits mathematically exact.
  • Idempotency, the outbox pattern, and circuit breakers are what make the payment critical path safe under retries and partial failures.
  • Reconciliation against the PSP is not optional — build it from day one to catch silent drift before it becomes a financial or trust problem.
  • Scale the write path via sharding and asynchronous aggregation, and the realtime path via per-split pub/sub fan-out rather than naive broadcast.
📌
The one idea to remember

A split payment is not one transaction. It is N independent transactions sharing a parent context. Every hard problem in this design — scaling, correctness, refunds, fraud, live view — gets easier the moment you internalize that, and gets harder the moment you forget it.