System Design: Currency Conversion Engine for International Remittance

Currency Conversion Engine for International Remittance

Currency Conversion Engine for International Remittance

How to design a system that shows a customer an exchange rate and then guarantees, at massive scale, that the money actually moves at that exact rate — covering architecture, rate locking, consistency, and a million-requests-per-minute scaling plan.

01

Introduction & History

Sending money from one country to another sounds simple: you type in an amount, pick a destination, and press send. But hidden inside that single button press is one of the hardest problems in financial engineering — converting one currency into another, at a price that is constantly moving, while promising the customer that the price they saw is the price they get. This guide designs that system from the ground up, for a remittance company that might be sending money for a student paying college fees in Canada, a construction worker in the Gulf sending savings home to the Philippines, or a small business paying an overseas supplier.

Currency conversion itself is an old idea. Money changers have existed since ancient trade routes connected Rome, Persia, and China, physically swapping coins of different empires at rates they decided on the spot. The modern foreign exchange (FX) market, where currencies are traded electronically between banks, was born after 1971 when major currencies stopped being pegged to gold and started floating freely against each other. Since then, exchange rates move every second based on interest rates, trade flows, political events, and pure supply and demand. A rate that was true when you opened this article may already be stale by the time you finish reading this sentence.

Digital remittance companies such as Wise, Remitly, Xoom, and WorldRemit built their businesses on top of this constantly-moving market. Their core promise to a customer is trust: “the rate we show you is the rate you get.” Delivering on that promise while operating at global scale — potentially processing millions of quote requests every minute during peak events like salary day in the Gulf or festival season in South Asia — is a genuinely hard distributed systems problem, and it is exactly the kind of problem interviewers love to ask about because it touches consistency, caching, concurrency, financial correctness, and extreme scale all at once.

Everyday analogy

Think of an airline showing you a ticket price. You see “$450” on the screen, but airline prices change by the minute depending on demand. If the airline let the final charge drift from the price you saw, you would never trust that website again. Airlines solve this by “holding” a fare for a few minutes while you complete checkout. A remittance company must do the exact same thing with an exchange rate — hold it for a short window, and honor it exactly if you confirm within that window.

1.1 From wire transfers to real-time apps

For most of the 20th century, sending money abroad meant a bank-to-bank wire transfer through the SWIFT messaging network, a process that could take three to five business days and involved intermediary “correspondent” banks each taking a cut, with the exchange rate typically decided unilaterally by the sending bank and disclosed only after the fact. The first generation of digital remittance startups in the 2010s didn’t just move the same process onto a website — they rebuilt the pricing model itself, making the exchange rate visible and quoted upfront as a core product feature, which is what turned currency conversion from a back-office banking function into a customer-facing, real-time system that has to be fast, transparent, and provably honest.

Pre-1971

Gold-pegged Rates

Major currencies were tied to gold, and exchange rates barely moved. Currency conversion was a slow, largely bureaucratic operation between banks.

1970s–2000s

SWIFT-era Bank Wires

Free-floating currencies meant live FX markets, but retail customers still saw only after-the-fact rates on multi-day bank wires routed through correspondent banks.

2010s+

Real-time Remittance Apps

Wise, Remitly, and peers surfaced live rates as a product feature, backed by streaming FX ingestion, in-memory rate caches, and short-lived rate locks — the pattern this guide unpacks.

1.2 Why this system matters for an architect

This is not a toy CRUD problem. A currency conversion engine sits at the intersection of five hard sub-problems that architects are expected to reason about together:

  • Real-time data ingestion — pulling live FX rates from multiple providers, often tens of times per second per currency pair.
  • Strong consistency for money — the rate quoted and the rate applied must be provably identical, not “close enough.”
  • Extreme read scale — quote requests can be a much higher volume than actual transactions, since customers check rates repeatedly before deciding to send.
  • Regulatory and audit requirements — every quote and every executed transaction must be logged immutably for compliance, tax, and dispute resolution.
  • Global low latency — customers in India, Philippines, Kenya, Mexico, and the US all expect sub-second quote responses.
02

Problem & Motivation

Let’s state the core problem precisely, because precision is what separates a strong system design answer from a vague one.

📌
Problem statement

Design a system for an international remittance service that (a) shows customers a live currency exchange rate when they request a quote, (b) allows the customer some time to review and confirm the transfer, and (c) guarantees that the money is actually converted at exactly the rate that was shown — never a different rate — even though the real market rate is changing continuously and the platform may be serving on the order of a million requests per minute globally.

2.1 Why “just use the live rate” doesn’t work

A naive design fetches the current market rate at the exact moment the customer clicks “confirm” and applies it. This seems simple, but it breaks the fundamental promise: the number the customer approved and the number they get charged can differ, sometimes significantly during volatile market moments (a central bank rate announcement can move a currency pair by 1–2% in seconds). Customers experiencing “rate slippage” — a worse rate at execution than at quote time — lose trust immediately and often escalate to regulators, since remittance is a regulated financial service in almost every country.

2.2 Two real problems hiding inside one

When you unpack “show the rate and honor the rate,” you actually find two distinct engineering problems, and good candidates in an interview separate them explicitly:

Sub-problemWhat it really isPrimary technique
Rate DiscoveryGetting an accurate, current market rate from external FX liquidity providers, fast, and at huge read volumeMulti-provider ingestion + in-memory caching + fan-out
Rate GuaranteeFreezing a specific rate for a specific customer for a short, bounded window, and proving at execution time that the frozen rate was honoredRate locking with TTL + idempotent execution + immutable audit trail

2.3 Business constraints that shape the design

  • Quote validity window: typically 15 to 60 seconds. Long enough for a customer to review, short enough that the company’s own currency risk exposure stays bounded (the company itself has to buy/sell currency in the wholesale FX market, and if the customer’s locked rate is stale relative to the real market, the company absorbs the difference).
  • Margin / markup: the rate shown to the customer is the wholesale mid-market rate plus a margin (the company’s revenue). This margin calculation must also be locked along with the rate.
  • Regulatory audit: regulators (like FinCEN in the US, FCA in the UK, RBI in India) require proof of what rate was quoted, when, and what was executed — often for 5–7 years.
  • Idempotency: a customer’s app might retry a “confirm” call due to a flaky mobile network. The same locked quote must never be executed twice.
  • Scale: during high-traffic windows (payday, festivals, market volatility spikes driving people to check rates repeatedly), the quote-fetching path can see enormous read traffic even though the actual money-moving transactions are a much smaller fraction of that traffic.
~1M/minPeak global request volume
15–60sQuote validity TTL window
95%+Read (quote) share of total traffic
5–7 yrsRegulatory audit retention
💬
What an interviewer may ask

“What happens if the customer confirms one millisecond after the quote expires?” A strong answer: the system must treat expiry as a hard boundary checked server-side (never trust client-side timers), and on expiry, the customer must be shown a fresh quote and asked to reconfirm — the system never silently executes at a rate the customer didn’t explicitly approve.

03

Architecture & Components

Before diving into individual pieces, here is the full system laid out end to end. Every box below names both its function and its role in the request path (gateway, load balancer, cache, queue, and so on), because in an interview it is not enough to draw circles — you must be able to say what kind of component each box is and why it sits where it sits.

flowchart TB subgraph CLIENT[“Client Layer”] MOB[“Mobile App Client”] WEB[“Web App Client”] PART[“Partner Integration Client”] end subgraph EDGE[“Edge and Gateway Layer”] CDN[“CDN Edge Cache”] LB[“Load Balancer L4 and L7”] GW[“API Gateway Auth RateLimit Routing”] end subgraph APP[“Application Service Layer”] AUTH[“Auth Service”] QUOTE[“Quote Service”] LOCK[“Rate Lock Service”] ORCH[“Transaction Orchestrator”] LEDGER[“Ledger Service”] PAY[“Payment Execution Service”] NOTIFY[“Notification Service”] end subgraph RATE[“Rate Ingestion Layer”] FEED1[“FX Provider Feed A”] FEED2[“FX Provider Feed B”] INGEST[“Rate Ingestion Workers”] RATECACHE[“Rate Cache Redis Cluster”] end subgraph DATA[“Data and Messaging Layer”] KAFKA[“Message Queue Kafka”] LOCKSTORE[“Quote Lock Store Redis TTL”] PGPRI[“Primary Database Postgres Sharded”] PGREP[“Read Replica Database”] AUDIT[“Audit Log Store”] end subgraph OBS[“Observability Layer”] METRICS[“Metrics Prometheus”] TRACE[“Distributed Tracing Jaeger”] LOGS[“Centralized Logging”] end MOB –> CDN WEB –> CDN PART –> LB CDN –> LB LB –> GW GW –> AUTH GW –> QUOTE QUOTE –> RATECACHE RATECACHE –> LOCK LOCK –> LOCKSTORE QUOTE –> ORCH ORCH –> LEDGER ORCH –> PAY ORCH –> KAFKA KAFKA –> NOTIFY LEDGER –> PGPRI PGPRI –> PGREP ORCH –> AUDIT FEED1 –> INGEST FEED2 –> INGEST INGEST –> RATECACHE GW –> METRICS ORCH –> TRACE ORCH –> LOGS
Figure 3.1 — End-to-end architecture. Every box is labeled with its component role so the diagram doubles as a component inventory.

3.1 Component-by-component breakdown

Edge

CDN

Serves static app assets and, for public marketing rate displays, can cache a slightly delayed indicative rate for a few seconds. Never serves the authoritative lockable quote — that always goes to origin. Offloads a huge fraction of anonymous, non-transactional traffic away from the core system.

Edge

Load Balancer

Sits at Layer 4 for raw connection distribution and Layer 7 for smarter routing — path-based routing, health-check failover, connection draining. At a million requests per minute the LB tier itself must be horizontally scaled and geographically distributed.

Edge

API Gateway

The single front door for all client traffic. Handles authentication (JWT/OAuth), coarse-grained rate limiting, request routing, schema validation, and TLS termination. Deliberately kept “thin” — no business logic about currency lives here.

App

Auth Service

Validates customer identity and session, and, critically for a regulated fintech, enforces KYC/AML status before a quote can proceed to an executable transaction.

App

Quote Service

The heart of rate discovery. Given a currency pair and an amount, it reads the current rate from the in-memory Rate Cache, applies markup logic, and returns a quote containing the locked rate, resulting destination amount, and expiry timestamp.

App

Rate Lock Service

The heart of rate guarantee. Takes the quote produced above and writes it into a fast key-value store with a TTL exactly matching the quote’s validity window. After this point nothing about that quoteId’s rate can change.

Data

Quote Lock Store

Redis cluster holding quoteId to {rate, amount, currencyPair, expiryTime, customerId} with a native TTL. When the TTL expires, Redis evicts the key automatically — an elegant way to enforce expiry without a background cleanup job.

App

Transaction Orchestrator

When the customer hits confirm, this service receives the quoteId, looks it up in the Lock Store, and if (and only if) it is still present and unexpired, drives the multi-step process of actually moving money. Turns a promise (the quote) into a fact (the transaction).

App

Ledger Service

An append-only, double-entry accounting system recording every debit and credit. Intentionally separate from the business database because ledgers have unique correctness requirements (they must always balance to zero) and are usually audited independently.

App

Payment Execution Service

Talks to banking rails, card networks, or local payout partners to actually disburse funds in the destination currency, at the locked rate. Also where the company’s own FX hedge and settlement with liquidity providers happens.

Rate

Rate Ingestion Workers

Continuously pull or subscribe to live rates from multiple external FX data providers (Bloomberg, Refinitiv, XE, or direct bank liquidity feeds), normalize them, run sanity checks, and push accepted rates into the Rate Cache.

Data

Rate Cache

Redis cluster holding the latest accepted mid-market rate per currency pair, refreshed continuously. This is what lets the Quote Service answer in single-digit milliseconds instead of calling an external FX provider on every request.

Data

Message Queue (Kafka)

Decouples the orchestrator from downstream consumers: notifications, analytics, compliance screening, and reconciliation jobs all subscribe to the same transaction-executed and rate-updated event streams independently.

Data

Primary DB & Replicas

A sharded relational store (commonly PostgreSQL, sharded by customer region or ID) holding customer transaction history, account details, and reference data. Read replicas absorb reporting and history-lookup traffic away from the primary.

Data

Audit Log Store

An immutable, write-once store (append-only table or object storage like S3 with object lock) capturing every quote shown and every transaction executed, including the exact rate, timestamp, and provider source — the regulator’s proof of “what rate was quoted, what was executed.”

Ops

Observability Stack

Metrics (Prometheus/Grafana), distributed tracing (Jaeger/OpenTelemetry), and centralized logging (ELK/Loki) — covered in depth in Section 10.

💬
What an interviewer may ask

“Why is the Rate Lock Store separate from the Rate Cache?” Good answer: they have fundamentally different write patterns and correctness requirements. The Rate Cache is updated continuously by the system (many writers, one logical value per pair) and is allowed to change. The Lock Store is written once per quote and must never change after that — it’s a per-customer, per-quote snapshot, not a shared mutable value. Conflating them would risk a rate lock silently drifting when the cache refreshes.

04

Internal Working

Let’s trace exactly what happens inside the system, component by component, for the two operations that matter most: getting a quote, and executing a transfer against that quote.

4.1 How a quote is built

  1. Client sends GET /quotes?from=USD&to=INR&amount=1000 through the CDN and Load Balancer to the API Gateway.
  2. API Gateway authenticates the request and forwards it to the Quote Service.
  3. Quote Service reads the current mid-market rate for USD to INR from the Rate Cache — an in-memory read, typically under 2 milliseconds.
  4. Quote Service applies markup logic: finalRate = midMarketRate * (1 - marginPercent) for the customer-facing sell rate (the exact formula depends on whether the company earns margin by marking the rate up or down, and can include tiered pricing).
  5. Quote Service generates a unique quoteId (UUID) and computes an expiry timestamp (now + TTL, typically 15–60 seconds).
  6. Quote Service calls Rate Lock Service to persist {quoteId, fromCurrency, toCurrency, sourceAmount, lockedRate, destinationAmount, expiresAt, customerId} into the Lock Store with a matching TTL.
  7. Response returned to the client with the locked rate and a countdown timer for the UI.

4.2 How a transfer is executed against a locked quote

  1. Client sends POST /transfers {quoteId, idempotencyKey}.
  2. Transaction Orchestrator looks up quoteId in the Lock Store.
  3. If the key is missing (expired or never existed), the orchestrator rejects with a specific QUOTE_EXPIRED error — never falls back to “just use current rate” silently.
  4. If found, the orchestrator checks the idempotency key against a short-lived idempotency table to guard against duplicate submissions.
  5. The orchestrator opens a distributed transaction workflow (often a saga) spanning: debit source account in Ledger Service, call Payment Execution Service to disburse in destination currency at the exact locked rate, and mark the quote as consumed (deleting it from the Lock Store to prevent replay even within the TTL window).
  6. On success, a TransferExecuted event is published to Kafka containing the locked rate that was applied, alongside the original quoted rate for automated reconciliation.
  7. Notification Service consumes the event and sends the customer a receipt confirming the exact rate — the customer-facing proof that quote and execution matched.

4.3 Idempotency: why it matters here specifically

Mobile networks are unreliable. A customer’s confirm tap might time out client-side and get retried automatically, even though the server actually processed it. Without idempotency protection, this could debit the customer twice for one transfer. The fix is a client-generated idempotency key stored alongside the transaction result for a window (typically 24 hours); if the same key arrives again, the orchestrator returns the original result instead of re-executing.

TransactionOrchestrator.java — idempotency check and atomic lock claim
public TransferResult executeTransfer(String quoteId, String idempotencyKey) {

    Optional<TransferResult> existing = idempotencyStore.find(idempotencyKey);
    if (existing.isPresent()) {
        // Same request seen before - return original outcome, do not re-execute
        return existing.get();
    }

    LockedQuote quote = lockStore.get(quoteId)
        .orElseThrow(() -> new QuoteExpiredException(quoteId));

    // Atomically claim the quote so two concurrent requests can never
    // both execute against the same quoteId (Redis GETDEL semantics)
    boolean claimed = lockStore.deleteIfPresent(quoteId);
    if (!claimed) {
        throw new QuoteAlreadyConsumedException(quoteId);
    }

    TransferResult result = runTransferSaga(quote);
    idempotencyStore.save(idempotencyKey, result, Duration.ofHours(24));
    return result;
}

Notice the deleteIfPresent call: this is doing double duty. It reads the locked quote and atomically removes it in one operation (Redis’s GETDEL command, for example), so that if two requests race to consume the same quoteId — say, a retried request arriving a few milliseconds after the original — only one of them can win the claim. The other gets a clean, explicit error instead of silently re-executing.

💬
What an interviewer may ask

“Two requests hit the orchestrator with the same quoteId at nearly the same instant — what stops a double-spend?” The answer they’re looking for: atomic claim-and-delete on the lock store (not check-then-delete as two separate steps, which has a race condition), combined with idempotency keys for the client-retry case specifically.

4.4 Rate ingestion internals

Rate Ingestion Workers subscribe to multiple external providers in parallel (never just one — see Section 8 on HA). Each incoming rate update goes through a sanity filter before being trusted:

RateSanityFilter.java — rejects implausible single-tick jumps
public boolean isRateSane(CurrencyPair pair, BigDecimal newRate) {
    BigDecimal previousRate = rateCache.getLastAccepted(pair);
    if (previousRate == null) {
        return true; // first rate seen for this pair, accept it
    }

    BigDecimal percentChange = newRate.subtract(previousRate)
        .abs()
        .divide(previousRate, 6, RoundingMode.HALF_UP);

    // Reject implausible single-tick jumps (likely a bad feed),
    // route to alerting and fall back to the secondary provider
    return percentChange.compareTo(MAX_ALLOWED_TICK_DEVIATION) <= 0;
}

4.5 Double-entry ledger internals

The Ledger Service never records a single number like “customer sent $1000.” Following standard double-entry accounting, every transfer produces at least two balanced entries — a debit and a credit — so the books always sum to zero and any single missing or duplicated entry is immediately detectable.

LedgerService.java — atomic double-entry write inside one DB transaction
@Transactional
public void recordTransfer(LockedQuote quote, String transferId) {
    LedgerEntry debit = LedgerEntry.builder()
        .transferId(transferId)
        .account(quote.getCustomerSourceAccount())
        .currency(quote.getFromCurrency())
        .amount(quote.getSourceAmount().negate())
        .rateApplied(quote.getLockedRate())
        .build();

    LedgerEntry credit = LedgerEntry.builder()
        .transferId(transferId)
        .account(quote.getDestinationPayoutAccount())
        .currency(quote.getToCurrency())
        .amount(quote.getDestinationAmount())
        .rateApplied(quote.getLockedRate())
        .build();

    ledgerRepository.saveAll(List.of(debit, credit));
    // Both rows commit together or not at all - the ledger can never
    // be left in a half-written, unbalanced state.
}

4.6 CAP theorem: where each store sits

Every distributed component in this design makes a deliberate, explicit choice about where it sits on the CAP spectrum. Partition tolerance is non-negotiable in any real distributed system, making the practical choice really about consistency versus availability during a partition:

ComponentCAP choice during a partitionWhy this is the right choice here
Rate CacheAvailability-favoring (AP)Serving a slightly stale rate for a few hundred milliseconds is far better than refusing to serve a quote at all; freshness is best-effort, not must-be-exact
Quote Lock StoreConsistency-favoring (CP) for the write; availability-favoring for readsA lock write must not be lost or duplicated (two nodes disagreeing about whether a quoteId is claimed is unacceptable), so the write path prioritizes consistency even at some availability cost
Ledger DatabaseStrongly consistent (CP), full ACIDMoney must never be double-counted or lost; the ledger is the one place in the system where availability is knowingly sacrificed during a partition rather than risk an inconsistent balance
Audit LogConsistency-favoring, append-onlyRegulatory requirement for tamper-evidence outweighs any availability concern

4.7 Concurrency control

The Lock Store’s atomic claim-and-delete operation is a form of optimistic concurrency control: rather than acquiring a heavyweight distributed lock before every read, the system lets many readers check a quote freely, and only enforces exclusivity at the single moment of consumption, using an atomic primitive provided by the store itself (Redis’s single-threaded command execution model guarantees that GETDEL cannot be interleaved with another client’s identical call on the same key). This keeps the hot read path completely lock-free while still guaranteeing exactly-once consumption at execution time.

4.8 Consensus for multi-node coordination

Where the system does need strict agreement across nodes — for example, Redis Cluster’s own decision about which node currently holds a given shard, or a Kafka partition’s leader election — the underlying infrastructure relies on established consensus protocols (Raft in Redis Cluster’s newer implementations, ZooKeeper / KRaft-based consensus in Kafka) rather than the application layer reinventing distributed consensus, which is a common and costly mistake in system designs at this level of complexity.

05

Data Flow & Lifecycle

Two diagrams capture the lifecycle best: a sequence diagram showing the interaction between components across a full quote-to-settlement journey, and a state diagram showing the states a single quote or transaction moves through.

sequenceDiagram participant C as Customer App participant GW as API Gateway participant Q as Quote Service participant RC as Rate Cache participant LS as Lock Store Redis participant O as Transaction Orchestrator participant L as Ledger Service participant P as Payment Execution Service C->>GW: Request quote USD to INR GW->>Q: Forward quote request Q->>RC: Fetch latest mid market rate RC–>>Q: Return rate and timestamp Q->>LS: Lock rate with TTL 30s and quoteId LS–>>Q: Lock confirmed Q–>>GW: Quote with locked rate and expiry GW–>>C: Show locked rate to customer C->>GW: Confirm transfer with quoteId GW->>O: Execute transaction for quoteId O->>LS: Validate lock is still active LS–>>O: Lock valid, return locked rate O->>L: Create ledger entry with locked rate O->>P: Execute payment at locked rate P–>>O: Payment success O–>>GW: Transaction confirmed GW–>>C: Show receipt with same locked rate
Figure 5.1 — Sequence diagram of a quote-to-settlement journey. The exact same locked rate value flows unchanged from the quote step to the final receipt.
stateDiagram-v2 [*] –> QuoteRequested QuoteRequested –> RateFetched RateFetched –> RateLocked RateLocked –> QuoteExpired: TTL exceeded no confirmation RateLocked –> Confirmed: Customer confirms within TTL Confirmed –> PaymentProcessing PaymentProcessing –> PaymentFailed: Provider error PaymentProcessing –> Settled: Payment success PaymentFailed –> RefundInitiated RefundInitiated –> [*] QuoteExpired –> [*] Settled –> [*]
Figure 5.2 — State lifecycle of a single quote or transaction. Every transition is driven server-side; the client never has authority to move a quote out of RateLocked.

5.1 Reconciliation: closing the loop

Even with a correct design, financial systems run continuous reconciliation as a defense-in-depth measure. A background job periodically compares the rate recorded in the Audit Log for each executed transaction against the rate recorded in the Ledger entry — if the platform ever has a bug that lets these two drift, the reconciliation job catches it before a customer or regulator does.

flowchart LR A[“FX Rate Providers”] –> B[“Rate Ingestion Workers”] B –> C{“Rate Deviation Check”} C –>|Within Threshold| D[“Update Rate Cache”] C –>|Anomaly Detected| E[“Alert and Use Fallback Provider”] E –> D D –> F[“Publish Rate Update Event Kafka”] F –> G[“Quote Service Consumers”] D –> H[“Reconciliation Job”] H –> I{“Compare Quoted vs Applied Rate”} I –>|Mismatch| J[“Alert Finance and Ops Team”] I –>|Match| K[“Close Audit Record”]
Figure 5.3 — Rate ingestion pipeline feeding the cache, plus the reconciliation loop that continuously verifies quoted rate equals applied rate.
06

Advantages, Disadvantages & Trade-offs

Advantages

Rate Locking Benefits

  • Customer trust: the quoted rate is provably the executed rate.
  • Bounded FX risk window for the business (exposure is capped at the TTL, not open-ended).
  • Clean audit trail for regulators — every rate decision is timestamped and immutable.
  • Decouples quote-serving (read-heavy, cache-friendly) from execution (write-heavy, strongly consistent), so each scales independently.
Disadvantages

Costs of Rate Locking

  • The business absorbs the difference if the real market moves against the locked rate before execution — a real financial cost that must be hedged or priced into the margin.
  • Extra infrastructure: a dedicated lock store, TTL management, and reconciliation jobs add operational complexity versus “just charge live rate.”
  • Short TTLs frustrate customers who hesitate; long TTLs increase the business’s FX risk. A genuine tuning trade-off, not a solved problem.

6.1 Key trade-off: TTL length

TTL choiceBenefitCost
Short (10–15s)Minimal FX risk exposure for the business; rate stays very close to live marketCustomers on slow networks or hesitant to confirm see quotes expire often, hurting conversion
Medium (30–60s)Comfortable UX; most transfers complete within this windowBusiness must hedge a slightly larger risk window, usually priced into margin
Long (5+ minutes)Very forgiving UX, good for partner or B2B integrations with slower confirm flowsMeaningful FX exposure; typically requires the business to actually pre-purchase currency to cover the lock, not just price around it

6.2 Key trade-off: cache freshness versus provider load

Refreshing the Rate Cache more frequently keeps quotes closer to the true market price but increases load and cost on external FX data providers (many charge per API call or per data feed connection). Most production systems settle on continuous streaming subscriptions (push, not poll) for major pairs and a slower polling cadence for exotic, low-volume currency pairs.

💬
What an interviewer may ask

“How would you decide the TTL for a specific corridor, like USD to a very volatile emerging-market currency versus USD to EUR?” Strong answer: TTL and margin should both be corridor-aware. Volatile or thin currency pairs justify either a shorter TTL or a larger margin buffer, because the risk of the market moving against the locked rate within the window is higher.

07

Performance & Scalability — Designing for a Million Requests a Minute

A million requests per minute is roughly 16,700 requests/second sustained, and real traffic is never flat — expect 3–5x bursts around payday clusters, market volatility events, or marketing campaigns, so the system should be designed to comfortably absorb 50,000–80,000 requests/second at peak on the quote-read path specifically, since quote reads dominate total volume (customers check rates far more often than they actually transfer).

7.1 Read/write split is the foundation

The single most important scaling decision is recognizing that quote reads and transfer writes have wildly different volumes and different consistency needs, and must be scaled as separate pools:

PathTypical share of trafficConsistency needScaling strategy
Quote (read)~95%+Eventually consistent is fine (rate cache lags real market by milliseconds)Horizontally scaled stateless service instances + in-memory cache reads, no database hit per request
Transfer (write)<5%Strongly consistent, must be exactly-onceSmaller, more heavily guarded pool with idempotency and distributed transaction control

7.2 Caching is what makes a million requests per minute feasible

The Quote Service must never call an external FX provider synchronously on the customer’s request path. Every quote request is served from the in-memory Rate Cache, which is refreshed asynchronously by the Rate Ingestion Workers independent of customer traffic. This converts an otherwise-impossible fan-out (millions of customer requests each needing a fresh external FX API call) into a simple in-memory lookup, decoupled entirely from external provider rate limits.

7.3 Horizontal scaling of the Quote Service

Because Quote Service instances are stateless (all state — the current rate — lives in the shared Rate Cache, and the resulting lock lives in the shared Lock Store), the service can be scaled horizontally behind the load balancer with no coordination needed between instances. Auto-scaling groups driven by CPU or request-latency metrics can add capacity in seconds to absorb bursts.

QuoteService.java — pure, stateless quote generation safe to run on any instance
public Quote generateQuote(QuoteRequest request) {
    Rate rate = rateCache.getCurrentRate(request.getCurrencyPair()); // O(1) in-memory read
    BigDecimal lockedRate = pricingEngine.applyMargin(rate, request);
    String quoteId = UUID.randomUUID().toString();
    Instant expiresAt = Instant.now().plus(quoteTtl);

    LockedQuote lockedQuote = new LockedQuote(
        quoteId, request.getFrom(), request.getTo(),
        request.getAmount(), lockedRate, expiresAt, request.getCustomerId()
    );

    lockStore.put(quoteId, lockedQuote, quoteTtl); // single network hop, no local state kept

    return Quote.from(lockedQuote);
}

7.4 Rate Cache must be partitioned and replicated

A single Redis node cannot serve tens of thousands of reads per second from every regional pool reliably. The standard approach:

  • Redis Cluster with sharding by currency pair (there are only a few hundred meaningful pairs, so this is a small, well-known key space — ideal for full replication rather than sharding by hash, since every shard can hold the entire rate table cheaply).
  • Read replicas per region — since exchange rates are the same globally at a given instant, the Rate Cache can be replicated to every geographic region so a customer in Manila reads from a cache physically near them, not round-tripping to a single central cluster.
  • Local in-process cache with a very short TTL (hundreds of milliseconds) as an optional extra layer inside each Quote Service instance, shaving off even the Redis network hop for the highest-traffic pairs, at the cost of very slightly staler data.

7.5 Rate limiting and backpressure

At this scale, some traffic will be abusive (bots scraping rates, competitor price-monitoring) or simply excessive. The API Gateway enforces per-customer and per-IP rate limits using a token bucket algorithm, protecting the Quote Service pool for genuine users.

GatewayRateLimiter.java — token bucket sketch at the gateway layer
public boolean allowRequest(String clientKey) {
    Bucket bucket = buckets.computeIfAbsent(clientKey,
        k -> Bucket.builder()
            .addLimit(Bandwidth.classic(100, Refill.greedy(100, Duration.ofSeconds(60))))
            .build());
    return bucket.tryConsume(1);
}

7.6 Asynchronous, queue-backed writes for the transfer path

While the transfer path is a much smaller fraction of overall volume, it still needs to handle real spikes (a payday moment where a huge fraction of confirmed quotes execute within the same minute). The orchestrator publishes execution requests onto Kafka partitioned by customer ID, letting downstream payment execution workers scale horizontally and process the queue at whatever rate the payment rails can sustain, smoothing bursts instead of overwhelming the payment provider’s own APIs.

7.7 Capacity math walkthrough

It helps to work the numbers concretely rather than treating “a million requests a minute” as an abstract phrase. The average rate resolves to $1{,}000{,}000 / 60 approx 16{,}667$ requests per second, and applying a peak factor of $3times$ to $5times$ pushes the design target to roughly $50{,}000$ to $85{,}000$ requests per second.

  • Sustained load: 1,000,000 requests / 60 seconds ≈ 16,667 requests/second average.
  • Peak burst factor: real-world traffic on payday or during a viral marketing moment can spike 3–5x above average for short windows, so the system should be provisioned (or able to auto-scale within seconds) to roughly 50,000–85,000 requests/second.
  • Read/write split: if 95% of requests are quote reads, that’s ~47,500–80,000 reads/second against the Rate Cache, and only ~2,500–4,000 writes/second against the Lock Store and downstream execution path at peak — a very different scaling target for each store.
  • Per-instance capacity: if a single Quote Service instance (a modest container, a few CPU cores) can sustain roughly 2,000–3,000 requests/second doing a pure in-memory cache read plus a lightweight lock write, the fleet needs on the order of 20–30 instances at peak — a small, very achievable number once the design has correctly removed all synchronous external calls from the hot path.

This kind of back-of-envelope math is exactly what interviewers want to see: not memorized numbers, but the ability to reason from a stated requirement down to a concrete instance count and identify which store needs the most headroom.

7.8 Capacity planning table

LayerTarget capacity at peakScaling lever
Load Balancer / Gateway80,000+ req/sMulti-region LB, auto-scaled gateway pool
Quote Service80,000+ req/sStateless horizontal auto-scaling
Rate Cache (Redis)200,000+ reads/s per regionRegional replicas, cluster sharding for headroom
Lock Store (Redis)80,000+ writes/s for locksCluster sharded by quoteId hash
Transaction Orchestrator~1,000–5,000 executions/s at peakHorizontal pool + Kafka-backed queueing for smoothing
Primary DatabaseWrite-optimized, sharded by customer regionSharding + read replicas for reporting queries

7.9 Networking considerations at this volume

Raw request-per-second numbers only tell part of the story; the networking layer itself needs deliberate tuning at this scale:

  • HTTP/2 (or HTTP/3 where supported) between clients and the gateway multiplexes many logical requests over a single TCP connection, dramatically cutting connection-setup overhead versus HTTP/1.1 when a mobile app is polling for rate updates or a countdown timer.
  • Connection pooling and keep-alive between internal services (Gateway to Quote Service, Quote Service to Redis) avoids the cost of a fresh TCP and TLS handshake on every internal call — at tens of thousands of requests per second, handshake overhead alone can become a dominant cost if connections aren’t reused.
  • DNS-based global traffic routing (GeoDNS or anycast) sends a customer’s request to the nearest healthy region automatically, which both reduces latency and naturally load-balances traffic across regions without any single global load balancer becoming a bottleneck.
  • gRPC for internal service-to-service calls (Quote Service to Rate Cache client, Orchestrator to Ledger Service) is a common choice over REST/JSON internally, since binary serialization and HTTP/2 multiplexing meaningfully reduce per-call overhead at this request volume, even though the customer-facing API remains REST/JSON for broad client compatibility.
💬
What an interviewer may ask

“A million requests a minute — where’s your bottleneck going to be first?” Good candidates identify the Rate Cache and Lock Store as the components under the most sustained pressure (since almost every request touches them), and explain that both must be clustered/sharded and regionally replicated — a single Redis instance, however powerful, cannot serve this load alone.

💬
What an interviewer may ask

“Why not put the rate directly in the database instead of a cache?” Good answer: a relational database optimized for durability and transactional writes cannot sustain tens of thousands of reads per second per region at sub-millisecond latency as cheaply or reliably as an in-memory store; the rate cache exists specifically to absorb the read-heavy 95%+ of traffic without ever touching disk-backed storage.

08

High Availability & Reliability

8.1 No single FX provider

An outage or bad data tick from one FX rate provider must not take down the whole system. Rate Ingestion Workers subscribe to at least two independent providers per major currency pair; if the primary feed goes silent (no update within an expected heartbeat window) or produces a rate that fails the sanity check from Section 4.4, ingestion automatically fails over to the secondary provider, and an alert fires for the operations team.

8.2 Redis Cluster failover

Both the Rate Cache and the Lock Store run as multi-node clusters with replicas. If a primary node fails, a replica is promoted automatically (Redis Sentinel or Redis Cluster’s native failover) within seconds. Because the Rate Cache is a derived, refreshable value (it can be rebuilt from the next provider tick), losing it briefly is far less damaging than losing the Lock Store, which holds customer-facing promises — so the Lock Store is configured with a higher replication factor and, where the provider supports it, cross-AZ synchronous replication for the lock write itself.

8.3 Multi-region deployment

The entire stack (gateway, quote service, caches, orchestrator) is deployed across at least two geographic regions with active-active or active-passive routing via global DNS/traffic manager. A regional outage fails traffic over to the healthy region. The one subtlety: a lock created in Region A must be visible to an execution request that lands in Region B (possible if a customer’s network path changes mid-session), which is why the Lock Store typically uses a globally-replicated data store or sticky regional routing keyed by customer session for the life of a single quote.

8.4 Graceful degradation

If the Rate Cache becomes fully unavailable in a worst-case scenario, the system should fail closed on new quotes (show “rates temporarily unavailable” rather than serving a stale or guessed rate) while still allowing already-locked, in-flight transactions to complete using their existing lock — the two failure domains are handled differently on purpose.

8.5 Circuit breakers around external dependencies

PaymentExecutionService.java — circuit breaker guarding an external payout provider
public PaymentResult executePayment(PaymentRequest request) {
    if (circuitBreaker.isOpen("payoutProviderX")) {
        return payoutRouter.routeToBackupProvider(request);
    }
    try {
        PaymentResult result = payoutProviderXClient.send(request);
        circuitBreaker.recordSuccess("payoutProviderX");
        return result;
    } catch (ProviderTimeoutException e) {
        circuitBreaker.recordFailure("payoutProviderX");
        return payoutRouter.routeToBackupProvider(request);
    }
}

8.6 Disaster recovery

The Ledger and Audit stores, being the systems of record, are backed up continuously (write-ahead log shipping to a secondary region) with a target Recovery Point Objective (RPO) close to zero and a Recovery Time Objective (RTO) measured in minutes — financial regulators typically require both to be documented and tested.

💬
What an interviewer may ask

“Your Lock Store cluster goes down mid-peak. What happens to a customer who already has a locked quote?” Strong answer: this is exactly why the Lock Store gets the highest reliability investment in the whole system — cross-AZ replication and fast automated failover — because losing an active lock either strands a customer’s confirmed intent or forces re-quoting mid-flow, both of which are worse failure modes than losing the (easily rebuilt) Rate Cache.

09

Security

9.1 Authentication & authorization

Every request is authenticated via OAuth 2.0 / JWT at the API Gateway. Fine-grained authorization ensures a customer can only lock and execute quotes tied to their own account, enforced server-side on every call, never trusted from client-supplied identifiers.

9.2 Protecting the rate and lock APIs

  • Tamper-proof quoteId: IDs are cryptographically random UUIDs, not sequential integers, so a customer cannot guess or enumerate another customer’s active quote.
  • Server-side rate application only: the client never sends a rate value to the server; it only ever sends a quoteId, and the server looks up the authoritative locked rate itself. This closes off an entire class of client-side tampering attacks where a malicious client could otherwise submit a favorable rate directly.
  • mTLS between internal services (Quote Service, Orchestrator, Ledger, Payment Execution) so that even inside the private network, services authenticate each other.

9.3 Fraud & AML controls

Before a locked quote can be executed, the orchestrator checks the customer’s KYC/AML status and runs the transaction through sanctions-list screening and velocity checks (unusual transfer patterns), consistent with global anti-money-laundering regulation. High-risk transactions are held for manual review rather than auto-executed.

9.4 Data protection

Customer PII and payment credentials are encrypted at rest (AES-256) and in transit (TLS 1.3). Sensitive fields in logs (account numbers, full names) are masked before reaching the centralized logging system, since logs are a common accidental leak vector.

9.5 Secrets management

Credentials for FX data providers and payment rails are stored in a dedicated secrets manager (HashiCorp Vault or a cloud KMS-backed secret store) with short-lived, automatically rotated credentials rather than long-lived static API keys embedded in configuration.

9.6 Preventing replay and double-spend

Covered in depth in Section 4.3 — the atomic claim-and-delete on the Lock Store plus idempotency keys together close the two realistic replay vectors: a client retry, and a malicious actor attempting to resubmit a captured request.

💬
What an interviewer may ask

“How do you stop a customer from tampering with the rate in the confirm request?” The answer: the confirm request should carry only an opaque quoteId, never a rate value — the server is the sole source of truth for what rate a given quoteId represents, so there is nothing for the client to tamper with.

10

Monitoring, Logging & Metrics

10.1 The metrics that matter most for this system

MetricWhy it matters
Quote-to-rate-cache latency (p50/p95/p99)Directly determines whether the “sub-second quote” promise holds under load
Rate staleness (time since last accepted provider tick, per pair)Detects a silently stalled FX feed before it causes bad quotes
Quote expiry rate (% of quotes that expire unconfirmed)Signals whether TTL is too short, or whether UX friction is causing drop-off
Lock-to-execution mismatch countShould be zero at all times — any non-zero value is a P1 incident, since it means the core promise of the system broke
Requests per second by endpoint, regionCore capacity signal driving auto-scaling decisions
Circuit breaker open eventsEarly warning of a degrading external dependency (FX provider or payout rail)

10.2 Distributed tracing

Every request carries a correlation ID (propagated via OpenTelemetry) from the API Gateway through Quote Service, Rate Cache, Lock Store, and — for executed transfers — through the Orchestrator, Ledger, and Payment Execution Service. This lets an engineer reconstruct the full path of a single customer’s transfer in seconds when investigating a support ticket, rather than grepping logs across a dozen services manually.

10.3 Alerting philosophy

Alerts are tiered by business impact, not just technical severity:

  • P1 (page immediately): lock-to-execution rate mismatch detected; Lock Store cluster degraded; payment execution error rate above threshold.
  • P2 (urgent, business hours): a single FX provider feed down (system still healthy via failover, but risk elevated); elevated quote expiry rate.
  • P3 (informational): auto-scaling events, minor latency degradation within SLA.

10.4 Dashboards for different audiences

Engineering dashboards track latency and error budgets. A separate finance/risk dashboard tracks aggregate FX exposure (sum of all currently-locked, unexecuted quote amounts per currency pair) in near real time, since that number represents the company’s live financial risk at any given moment.

💬
What an interviewer may ask

“What’s the single most important alert in this whole system?” The strongest answer: a detected mismatch between the quoted rate and the applied rate on any executed transaction. Everything else in the system exists to prevent that number from ever going above zero, so it deserves the tightest, fastest alerting path in the entire stack.

11

Deployment & Cloud

11.1 Containerized microservices on Kubernetes

Each service (Quote Service, Rate Ingestion Workers, Transaction Orchestrator, Ledger Service, Payment Execution Service, Notification Service) is packaged as an independent container and deployed on Kubernetes, letting each be scaled, deployed, and rolled back independently based on its own traffic profile — the Quote Service pool, for instance, needs far more replicas than the Ledger Service pool.

11.2 Blue-green and canary deployments

Given the financial stakes, changes to the Quote Service or Transaction Orchestrator are rolled out via canary deployment: a small percentage of traffic (5%) is routed to the new version while automated checks confirm the lock-to-execution mismatch metric from Section 10.1 stays at zero before the rollout proceeds further. Any anomaly triggers automatic rollback.

11.3 Infrastructure as code

The entire environment — Kubernetes clusters, Redis clusters, Kafka topics, database instances, networking, and IAM policies — is defined declaratively (Terraform or similar) so that a new region can be stood up reproducibly, which matters both for disaster recovery and for expanding into new remittance corridors.

11.4 Multi-region cloud topology

A common pattern is regional active-active deployment: US, EU, and APAC regions each run a full stack, with the Rate Cache replicated globally (rates are the same everywhere at a given instant) and customer data/ledger sharded regionally to satisfy data residency regulations that many countries impose on financial data.

11.5 Cost optimization

The Quote Service pool, being stateless and bursty, is a good candidate for spot/preemptible instances mixed with a baseline of reserved capacity, since a terminated instance simply drops out of the load balancer pool with no data loss. The Ledger and Lock Store tiers, by contrast, run on stable, reserved infrastructure given their correctness sensitivity.

💬
What an interviewer may ask

“Would you use serverless functions for the Quote Service?” A balanced answer: serverless can work for bursty, stateless read paths and is attractive for the pure horizontal scaling story, but cold-start latency can conflict with the sub-second quote SLA at extreme scale, so most production systems at this volume prefer pre-warmed, auto-scaled container pools over serverless for the hottest path, while using serverless comfortably for lower-volume, latency-tolerant jobs like reconciliation.

12

Databases, Caching & Load Balancing

12.1 Choosing the database for each store

StoreTechnology choiceReasoning
Rate CacheRedis Cluster (in-memory)Sub-millisecond reads at massive volume; data is derived and rebuildable, so durability requirements are relaxed
Quote Lock StoreRedis Cluster with native TTL, high replication factorTTL expiry is a first-class feature here, not something to build manually; fast atomic operations for claim-and-delete
LedgerRelational (PostgreSQL), append-only tables, strict ACID transactionsDouble-entry accounting demands strong consistency and transactional guarantees; relational databases are the proven choice for financial ledgers
Transaction / Customer HistoryPostgreSQL, sharded by customer regionRelational integrity for account relationships; sharding needed purely for write/storage scale, not for consistency reasons
Audit LogAppend-only object storage (S3 with object lock) or an immutable log tableRegulatory requirement for tamper-evidence and long retention; object storage is cheap at the multi-year retention scale required

12.2 Caching strategy deep dive

The Rate Cache uses a write-through pattern from the Rate Ingestion Workers (the workers are the only writers; the Quote Service pool only ever reads). This avoids the classic cache-invalidation complexity that arises when many independent services all try to write to a shared cache — here, there is exactly one writer role, which drastically simplifies correctness.

12.3 Load balancing algorithms

  • Layer 4 (network) load balancing at the outermost edge for raw connection distribution across regions, using least-connections or consistent-hashing algorithms.
  • Layer 7 (application) load balancing at the API Gateway tier, which can route based on request path (separating quote traffic from execution traffic onto differently-sized backend pools) and perform active health checks, pulling an unhealthy Quote Service instance out of rotation within seconds.
  • Client-side load balancing between internal services (Orchestrator to Ledger Service) via a service mesh (Istio/Linkerd), which also provides mTLS and fine-grained traffic shaping for canary releases.

12.4 Sharding the primary database

Customer transaction history is sharded by a hash of customer ID combined with home region, which keeps each customer’s queries local to one shard (avoiding cross-shard joins for the common case of “show me my transaction history”) while distributing write load evenly across the cluster as the customer base grows.

💬
What an interviewer may ask

“Why relational for the ledger but Redis for the rate cache — isn’t that inconsistent?” Good answer: the choice always follows the consistency and durability requirement of the data, not a single dogmatic database philosophy. The ledger must never lose a write and must support atomic multi-row transactions — a relational database’s ACID guarantees are exactly built for that. The rate cache is disposable, rebuildable, read-dominated data — an in-memory store optimized for raw speed is the right tool for a completely different job.

13

APIs & Microservices

13.1 Core API contract

POST /v1/quotes — request and response contract
POST /v1/quotes
Request:  { "fromCurrency": "USD", "toCurrency": "INR", "sourceAmount": 1000.00 }
Response: {
  "quoteId": "8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11",
  "fromCurrency": "USD",
  "toCurrency": "INR",
  "lockedRate": 83.42,
  "sourceAmount": 1000.00,
  "destinationAmount": 83420.00,
  "expiresAt": "2026-08-04T10:15:45Z"
}
POST /v1/transfers — execute against a locked quoteId with idempotency
POST /v1/transfers
Headers:  Idempotency-Key: <client-generated-uuid>
Request:  { "quoteId": "8f14e45f-ceea-4c7f-b1ab-3c6d1c2e9a11" }
Response: {
  "transferId": "b2f9...",
  "status": "SETTLED",
  "appliedRate": 83.42,
  "destinationAmount": 83420.00
}

Notice that the applied rate returned from the Transfer API is expected to always exactly equal the locked rate from the original Quote API response — this equality is, in effect, the single most important test case in the entire system’s test suite.

13.2 Why microservices (and where the boundaries are)

The service boundaries in Section 3 are drawn along two principles: differing scaling needs (Quote Service scales very differently from the Ledger Service) and differing consistency needs (the Rate Ingestion path can tolerate eventual consistency; the Orchestrator and Ledger cannot). Splitting along these lines lets each team own a service with a genuinely distinct operational profile, rather than splitting arbitrarily by “noun” (which often produces services that must be deployed and scaled in lockstep anyway).

13.3 Synchronous versus asynchronous communication

InteractionStyleReasoning
Client to Quote ServiceSynchronous (REST/HTTP)Customer is waiting for an immediate answer
Client to Orchestrator (confirm)Synchronous, but internally may enqueue heavy downstream workCustomer needs a definitive success/failure response, even though settlement details may complete slightly afterward
Orchestrator to Notification ServiceAsynchronous (Kafka event)Customer doesn’t need to wait for a receipt email/SMS to be sent before getting their confirmation screen
Rate Ingestion to Rate CacheAsynchronous, continuous streamEntirely decoupled from any single customer request

13.4 API versioning and backward compatibility

Given how many partner integrations may depend on this API, all breaking changes go through explicit versioning (/v1/, /v2/) with a long deprecation window, and additive fields are always optional in responses so existing integrations don’t break silently.

💬
What an interviewer may ask

“Would you make the Quote-to-Transfer flow a single API call instead of two?” Good answer: no — separating “get a quote” from “confirm a transfer” is intentional and mirrors the real business need (the customer must see and approve a rate before money moves). Collapsing them into one call would remove the customer’s ability to review, and would also remove the natural seam where the system enforces its core “shown rate equals applied rate” guarantee.

14

Design Patterns & Anti-Patterns

14.1 Patterns used in this system

Pattern

Saga

The Transaction Orchestrator drives a multi-step transfer (ledger debit, payment execution, notification) as a saga with compensating actions (refund/reversal) if a later step fails, since a single ACID transaction cannot span the Ledger DB and an external payment rail.

Pattern

Cache-Aside / Write-Through

Rate Ingestion Workers write through to the Rate Cache; the Quote Service only ever reads. Single-writer role drastically simplifies cache-invalidation correctness.

Pattern

Circuit Breaker

Protects calls to external FX providers and payment rails, tripping to a fallback provider or a fail-closed error when a downstream dependency is unhealthy.

Pattern

Idempotent Receiver

The Transaction Orchestrator’s idempotency-key handling means a retried confirm request returns the original result rather than re-executing.

Pattern

Event Sourcing (partial)

The Audit Log and Kafka event stream together give a replayable history of every rate decision and transaction, valuable for both compliance and debugging.

Pattern

Bulkhead

Quote Service and Transaction Orchestrator run in separate resource pools so a spike or failure in one cannot starve the other of capacity.

14.2 Anti-patterns to avoid

Anti-patternWhy it’s dangerous here
Fetching a “live” rate at execution time instead of honoring the lockDirectly breaks the core promise of the system; the single most important anti-pattern to call out explicitly
Client-supplied rate values trusted by the serverOpens a direct financial tampering vector
Check-then-act on the Lock Store instead of an atomic claimIntroduces a race condition enabling double-spend
One shared mutable cache for both “current rate” and “locked rate”A cache refresh could silently mutate an already-quoted rate
Synchronous calls to external FX providers on the customer request pathMakes the system’s latency and availability hostage to a third party, and cannot scale to the required request volume
Silent retries without idempotency protectionRisks duplicate transfers under flaky network conditions
💬
What an interviewer may ask

“What’s the worst anti-pattern a junior engineer might introduce here without realizing it?” A great answer to lead with: re-fetching the live rate at execution time “just to double check,” reasoning that it seems safer. In reality this silently reintroduces rate slippage and defeats the entire purpose of rate locking — the locked rate must be treated as the single source of truth for that transaction, full stop, even if the live market has since moved.

15

Best Practices & Common Mistakes

15.1 Best practices

  • Always check quote expiry server-side, never trust a client-reported countdown timer.
  • Make the locked rate immutable once written — no update operations on a Lock Store entry, only create and atomic delete-on-consume.
  • Log the source FX provider and exact ingestion timestamp alongside every rate, so any dispute can be traced back to the raw market data that produced it.
  • Treat quote-rate-equals-applied-rate as an automated, continuously-running invariant check, not just a one-time test at launch.
  • Design margin/markup logic as a pure, versioned function so past quotes can always be recalculated exactly as they were computed at the time, for audit purposes.
  • Separate the “read-mostly, cache-friendly” quote path from the “write-heavy, consistency-critical” execution path at every layer — service, database, and even on-call rotation.

15.2 Common mistakes

  • Under-provisioning the Lock Store: teams often size Redis for the Rate Cache’s read load and forget the Lock Store needs write throughput matching peak quote-generation volume, not transfer volume.
  • Treating TTL expiry as best-effort: relying purely on Redis’s TTL eviction without also validating expiry explicitly in application logic during a network partition or clock-skew edge case.
  • Ignoring corridor-specific volatility when setting a single global TTL and margin for every currency pair, which under-prices risk on volatile pairs and over-prices it on stable ones.
  • Skipping load testing the Lock Store specifically — teams often load-test the Quote Service’s read path thoroughly but under-test the write-heavy lock-and-claim path, which behaves very differently under contention.
  • Rounding rates and amounts inconsistently between services: if the Quote Service and the Ledger Service round a converted amount using different decimal precision or rounding modes, the two can disagree by a fraction of a currency unit — small individually, but a real regulatory and reconciliation problem in aggregate across millions of transactions, so rounding rules must be defined once, centrally, and applied identically everywhere.

15.3 A simple rule of thumb

📌
The one-question tie-breaker

Whenever a design decision in this system is unclear, the tie-breaker question is always the same: “does this change risk the quoted rate and the applied rate ever being different values?” If the answer is even possibly yes, the design is wrong, regardless of how much simpler or cheaper it would be — this single invariant is the true north star for every component described in this guide.

16

Real-World Industry Examples

Several well-known companies have publicly discussed pieces of how they approach this exact problem, and the patterns above map closely onto their public engineering narratives.

Remittance

Wise (formerly TransferWise)

Built its entire brand promise around transparent, mid-market-rate-based pricing, and has written publicly about maintaining a real-time rate engine and honoring quoted rates for a short window during checkout — a close real-world analog to the Quote and Lock services described here.

Fintech

Revolut

Operates a multi-currency wallet product where in-app currency conversion must feel instantaneous; their architecture is known to rely heavily on in-memory rate caching and short-lived rate locks for card transactions and in-app conversions, similar in spirit to the caching and TTL-lock approach in this design.

Global Payments

PayPal / Xoom

As a large-scale global payments platform, PayPal’s remittance arm (Xoom) operates across many currency corridors and must reconcile quoted versus settled amounts across banking partners in different countries, reflecting the same quoted-vs-applied reconciliation discipline described in Section 5.

General Pattern

Amazon & Netflix

Amazon’s checkout “price hold” during payment processing and Netflix’s approach to stateless, horizontally-scaled read services behind aggressive caching are the same general architectural patterns — decouple the read-heavy discovery path from the write-heavy commit path — applied outside of the FX domain.

💬
What an interviewer may ask

“How is this different from something like an e-commerce ‘price at checkout’ problem?” Good answer: structurally very similar (both need a temporary lock on a value that’s otherwise continuously changing), but currency conversion carries stricter regulatory audit obligations and a genuine, continuously-moving underlying market price, versus e-commerce prices which usually change far less frequently and are set by the seller rather than a live external market.

17

FAQ, Summary & Key Takeaways

Q1

What happens if two rate providers disagree on the current rate?

The Rate Ingestion layer applies a configured trust hierarchy (a primary provider is authoritative when healthy) and sanity-checks incoming ticks against the last accepted value; a provider whose feed diverges too far or too fast from the trusted value is treated as suspect and temporarily excluded until it stabilizes.

Q2

How is the customer’s FX risk different from the company’s FX risk?

The customer’s risk window closes the instant they see a locked rate — they are fully protected from market movement for the TTL duration. The company, on the other hand, carries that same market movement risk for every outstanding locked quote until it either executes or expires, which is why aggregate outstanding-lock exposure is tracked as a live financial metric (Section 10.4).

Q3

Could this system use blockchain or stablecoins instead of traditional banking rails?

Some remittance providers do use stablecoin rails for parts of cross-border settlement to reduce cost and latency, but the quote-and-lock architecture described here is largely independent of which settlement rail is used underneath — the same rate-locking guarantee applies whether the Payment Execution Service ultimately settles via SWIFT, a local payout network, or a blockchain-based rail.

Q4

Why not just show the customer a rate range instead of an exact locked number?

A range is technically easier to guarantee, but it directly undermines the trust proposition that differentiates a modern remittance product from older-generation money transfer services, which historically were criticized for vague or hidden pricing.

Q5

What data structure choices actually matter here, beyond “use Redis”?

Two are worth calling out by name in an interview: a hash map (Redis’s native data type) for O(1) lookup of the current rate per currency pair, keyed simply by the pair string (“USD_INR”); and a TTL-indexed structure (Redis’s internal expiry mechanism, conceptually a min-heap ordered by expiry time) for the Lock Store, which lets the store efficiently evict expired quotes without scanning the entire key space on every tick.

Q6

How should the system behave if the destination currency’s local payout partner is down, but the rate was already locked and the source debit already happened?

This is precisely what the saga pattern’s compensating action exists for (Section 14.1): the orchestrator triggers a reversal of the source-side ledger debit and, if any funds were already moved, an automated refund flow, while keeping the original locked rate recorded in the audit trail for full traceability of what was promised versus what actually settled.

📌
Key takeaways
  • The core problem splits cleanly into two: rate discovery (fast, cacheable, eventually consistent) and rate guarantee (short-lived, strongly consistent, immutable once locked).
  • A dedicated Rate Lock Store with native TTL is what actually delivers the “quoted rate equals applied rate” promise — not the Rate Cache itself.
  • Read (quote) and write (transfer) paths must be treated as separate scaling problems, since quote traffic can be orders of magnitude larger than transfer traffic.
  • At a million-requests-a-minute scale, the Rate Cache and Lock Store are the components under the most sustained pressure, and both need clustering, sharding, and regional replication.
  • Idempotency keys plus atomic claim-and-delete on the lock together eliminate double-spend risk from both client retries and malicious replay.
  • Continuous automated reconciliation between quoted and applied rates is a non-negotiable safety net, not an optional nice-to-have, in a regulated financial system.
  • The single anti-pattern to never fall into: re-fetching a live rate at execution time instead of trusting the lock.

17.1 The one idea to remember

If you take one architectural lesson from this guide into your next system design interview or your own production system, let it be this: separate the parts of your system that are allowed to change from the parts that must never change once promised. A live market rate is allowed to change every second — that’s the Rate Cache’s whole job. A rate shown to a specific customer, tied to a specific quoteId, must become frozen and immutable the instant it is shown — that’s the Rate Lock Store’s whole job. Almost every design decision in this document, from the choice of database, to the TTL trade-off, to the CAP-theorem posture of each component, to the anti-patterns worth memorizing, falls directly out of keeping those two responsibilities cleanly separated and never letting one leak into the other.