Imagine two friends buy the exact same jacket from the exact same online store, at almost the exact same time. One of them sees a checkout page with a single “Buy Now” button. The other sees a checkout page broken into three clear steps, with a progress bar at the top. Neither of them notices anything odd — to each of them, the page they see just is the checkout page. But behind the scenes, the store deliberately showed two different designs to two different, randomly chosen groups of customers, specifically so it could measure which design gets more people to actually finish their purchase. This is called an A/B test (also called a “split test” or, when there are more than two variants, a “multivariate test”): you split your traffic into groups, show each group a different variant of something, and measure which variant performs better against a clearly defined metric.

A/B testing itself is an old idea — it’s really just the scientific method (form a hypothesis, run a controlled experiment, measure the outcome) applied to product decisions, and statisticians have used randomised controlled trials for over a century in agriculture and medicine. What’s changed is the scale and speed at which internet companies can run these experiments. A large e-commerce platform doesn’t run one experiment a year; it often runs dozens of experiments simultaneously, on the same page, for the same users, and expects results within days, not months. Doing this manually, or even with simple flat-file configuration, breaks down completely once you’re dealing with millions of shoppers checking out per minute during a big sale event.

This tutorial builds, from the ground up, a system that can: consistently assign each shopper to a checkout-flow variant, serve that decision to the checkout page in milliseconds without adding latency to the critical purchase path, safely support many concurrent experiments without them interfering with each other, and compute statistically sound conversion-rate results in near real time — all while surviving traffic spikes of a million or more requests per minute.

1.1 A Short Timeline of Product Experimentation

Let’s unpack why this is a genuinely hard system design problem, layer by layer.

2.1 The Assignment-Consistency Problem

Once a shopper is assigned to “Variant B” of the checkout flow, they must see Variant B every single time they return to checkout — on every page load, every retry after a failed payment, even if they close the tab and come back an hour later. If the assignment is inconsistent, two things go wrong: the shopper gets a confusing, glitchy experience, and the experiment’s data becomes statistically invalid, because you can no longer cleanly attribute an outcome to a single variant. This means assignment must be deterministic (the same input always produces the same output) rather than randomly re-rolled on every request.

2.2 The Zero-Added-Latency Problem

The checkout page is the single most latency-sensitive page on the entire site — every extra 100 milliseconds of load time has a measurable, well-documented negative effect on conversion. This means the experiment-assignment lookup absolutely cannot involve a slow network call to a heavyweight backend service sitting in the critical path. The system has to be architected so that variant assignment is effectively “free” from the checkout page’s point of view — typically resolved locally at the edge or from an extremely fast in-memory cache, not from a round trip to a central experimentation database.

2.3 The “Millions of Requests a Minute” Scale Problem

At peak (think Black Friday, a flash sale, or a viral product moment), checkout-related traffic can spike to well over a million requests per minute — roughly 16,000–20,000+ requests per second sustained, with even higher instantaneous bursts. Every layer of this system — assignment, event logging, and metrics aggregation — has to be designed to absorb that volume without becoming the bottleneck, and without dropping experiment exposure or conversion events, since dropped events silently corrupt the experiment’s results.

2.4 The Statistical Validity Problem

Even with perfect engineering, it’s easy to get the statistics wrong. Peeking at results too early and stopping a test the moment it looks “significant” (a mistake called “peeking”) inflates the false-positive rate. Running many experiments simultaneously on overlapping user populations can cause interaction effects, where one experiment’s result is contaminated by another. The system needs to bake in safeguards — proper sample-size planning, sequential-testing-aware statistics, and experiment isolation and interaction detection — rather than leaving correct statistical practice entirely up to whichever team happens to be running a given test.

2.5 The Blast-Radius Problem

Because this system controls what the checkout page actually looks like and does, a misconfigured experiment (for example, one variant accidentally hiding the payment button, or routing a percentage of real users into a broken code path) can directly cause lost revenue and abandoned carts within minutes. The system needs fast, reliable kill switches and automated guardrail monitoring that can pull the plug on a bad experiment before it causes significant damage.

2.6 The Fixed-Split Versus Adaptive-Allocation Problem

A traditional A/B test holds the traffic split fixed (say, 50/50) for the entire duration of the experiment, even after it becomes fairly clear which variant is winning — this is intentional, because a fixed split is what makes the classical statistical guarantees valid. But holding a losing variant at 50% traffic for the full test duration has a real opportunity cost: every shopper routed to a worse checkout flow during that time is a shopper who converted at a lower rate than they otherwise would have. Some experimentation platforms address this with adaptive allocation (multi-armed bandit) approaches that gradually shift more traffic toward the currently-better-performing variant while the test is still running. This is a genuine design trade-off, not a strictly-better replacement for classical fixed-split testing: bandit approaches reduce the cost of running the experiment but make the statistical analysis meaningfully more complex, and are generally better suited to short-lived, tactical decisions than to experiments where you specifically need a clean, unbiased estimate of the effect size for a long-term strategic decision.

Before drawing a single architectural box, let’s put names to the small vocabulary that shows up in every layer of this design.

3.1 A Closer Look: Deterministic Bucketing

How do you make sure the same shopper always lands in the same variant, without needing to look anything up in a database on every request? The standard technique is consistent hash bucketing. You take a stable identifier for the user (a logged-in user id, or a long-lived anonymous device/cookie id for guests), combine it with the experiment’s unique name, run it through a fast, well-distributed hash function (like MurmurHash), and map the resulting hash value into a bucket — commonly 0 to 9999 or 0 to 99. If the experiment is a 50/50 split, buckets 0–4999 might mean “Control” and 5000–9999 might mean “Treatment.” Because hashing the same input always produces the same output, the same user always lands in the same bucket for that experiment, with zero storage or lookup required to reproduce the decision — it can be computed anywhere, including directly at the edge, purely from the user id and experiment name.

Salting the hash input with the experiment name (rather than hashing the user id alone) is what allows the same user to be independently, unpredictably assigned across different experiments — otherwise a user who’s always in “Treatment” for one test would suspiciously always land in “Treatment” for every other test too, which would bias results whenever two experiments happen to interact.

Now let’s design the system end to end. Given the “millions of requests per minute” requirement, the guiding principle throughout is: keep the checkout critical path as close to zero-added-latency as possible by resolving assignment locally at the edge, and push all the heavy lifting (event ingestion, aggregation, statistics) into an asynchronous, horizontally scalable pipeline that never blocks a purchase. Every box below states which infrastructure component — API Gateway, Load Balancer, and so on — fronts or manages it.

Let’s walk through the components in detail.

4.1 API Gateway

The API Gateway is the single front door for all external traffic: it terminates TLS, authenticates requests, applies coarse-grained rate limiting, and routes to the correct internal service — either the Checkout Service for actual purchase traffic, or the Event Ingestion Service for asynchronous exposure and conversion event traffic. Keeping these two traffic types routed separately (even though both pass through the same gateway) means a spike in event-logging volume can never starve actual checkout request capacity.

4.2 Load Balancer

Every stateless service tier — Checkout Service, Experiment Config Service, Event Ingestion Service, Statistics Engine, Kill Switch Service, Dashboard — sits behind its own dedicated Load Balancer, using Layer 7 routing with health-check-based failover so unhealthy pods are automatically removed from rotation. At the target scale of a million-plus requests per minute, each of these tiers is independently horizontally autoscaled behind its Load Balancer, so a surge in checkout traffic doesn’t require over-provisioning the (much lower-volume) Config or Dashboard tiers.

4.3 Edge / Client-Side Bucketing SDK

This is the component that makes zero-added-latency assignment possible. A lightweight SDK, embedded in the checkout page or app, holds a locally cached copy of active experiment definitions (which experiments are running, their bucket ranges, and their variant configurations) and computes the deterministic hash-based assignment entirely on the client or at the CDN edge — no network round trip required for the actual bucketing decision on each page load.

4.4 Experiment Config Service

This is the low-traffic, low-latency-tolerant control-plane service where product teams define experiments (name, hypothesis, traffic split, variant configs, target metric, guardrail metrics, start and end dates). It periodically pushes updated experiment definitions out to the edge config cache (for example every few seconds via a pub/sub push or short-polling mechanism), so new experiments and kill-switch changes propagate quickly without every single checkout request needing to hit this service directly.

4.5 Event Ingestion Service

Receives exposure events (“user X saw variant B of experiment Y”) and conversion events (“user X completed checkout”) from the client. This service is intentionally kept extremely thin — minimal validation, no heavy business logic — because it must absorb the full firehose of traffic at checkout scale, immediately handing events off to the message broker rather than doing any expensive processing inline.

4.6 Message Broker (Kafka)

All exposure and conversion events flow through Kafka, partitioned by experiment id, decoupling the high-throughput, latency-sensitive ingestion path from the heavier downstream aggregation and statistics work. This also provides durability and replay-ability — if a bug is found in the aggregation logic, events can be reprocessed from the retained log.

4.7 Stream Processing Layer & Statistics Engine

A windowed stream-processing layer continuously aggregates raw events into per-experiment, per-variant counters (exposures, conversions, revenue) at short time intervals. The Statistics Engine consumes these aggregates and applies proper statistical methodology — including sequential testing corrections so that “peeking” at results early doesn’t invalidate the experiment — to produce a continuously updated verdict: no significant difference yet, Treatment winning, Control winning, or a guardrail breach.

4.8 Auto Kill Switch Service

If the Statistics Engine detects a guardrail metric breach (for example, a spike in checkout errors or a payment failure rate for one variant), this service automatically pushes an updated experiment config that routes 100% of traffic to the safe control variant, without waiting for a human to notice and react.

Let’s trace what actually happens, step by step, for one shopper going through checkout.

5.1 How Config Propagation Actually Works

It’s worth being precise about the mechanism connecting the Experiment Config Service to millions of individual shopper sessions, since “push config to the edge” can otherwise sound a little hand-wavy. In practice this works through a short-interval polling or pub/sub-based fan-out: the Experiment Config Service publishes its current, versioned configuration bundle (typically well under a few hundred kilobytes, even with hundreds of active experiments, since each experiment’s definition is tiny) to a small number of regional CDN origin points. Each CDN edge node, and each client-side SDK instance, independently pulls or subscribes to updates on a short interval — commonly every few seconds to under a minute, tunable per deployment. Because the config bundle is versioned, a client can cheaply check “is my cached version still current?” without re-downloading the full bundle on every check, using a simple version or etag comparison. This design means a brand-new experiment, or a kill-switch update, doesn’t need to reach a million individual sessions instantly and atomically — it just needs to reach the relatively small number of CDN edge nodes quickly, and those nodes then serve the updated config to every session that requests it going forward, which is what lets propagation stay fast even at extreme shopper-side scale.

The sequence below traces one exposure-to-conversion lifecycle end to end, again labelling which tiers sit behind a Load Balancer versus which are asynchronous, queue-driven components.

6.1 Java Code Example: Deterministic Bucketing

public class ExperimentBucketer {

    private static final int BUCKET_COUNT = 10000;

    // Deterministically assigns a user to a bucket for a given experiment,
    // using a stable hash so the same user always lands in the same bucket.
    public int assignBucket(String userId, String experimentName) {
        String input = experimentName + ":" + userId;
        long hash = Hashing.murmur3_128()
                .hashString(input, StandardCharsets.UTF_8)
                .asLong();
        int bucket = (int) Math.floorMod(hash, BUCKET_COUNT);
        return bucket;
    }

    public String resolveVariant(String userId, ExperimentConfig config) {
        if (!config.isActive()) {
            return config.getDefaultVariant();
        }
        int bucket = assignBucket(userId, config.getExperimentName());
        for (VariantRange range : config.getVariantRanges()) {
            if (bucket >= range.getStart() && bucket < range.getEnd()) {
                return range.getVariantName();
            }
        }
        return config.getDefaultVariant();
    }
}

6.2 Java Code Example: Non-Blocking Event Emission

public class ExposureEventEmitter {

    private final AsyncHttpClient httpClient;
    private static final Duration TIMEOUT = Duration.ofMillis(150);

    // Failures are swallowed intentionally: losing an event is acceptable,
    // blocking a purchase is not.
    public void emitExposure(String userId, String experimentName, String variant) {
        ExposureEvent event = ExposureEvent.builder()
                .userId(userId)
                .experimentName(experimentName)
                .variant(variant)
                .timestamp(Instant.now())
                .build();

        httpClient.preparePost("/v1/events/exposure")
                .setBody(JsonUtil.toJson(event))
                .setRequestTimeout((int) TIMEOUT.toMillis())
                .execute()
                .toCompletableFuture()
                .exceptionally(ex -> {
                    metricsRecorder.incrementCounter("exposure_event_emit_failed");
                    return null; // swallow - never propagate to checkout flow
                });
    }
}

6.3 Java Code Example: Sequential Significance Check

public class SequentialStatsEngine {

    // Uses a sequential testing boundary rather than a fixed-sample p-value,
    // so results can be checked at any time without inflating false positives.
    public ExperimentVerdict evaluate(VariantAggregate control, VariantAggregate treatment) {
        double controlRate = control.getConversions() / (double) control.getExposures();
        double treatmentRate = treatment.getConversions() / (double) treatment.getExposures();

        double zScore = computeSequentialZScore(control, treatment);
        double currentBoundary = sequentialBoundary(control.getExposures() + treatment.getExposures());

        if (Math.abs(zScore) > currentBoundary) {
            String winner = treatmentRate > controlRate ? "TREATMENT" : "CONTROL";
            return ExperimentVerdict.significant(winner, controlRate, treatmentRate);
        }
        return ExperimentVerdict.inconclusive(controlRate, treatmentRate);
    }

    private double computeSequentialZScore(VariantAggregate control, VariantAggregate treatment) {
        double p1 = control.getConversions() / (double) control.getExposures();
        double p2 = treatment.getConversions() / (double) treatment.getExposures();
        double pooled = (control.getConversions() + treatment.getConversions())
                / (double) (control.getExposures() + treatment.getExposures());
        double se = Math.sqrt(pooled * (1 - pooled)
                * (1.0 / control.getExposures() + 1.0 / treatment.getExposures()));
        return (p2 - p1) / se;
    }

    private double sequentialBoundary(long totalSampleSize) {
        // Boundary widens for small samples and tightens as sample size grows,
        // following an alpha-spending function to control cumulative false-positive risk.
        return 1.96 + (5.0 / Math.sqrt(totalSampleSize + 1));
    }
}

Every architectural choice above buys something and pays for something. Let’s put the trade-offs on a single table before revisiting the highest-stakes ones.

It’s worth noting explicitly that these trade-offs are not one-time architectural decisions made once and forgotten — they need periodic revisiting as the platform’s usage grows. A configuration that was perfectly reasonable when the team ran five concurrent experiments might need real reconsideration once dozens of teams are running hundreds of experiments simultaneously across every corner of the checkout flow, at which point experiment layering, guardrail thresholds, and even the aggregation window sizes discussed later in this tutorial may all need retuning to match the platform’s new scale of usage.

8.1 Back-of-the-Envelope Capacity Estimation

The prompt specifically calls for a “million requests in a minute” scenario, so let’s ground the design in real numbers.

• Target sustained load: 1,000,000 requests/minute ≈ 16,667 requests/second sustained, with realistic peak bursts commonly 2–3x that, so provisioning for roughly 40,000–50,000 requests/second at burst is a safer target.
• Checkout page loads: If each page load triggers one bucketing decision (resolved locally, effectively free) and one exposure event, the Event Ingestion tier alone needs to sustain on the order of 16,000–50,000 events/second just from exposures, before adding conversion events, retries, and other instrumentation.
• Kafka partitioning: With, say, 200 concurrent active experiments and topics partitioned by experiment id, an even distribution gives each partition roughly 80–250 events/second at target load — comfortably within a single partition’s throughput capacity, while still allowing horizontal scale-out by adding partitions if a single very-high-traffic experiment needs more parallelism.
• Aggregate store write volume: Rather than writing one row per raw event, the Stream Processing Layer pre-aggregates into short (for example, 10-second) windowed counters per experiment and variant, collapsing tens of thousands of raw events per second into a vastly smaller number of aggregate upserts per second — this pre-aggregation step is what keeps the downstream metrics store from becoming the bottleneck.
• Config propagation: Even at this event volume, the Experiment Config Service itself sees comparatively tiny traffic (occasional pushes to CDN edge nodes, not once per checkout request), so it can be provisioned far more conservatively than the ingestion or checkout tiers.

This estimate shows the key insight driving the whole design: the “hot,” latency-critical path (bucketing) is engineered to cost effectively zero at any scale because it’s local computation, while the genuinely high-volume component (event ingestion) is a thin, horizontally-scalable, purely-additive write path that hands off to a queue immediately rather than doing expensive work inline.

8.2 Horizontal Scaling of the Ingestion Tier

The Event Ingestion Service is fully stateless and autoscales on request rate and CPU, sitting behind its Load Balancer. Because its only job is lightweight validation plus a publish to Kafka, individual pods can sustain very high throughput, and the tier scales near-linearly by adding pods.

8.3 Kafka Partitioning Strategy

Partitioning by experiment id keeps all events for one experiment ordered and co-located for the Stream Processing Layer, while spreading load across many partitions for parallelism. A very high-traffic single experiment (for example, a sitewide checkout redesign) can be given extra partitions specifically to avoid a hot-partition bottleneck.

In practice, it’s worth explicitly planning for the uneven distribution of traffic across experiments rather than assuming every experiment gets an equal share. A sitewide checkout redesign experiment might account for a large fraction of all checkout traffic, while dozens of smaller, more targeted experiments (say, testing a shipping-method label change for one specific product category) each see comparatively tiny volume. Assigning a fixed number of partitions per experiment regardless of its traffic share would leave the high-traffic experiment’s partitions overloaded while dozens of low-traffic experiments sit on mostly-idle partitions. A better approach ties partition allocation to expected traffic share at experiment creation time, with the option to dynamically add partitions to an already-running high-traffic experiment if its actual volume exceeds the original estimate — Kafka supports increasing partition counts for an existing topic, though consumer-side logic needs to tolerate the resulting temporary rebalance.

8.4 Pre-Aggregation and Windowing

The Stream Processing Layer uses short tumbling or sliding windows (for example, 10–30 seconds) to pre-aggregate raw events into per-variant counters before they ever reach the Aggregate Metrics Store, turning a firehose of individual events into a small, steady stream of counter increments — this is the single biggest lever for keeping the metrics store’s write volume manageable at scale.

8.5 Caching the Experiment Config at the Edge

Experiment configuration is small, changes infrequently relative to request volume, and is read on effectively every checkout page load — a textbook case for aggressive edge caching. Pushing config to the CDN and to client-side SDK caches, refreshed every few seconds, means the vast majority of requests never need to contact any backend service at all for the assignment decision.

This chapter first walks through the reliability principles that keep the purchase path safe, then covers the CAP theorem, sequential-testing statistics, algorithms, and disaster recovery considerations that together turn this from “a working experiment platform” into “a trustworthy one.”

9.1 Never Block the Purchase Path

This is the single most important reliability principle in the whole system: every experiment-related call (bucketing, event emission) must be non-blocking, with aggressive short timeouts and safe fallbacks. If the SDK’s local config cache is somehow empty or stale (a rare failure case), it falls back immediately to the default and control variant rather than waiting on any network call — a shopper should never see a broken or delayed checkout page because of an experimentation glitch.

9.2 Idempotent Event Processing

Because event emission can be retried (client-side network hiccups, at-least-once Kafka delivery semantics), each event carries a unique event id, and the Stream Processing Layer deduplicates on this id within its aggregation window, so a retried event never double-counts a conversion.

9.3 Circuit Breakers and Graceful Degradation

Calls from the Event Ingestion Service to Kafka are wrapped in a circuit breaker; if Kafka becomes briefly unavailable, the ingestion service buffers events locally for a short bounded period and drops the oldest events first if the buffer fills, rather than blocking or crashing — a brief gap in experiment data is an acceptable trade-off, a crashed ingestion tier affecting checkout availability is not.

9.4 Multi-Region Deployment

The Checkout Service, Event Ingestion Service, and edge config caches are deployed across multiple regions, with the CDN and Load Balancers routing shoppers to their nearest healthy region. The Aggregate Metrics Store and Statistics Engine can operate on a slightly relaxed, eventually-consistent view across regions, since experiment decisions don’t need sub-second global consistency to be useful. This matters in practice at global e-commerce scale: a shopper in one region should never experience degraded checkout performance because of an issue localised to infrastructure serving a completely different region.

9.5 Fast, Reliable Kill Switches

The Auto Kill Switch Service is treated as one of the most critical, highest-reliability components in the system, since its entire purpose is limiting damage from a bad experiment. It’s deployed with extra redundancy and a fast propagation path directly into the edge config push mechanism, so a detected guardrail breach can reach essentially all edge nodes within single-digit seconds. This service is also given its own independent monitoring and alerting, separate from the general system dashboards, specifically because if the kill switch itself silently fails, the team loses their primary safety net exactly when they need it most — a failure mode worth designing against deliberately rather than assuming away.

9.6 CAP Theorem in This System

Different parts of this system make deliberately different Consistency-vs-Availability trade-offs:

• Bucketing decision: Strongly favours Availability — a shopper must always get a fast, deterministic assignment even if their local config is a few seconds stale relative to the very latest experiment definition update. A brief propagation delay for a brand-new experiment is a completely acceptable cost.
• Order and payment writes: Strongly favours Consistency, exactly as in any e-commerce checkout system — you cannot have an eventually-consistent view of whether a payment succeeded.
• Aggregate experiment metrics: Favours Availability and eventual consistency — a dashboard showing metrics that are a few seconds behind real time is entirely fine for a business decision made over days, not milliseconds.

9.7 Why Sequential Testing Matters (and the Statistics Behind It)

Classic fixed-sample statistical significance testing assumes you decide your sample size in advance and only look at the result once, at the end. In practice, teams want to monitor a live experiment continuously and often stop it as soon as it “looks” significant — and doing this with a naive fixed-sample p-value dramatically inflates the real false-positive rate, because you’re effectively giving yourself many chances to get a lucky-looking result. Sequential testing methods (such as alpha-spending functions or always-valid confidence sequences) solve this by using a significance boundary that mathematically accounts for the fact that the data is being checked repeatedly over time, keeping the true false-positive rate controlled no matter when a team chooses to look at or stop the experiment. This is precisely why the code example in section 6.3 uses a sample-size-dependent boundary rather than a single fixed 1.96 z-score cutoff.

9.8 Data Structures and Algorithms Under the Hood

9.9 Disaster Recovery, Backup and Cost Optimisation

Raw experiment events are retained in Kafka for a short window (days) and then archived into a data lake or warehouse for cheaper, longer-term storage and deep post-hoc analysis, with periodic restore drills confirming the archival pipeline actually works. Since pre-aggregation dramatically shrinks the write volume hitting the more expensive Aggregate Metrics Store, the biggest cost lever in this system is tuning the aggregation window size correctly — too short wastes compute and storage on near-duplicate writes, too long delays how quickly a guardrail breach can be detected, directly trading cost against reaction speed.

Because this system controls what the checkout page actually renders and does, its security posture is inseparable from checkout’s own — a compromise here can directly redirect revenue or silently poison experiment results.

• Authentication & authorisation: The API Gateway enforces authentication for all traffic; only authorised product and experimentation teams can create or modify experiment configs through the Experiment Config Service, gated by role-based access control.
• Config integrity: Experiment configuration pushed to the edge is signed, and the client SDK verifies the signature before applying it, preventing a compromised CDN node or man-in-the-middle from silently injecting a malicious “variant” that alters checkout behaviour.
• Event validation: The Event Ingestion Service performs schema and sanity validation (plausible timestamps, known experiment ids, rate limits per client id) to prevent malformed or spoofed events from corrupting experiment results.
• Least privilege for the Kill Switch Service: It has narrow, specific permission to push config changes only, not broader system access, limiting blast radius if it were ever compromised.
• PII handling: User identifiers used for bucketing are hashed or pseudonymised wherever possible in logs and analytics stores, and raw event data retention follows the platform’s standard data-privacy and deletion policies.
• Encryption: TLS in transit everywhere; sensitive order and payment data at rest is encrypted, following the same standards as the rest of the checkout system.
• Secrets management: Credentials for signing experiment config bundles and for internal service-to-service authentication are stored in a dedicated secrets manager and rotated regularly, never embedded in client-distributed code or configuration.
• Web Application Firewall (WAF): Sits in front of the API Gateway to filter common attack patterns before they reach the Event Ingestion or Checkout Service tiers, providing a defence layer above application-level validation.

Because this system silently affects revenue, its monitoring has to be tuned specifically to catch failures that would otherwise go unnoticed until a team spots them in a dashboard days later.

11.1 Key Metrics to Track

11.2 Designing Good Alerts, Not Just Dashboards

A small number of high-signal alerts matter far more than a large dashboard nobody watches in real time:

• SRM alert: Automatically flags and can auto-pause any experiment whose actual traffic split deviates statistically significantly from its configured split, since this almost always indicates a real bug rather than random noise.
• Guardrail breach alert: Pages the responsible team (and can trigger the Auto Kill Switch) the moment a monitored guardrail metric crosses a predefined danger threshold for any active variant.
• Pipeline health alert: Fires if Kafka consumer lag or event-ingestion error rates spike, since a stalled pipeline means the team is currently flying blind on live experiment results.
• Latency parity alert: Fires if checkout page latency for any variant diverges meaningfully from others, since the experimentation system itself adding latency would contaminate the very conversion-rate result being measured.

• Kubernetes orchestrates all stateless service tiers (Checkout, Config, Ingestion, Statistics, Kill Switch, Dashboard), each with its own Horizontal Pod Autoscaler tuned to its own traffic pattern — ingestion scales on raw request volume, statistics scales more on CPU-bound aggregation work.
• CDN and edge compute host the bucketing SDK’s config bundle and, where supported, can run the bucketing logic directly at the edge (edge functions) for regions far from the origin, further minimising latency.
• Managed Kafka (or an equivalent managed streaming platform) provides the event backbone without the operational overhead of self-managed brokers, with topic partition counts tuned per the capacity estimate in section 8.1.
• Managed databases with cross-region read replicas back the Experiment Config DB and Results DB; the Aggregate Metrics Store is typically a time-series-optimised store chosen for high write throughput of small, frequent counter updates.
• CI/CD with canary deployment is used especially for the Statistics Engine and Kill Switch Service, since bugs there directly affect whether experiments are correctly measured and whether bad variants get shut down in time.
• Infrastructure as Code version-controls Kafka topic configuration, autoscaling policies, and edge caching rules, so capacity changes ahead of a known high-traffic event (like a planned flash sale) are reproducible and reviewable rather than manual, ad-hoc changes.

13.1 Experiment Config Database

Low write volume, read-light in production (since reads are served from edge caches, not this database directly) — a straightforward replicated relational or document store is sufficient, prioritising correctness and ease of administration over raw throughput.

13.2 Aggregate Metrics Store

This store needs to sustain frequent small counter increments (per experiment, per variant, per short time window) at high volume. A time-series-oriented or wide-column store, sharded by experiment id, fits this access pattern well — reads are typically “give me this experiment’s counters over the last N hours,” which maps naturally to a time-partitioned schema.

13.3 Sharding Strategy in Detail

• Aggregate Metrics Store sharding key: Sharded by experiment id, since nearly all queries are scoped to a single experiment; this keeps a given experiment’s full metric history co-located for fast dashboard queries.
• Replication: Leader-follower replication with reads served from followers for dashboard traffic, reserving leader reads for the Statistics Engine’s near-real-time significance checks where freshness matters more.
• Hot experiment handling: A small number of very high-traffic, sitewide experiments (like a full checkout redesign) can be given dedicated shard capacity, similar in spirit to how Kafka partition counts are increased for high-traffic experiments.

13.4 Distributed Cache

Beyond the client-side and edge config cache, an internal Redis-based cache can hold the most recently computed Statistics Engine verdicts, so the Dashboard tier can serve near-instant reads without recomputing significance tests on every page view.

13.5 Load Balancing Strategy

Every internal tier uses health-check-based Load Balancing; the Event Ingestion tier specifically benefits from consistent-hash-based routing keyed on experiment id where feasible, so that a given experiment’s events tend to land on the same downstream processing path, simplifying local buffering and reducing cross-node coordination.

13.6 A Quick Side-by-Side of the Stores

Every service boundary in this system is a place where the “never block the purchase path” principle either holds or breaks. The three APIs below make that discipline explicit.

14.1 Exposure Event API

POST /v1/events/exposure
Body:
{
  "eventId": "evt-9f31-exposure",
  "userId": "user-88213",
  "experimentName": "checkout-single-page-v2",
  "variant": "TREATMENT",
  "timestamp": "2026-08-03T14:22:01Z"
}

Response 202 Accepted:
{ "status": "queued" }

14.2 Experiment Config Fetch API (Edge / CDN)

GET /v1/config/active-experiments
Response 200 OK:
{
  "experiments": [
    {
      "name": "checkout-single-page-v2",
      "active": true,
      "variants": [
        { "name": "CONTROL",   "bucketRange": [0, 5000] },
        { "name": "TREATMENT", "bucketRange": [5000, 10000] }
      ]
    }
  ],
  "configVersion": "2026-08-03T14:20:00Z"
}

14.3 Statistics Engine Results API

GET /internal/v1/experiments/checkout-single-page-v2/results
Response 200 OK:
{
  "controlConversionRate":   0.0412,
  "treatmentConversionRate": 0.0447,
  "verdict":                 "TREATMENT_SIGNIFICANT",
  "sampleSize":              812340,
  "guardrails": {
    "paymentFailureRate": "WITHIN_BOUNDS"
  }
}

Each service owns its own data store and communicates through well-defined APIs or asynchronous events, never by reaching directly into another service’s database — the Checkout Service, for example, has no direct dependency on the Statistics Engine or its database, keeping the purchase path fully decoupled from experimentation analysis.

15.1 Patterns Used

15.2 Anti-Patterns to Avoid

15.3 Testing Strategy for a Revenue-Critical System

• Shadow mode for new experiments: Before an experiment is allowed to actually alter what shoppers see, its assignment and logging pipeline can run in a dry-run mode, verifying correct bucketing distribution and event flow with zero customer-facing impact.
• Load testing at target and burst scale: The full pipeline is regularly load-tested at the million-plus-requests-per-minute target, and at realistic burst multiples above it, specifically validating that checkout latency stays flat regardless of experimentation traffic.
• Chaos testing on Kafka and ingestion: Deliberately killing ingestion pods and broker nodes in staging verifies that circuit breakers and local buffering behave as designed, and that checkout itself remains fully unaffected.
• Statistical simulation testing: The Statistics Engine’s sequential-testing logic is validated against simulated data with known, injected effect sizes (including a “no real effect” null case) to confirm its false-positive and false-negative rates match theoretical expectations before it’s trusted with real experiment decisions.

Large e-commerce and technology companies broadly converged on very similar architectural building blocks for experimentation platforms — the underlying skeleton (edge assignment, async event pipeline, sequential statistics, automated guardrails) shows up again and again because the core problem, “safely and quickly measure the effect of a change on a critical, high-volume user flow,” is fundamentally the same shape everywhere it appears.

The most common questions that arise when engineers first approach this design, answered directly and without hedging.

Real-time checkout A/B testing sits at the intersection of latency, scale, and statistics — let’s condense everything into the ideas worth carrying forward.