Watch any major live-streamed event today — a World Cup final, a product launch keynote, an awards show, a celebrity’s live stream — and you’ll see a second screen’s worth of activity happening in real time alongside the video itself: a scrolling chat, a stream of floating hearts and emoji, a live reaction counter ticking upward by the thousand. This layer is not decoration.

For many viewers, the shared, real-time reaction of the crowd is a large part of why they watch live at all, rather than waiting to watch a recording later. Building the system that powers this — reliably, at low latency, for tens of millions of simultaneous participants — is one of the most demanding real-time fan-out problems in modern software engineering.

1.1 A short timeline of how we got here

Live group chat did not spring into existence at Twitch scale — each generation of the internet added one more constraint or expectation on top of the previous one, and today’s systems are the compressed accumulation of thirty years of that evolution.

• Late 1990s — IRC and the birth of live group chat. Internet Relay Chat established the basic pattern still used today: a client connects, joins a “room,” and receives every message broadcast to that room in near real time. IRC servers could handle thousands of users per room — a useful pattern, but nowhere near the scale modern live events require.
• 2011 — Twitch popularizes live chat alongside video. Twitch paired a live video stream with a persistent chat room, proving that synchronized, real-time text chat significantly increases engagement and watch time for live content, and setting the template that nearly every live-streaming product has followed since.
• 2015-2016 — Facebook Live and floating reactions. Facebook Live introduced animated, floating reaction bursts (hearts, likes, laughs) layered directly over the video during a live broadcast, turning passive viewing into a visibly participatory experience — and creating a new engineering challenge: aggregating an enormous burst of low-information-content reaction events without collapsing under the load.
• 2018 – present — Massive-scale global live events. Major sporting events, award shows, and product launches now routinely draw tens of millions of concurrent live viewers across streaming platforms, each expecting real-time chat and reactions that don’t lag, drop messages, or collapse under load exactly when the event reaches its most exciting moment — precisely the moment traffic spikes hardest.

This tutorial designs that system end to end: how millions of persistent real-time connections are managed, how a flood of chat messages and reactions is fanned out efficiently without duplicating enormous amounts of network traffic, how reaction bursts are aggregated instead of sent one-by-one, how the system stays healthy under sudden traffic spikes, and how moderation keeps the experience usable rather than a wall of spam. Code examples are in Java, and sections interviewers commonly probe are called out explicitly, since this is a frequently asked system design interview question at companies building live-streaming and social products, precisely because it forces a candidate to reason clearly about the difference between what a system merely could deliver in principle and what it actually needs to deliver to create a convincing, shared sense of liveness for its audience.

Why is broadcasting chat and reactions to millions of people simultaneously fundamentally different from a typical chat application? The differences are not just about size — they change the nature of the problem entirely.

2.1 Why sending every message to every viewer doesn’t work

Consider a naive design: every chat message and every reaction tap is individually sent to every connected viewer, the same way a small group chat works. For an event with ten million concurrent viewers and a modest 10,000 chat messages per second during a peak moment, that naive design would require sending 10,000 multiplied by 10,000,000, or one hundred billion individual message deliveries per second. No realistic infrastructure budget can sustain that, which is exactly why real systems never actually deliver every individual message or reaction to every viewer — instead, they sample, batch, aggregate, and selectively broadcast, trading a small amount of individual-message fidelity for the ability to operate at all at this scale.

2.2 A worked capacity example

It’s worth working through the arithmetic once to see exactly why aggregation is not optional. Suppose an event has ten million concurrent viewers, and during its most exciting thirty seconds, the platform receives one million reaction taps per second — a very plausible number for a major global sporting moment. Without aggregation, delivering each tap individually to every viewer would require one million multiplied by ten million deliveries per second, or ten quadrillion messages per second — a number so far beyond any realistic infrastructure that the naive design isn’t merely expensive, it is simply impossible to build at any budget. Now apply a 200-millisecond aggregation window: instead of one million individual tap messages per second, the aggregator emits perhaps five batched update messages per second per reaction type (one every 200 milliseconds), each fanned out to the gateway fleet and then to connected viewers. The total volume of meaningful, distinct messages that actually need to traverse the fan-out layer drops from millions per second to a small, easily-manageable handful — a reduction of many orders of magnitude, achieved without losing the felt sense of a massive, energetic crowd reacting together.

Before any architecture, we need shared vocabulary. Every design decision later in this tutorial reduces to a choice between these primitives.

3.1 Persistent connections: WebSocket, SSE, and long polling

To push real-time updates to a client without the client having to repeatedly ask “anything new?”, the server needs a way to send data whenever it wants. A WebSocket is a full-duplex, persistent TCP-based connection that, once established via an HTTP handshake, lets the server and client send messages to each other at any time with very low overhead per message. Server-Sent Events (SSE) is a simpler, one-directional alternative, well suited to this use case since a viewer mostly only receives updates (chat and reactions) rather than needing to send high-frequency data back over the same channel. Long polling, an older technique where the client repeatedly issues HTTP requests that the server holds open until there’s new data, is generally avoided at this scale due to its higher per-message overhead, though it occasionally remains useful as a compatibility fallback for older client environments that cannot establish a true WebSocket connection, at the cost of somewhat higher latency and server resource consumption for that smaller fallback population.

3.2 Fan-out on write vs. fan-out on read

This is the same fundamental trade-off found in many large-scale feed and messaging systems. Fan-out on write pushes each new message out to every connected recipient’s channel immediately as it arrives. Fan-out on read instead stores the message centrally and lets each client pull or subscribe to updates from that central store. Live chat systems typically use a fan-out-on-write model at the connection-gateway layer (a message published once is pushed out to all subscribed gateway servers, which push it to their connected clients), since read latency must be near-instant and the “room” size, while huge, is still bounded and known.

3.3 Publish–subscribe (pub-sub)

A pub-sub system lets many publishers send messages to a named channel or topic, and many subscribers receive every message published to that channel, without publishers and subscribers needing to know about each other directly. This is the natural backbone pattern for a live event’s chat: the event has one (or a few) logical “room” topics, chat services publish new messages to that topic, and every connection-gateway server subscribes to it in order to push messages down to its own locally-connected viewers.

3.4 Backpressure

Backpressure is the mechanism by which a system signals “slow down, I can’t keep up” to whatever is sending it data, rather than silently dropping data or, worse, crashing under an unbounded queue. In a live commenting system, backpressure matters at multiple layers: a slow client connection shouldn’t be allowed to make the server buffer unboundedly on its behalf, and a sudden burst of publisher traffic shouldn’t be allowed to overwhelm downstream consumers faster than they can process it.

3.5 Aggregation and sampling

Rather than delivering every individual event, the system frequently aggregates many small events (like reaction taps) into a single, periodic summary (like “42,381 hearts in the last second”), and samples a representative subset of a much larger stream (like showing each viewer only a portion of the full chat firehose) rather than delivering the complete stream to everyone. Both techniques trade a small amount of per-viewer completeness for a massive reduction in total system load, which is the central trade-off this entire system is built around.

3.6 Sharding a single logical room

Even though viewers experience an event as one single, unified “room,” the underlying implementation for a massive event typically shards that logical room across multiple physical pub-sub partitions or topics, commonly using a hash of viewer ID or a geographic dimension to decide which partition a given connection belongs to. Each gateway instance only needs to subscribe to the partitions relevant to its currently-connected viewers, which keeps any single partition’s subscriber count and message volume within a manageable range, rather than concentrating the entire event’s traffic onto one unpartitioned topic that would eventually become a bottleneck no matter how much hardware is thrown at consuming it.

3.7 Eventual consistency and why it’s the right trade-off here

A strictly consistent system would guarantee that every viewer sees chat messages and reaction counts in exactly the same order, updated at exactly the same instant. Achieving that at ten-million-viewer scale would require an enormous amount of cross-node coordination, adding latency and fragility that would work directly against the system’s actual goal of feeling instantaneous. Instead, this system deliberately embraces eventual consistency: different viewers may see slightly different subsets or orderings of chat messages, and reaction counters converge to the same value only after a short propagation delay, not necessarily at the exact same millisecond, across every connected client. This is an entirely acceptable, even invisible, trade-off for this specific product, precisely because no individual viewer has any way to detect or care about a few hundred milliseconds of divergence from what a different viewer, elsewhere in the world, happens to be seeing at that exact same instant.

The major building blocks that together support real-time chat and reactions at massive scale. Everything downstream of “the client tapped a heart” happens inside this diagram.

4.1 Edge / CDN layer

The first point of contact for a viewer’s connection, typically a globally-distributed edge network that terminates the initial connection close to the user geographically, reducing round-trip latency before traffic is routed to the nearest regional cluster of connection gateways.

4.2 Connection Gateway Cluster

A large horizontally-scaled fleet of servers that each hold open a large number of persistent WebSocket or SSE connections to viewers. Each gateway instance subscribes to the pub-sub topic(s) for the events its connected viewers are watching, and relays incoming messages down to those local connections. This is the layer that must scale to support the sheer count of concurrent open connections.

4.3 Ingest Service

Receives incoming chat messages and reaction taps from clients, performs lightweight validation (message length, basic well-formedness, authentication) and rate limiting per user, then routes chat text toward moderation and reaction taps toward the aggregator.

4.4 Moderation Pipeline

Applies automated filtering — profanity lists, spam heuristics, and machine-learning classifiers for harassment or policy violations — to chat messages before they’re published to the message bus, with an asynchronous path for messages needing deeper human review after initial publication.

4.5 Pub-Sub Message Bus

The shared backbone (commonly Kafka, Redis Streams, or a comparable system) that every connection gateway subscribes to for a given event’s topic. Publishing a message once here reaches every subscribed gateway, which is what allows the system to avoid the sender having to know about every individual downstream viewer.

4.6 Reaction Aggregator

Collects a torrent of individual reaction taps and periodically emits batched summaries (a running count, or a small burst of representative animated icons) onto the message bus, rather than forwarding every single tap as its own message.

4.7 Room / Session Registry

Tracks which viewers (and which gateway instances) are currently connected to which event, supporting operations like current-viewer-count display and directing new connections to the correct topic subscriptions.

4.8 Message Store

A durable store retaining the full, unsampled history of chat messages and raw reaction events for moderation review, post-event analytics, and enabling features like chat replay alongside a recorded version of the event afterward.

4.9 Notification and Highlight Service

A smaller, complementary service that watches the moderated chat and aggregated reaction streams for signals worth surfacing beyond the live room itself — for example, detecting an unusually sharp reaction spike that might indicate a highlight-worthy moment, or triggering a push notification to users who follow the event’s creator but aren’t currently watching. This service consumes from the same message bus as everything else, following the same decoupled, subscribe-only pattern, and its addition or removal has no impact whatsoever on the core chat and reaction delivery path, which is exactly the kind of clean extensibility a well-designed pub-sub backbone is meant to provide.

Step by step, how a single chat message and a single reaction tap actually travel through the system, from the tap on a viewer’s screen to the pixels that render on millions of others.

5.1 Life of a chat message

1. Client sends a chat message over its open WebSocket connection to its connected gateway instance.
2. Gateway forwards the raw message to the Ingest Service, tagged with the sender’s authenticated identity and the event/room ID.
3. Ingest validates and rate-limits: checks message length and basic formatting, and enforces a per-user rate limit (for example, no more than one message every few seconds) to prevent a single account from flooding the room.
4. Moderation screens the message through fast, synchronous automated checks (profanity/spam filters, a lightweight ML classifier); messages that clearly violate policy are dropped immediately, while borderline messages may be allowed through with a flag for asynchronous human review.
5. Publish to the bus: the message is published to the pub-sub topic for that event’s room.
6. Fan-out to gateways: every connection gateway instance subscribed to that topic receives the message.
7. Local delivery: each gateway instance decides, based on its own sampling/rate policy, which of its locally-connected viewers actually receive this particular message in their live chat view.
8. Durable write: in parallel, the message is persisted to the message store for moderation audit trails and later replay.

5.2 Life of a reaction tap

1. Client taps a reaction icon; the tap is sent as a minimal, lightweight event (often just an event type and timestamp) to the gateway.
2. Gateway forwards the tap to the Ingest Service, typically over a cheaper, less strictly-ordered path than chat messages, since individual reaction taps carry very little unique information.
3. Aggregator accumulates taps in a small, fixed-size time window (commonly a few hundred milliseconds), maintaining a running count per reaction type for the event.
4. Periodic flush: at the end of each window, the aggregator publishes a single batched update (“+38,204 hearts”) to the bus, rather than one message per tap.
5. Fan-out and render: gateways relay the batched update to connected clients, which render it as an incrementing counter and/or a burst of animated icons representing the aggregate, not one animation per individual tap.

How a viewer’s connection, and the event itself, moves through distinct phases over its lifetime — from cold start to the sharp spike to a graceful teardown.

6.1 Connection establishment

When a viewer opens the live event, their client authenticates and requests a real-time connection. A load balancer or the edge layer routes them to a connection gateway instance, typically chosen based on current load and geographic proximity. The gateway registers the connection against the event’s room in the session registry and subscribes (if not already subscribed, since many viewers on the same gateway share one upstream subscription) to that event’s pub-sub topic.

6.2 Steady-state streaming

For the duration of the event, chat messages and aggregated reaction updates flow continuously through the pipeline described in Section 5. The system also periodically pushes lighter-weight presence data, such as an approximate current-viewer count, which is itself typically computed as an aggregate rather than an exact live count for the same scalability reasons as reactions.

6.3 Traffic spikes at key moments

Live events are defined by unpredictable but highly correlated spikes — a goal, a surprise announcement, a dramatic reveal — where chat and reaction volume can jump by an order of magnitude within seconds. The system must absorb these spikes gracefully: aggregation windows and rate limits automatically smooth the load without any manual intervention, and auto-scaling policies (discussed in Section 9) add gateway and processing capacity ahead of anticipated high-traffic segments of major, pre-scheduled events.

6.4 Connection teardown and event end

As viewers leave (closing the app, event ending), their connections are torn down, and the session registry is updated. When an event ends, the system typically transitions the room from a live, high-throughput mode into a lower-throughput or read-only replay mode, and the pub-sub topic for that event can eventually be retired, freeing up gateway subscription capacity for other concurrent live events on the platform.

The concrete mechanics of aggregation, rate limiting, and sampling that make this system work — each chosen to trade a small, well-understood amount of fidelity for the ability to run at scale at all.

7.1 Windowed reaction aggregation

The aggregator maintains a simple in-memory counter per reaction type, reset at the end of each fixed time window (commonly 100–300 milliseconds — short enough to feel real-time, long enough to meaningfully compress volume).

import java.util.concurrent.atomic.LongAdder;
import java.util.concurrent.*;

// Accumulates reaction taps in fixed windows and flushes a single
// batched update per window instead of one message per tap.
public class ReactionAggregator {

    // LongAdder scales far better than AtomicLong under high write
    // contention from many concurrent threads incrementing the same counter.
    private final ConcurrentHashMap<String, LongAdder> counts = new ConcurrentHashMap<>();
    private final String eventId;
    private final MessageBusPublisher publisher;

    public ReactionAggregator(String eventId, MessageBusPublisher publisher) {
        this.eventId = eventId;
        this.publisher = publisher;
    }

    // Called on the hot path for every incoming reaction tap.
    public void recordTap(String reactionType) {
        counts.computeIfAbsent(reactionType, k -> new LongAdder()).increment();
    }

    // Called on a fixed schedule (e.g., every 200ms) by a background timer.
    public void flushWindow() {
        for (var entry : counts.entrySet()) {
            long windowCount = entry.getValue().sumThenReset();
            if (windowCount > 0) {
                publisher.publish(eventId, new ReactionBatch(entry.getKey(), windowCount));
            }
        }
    }
}

7.2 Token bucket rate limiting per user

To stop any single account from flooding a room with chat messages, each connection is governed by a token bucket: a bucket holds a small number of tokens, one token is consumed per message sent, and tokens refill at a steady rate over time. If the bucket is empty, further messages are rejected or queued until a token becomes available.

public class TokenBucketRateLimiter {
    private final double capacity;
    private final double refillRatePerSecond;
    private double tokens;
    private long lastRefillTimestampNanos;

    public TokenBucketRateLimiter(double capacity, double refillRatePerSecond) {
        this.capacity = capacity;
        this.refillRatePerSecond = refillRatePerSecond;
        this.tokens = capacity;
        this.lastRefillTimestampNanos = System.nanoTime();
    }

    public synchronized boolean tryConsume() {
        refill();
        if (tokens >= 1.0) {
            tokens -= 1.0;
            return true;
        }
        return false;  // caller should drop or throttle the message
    }

    private void refill() {
        long now = System.nanoTime();
        double elapsedSeconds = (now - lastRefillTimestampNanos) / 1_000_000_000.0;
        tokens = Math.min(capacity, tokens + elapsedSeconds * refillRatePerSecond);
        lastRefillTimestampNanos = now;
    }
}

7.3 Chat sampling for extreme rooms

For a room with millions of active chatters, even a heavily rate-limited stream can exceed what any single viewer’s client should reasonably render (a scrolling chat updating thousands of times per second is unreadable and wasteful to transmit). Gateways apply a reservoir sampling-style approach per delivery window: rather than forwarding every message that passes moderation, each gateway forwards a bounded, randomly-sampled subset to each connected client, sized to keep the visible chat feeling active and fast without overwhelming the client or the network link.

7.4 Estimating unique reactors with HyperLogLog

A raw reaction count answers “how many total taps happened,” but a related and often more interesting question for analytics and for certain product surfaces is “how many distinct people reacted,” since one enthusiastic viewer tapping the heart icon fifty times should not be conflated with fifty different viewers each tapping it once. Counting exact distinct reactors at this scale would require tracking a set of user IDs proportional in size to the total viewer count, which becomes expensive in memory when this needs to be tracked continuously across many concurrent live events. A HyperLogLog sketch estimates the number of distinct elements in a stream using a small, fixed amount of memory (typically a few kilobytes) regardless of how many total elements pass through it, trading a small, well-understood margin of statistical error for a massive reduction in memory footprint. This makes it practical to maintain a live “distinct reactors” estimate per event, per reaction type, continuously, without the memory cost scaling with audience size.

import com.clearspring.analytics.stream.cardinality.HyperLogLog;

// Maintains an approximate distinct-reactor count per event and
// reaction type using a fixed-size probabilistic sketch.
public class DistinctReactorEstimator {

    private final ConcurrentHashMap<String, HyperLogLog> sketches = new ConcurrentHashMap<>();

    // standardError of ~0.02 (2%) is a common, practical default.
    private HyperLogLog newSketch() {
        return new HyperLogLog(0.02);
    }

    public void recordReactor(String eventReactionKey, String userId) {
        sketches.computeIfAbsent(eventReactionKey, k -> newSketch())
                .offer(userId);
    }

    public long estimateDistinctReactors(String eventReactionKey) {
        HyperLogLog sketch = sketches.get(eventReactionKey);
        return sketch == null ? 0 : sketch.cardinality();
    }
}

7.5 Complexity and cost comparison

As with the earlier mutual-friends style of system, no single row here is universally “correct” — the right technique depends on whether the consuming feature needs an exact total, a distinct-entity estimate, or just a representative sample, and how much memory or compute budget that feature can justify at the event’s expected scale.

Every scalability technique here trades some fidelity or complexity for the ability to operate at all at this scale. Naming those trades explicitly is what separates a defensible design from a lucky one.

The consistent theme is that scale is purchased by deliberately giving up perfect, individual-event fidelity in exchange for a system that can survive contact with tens of millions of simultaneous participants — and, done well, viewers never notice the trade, because what’s preserved (the feel of a real-time, shared crowd reaction) is exactly what they actually came for.

The concrete engineering techniques that let this system sustain millions of concurrent connections and huge message throughput without ever exposing the customer to the strain underneath.

9.1 Horizontal scaling of connection gateways

No single machine can hold ten million connections. The gateway layer is horizontally scaled across a large fleet, with each instance holding a manageable share (commonly tens of thousands to low hundreds of thousands, depending on hardware and connection idle overhead) of the total connection count, and a load balancer or edge routing layer spreading new connections evenly across available capacity.

9.2 Efficient connection handling

Holding large numbers of mostly-idle persistent connections efficiently requires event-driven, non-blocking I/O (rather than one operating system thread per connection, which doesn’t scale to these numbers) — technologies like Netty in the Java ecosystem, or equivalent async I/O frameworks in other languages, are standard choices for this layer specifically because of their ability to multiplex huge connection counts over a small number of threads.

9.3 Pre-provisioned capacity for scheduled events

Unlike organic, gradually-growing traffic, a major scheduled live event (a championship match, a keynote) has a known start time and a roughly predictable audience size based on historical patterns and pre-registration signals. Capacity planning for these events typically pre-provisions gateway, aggregator, and message-bus capacity well ahead of the event start, rather than relying purely on reactive auto-scaling, since auto-scaling alone may not react fast enough to a traffic ramp that goes from near-zero to millions of connections within a few minutes of the event starting.

9.4 Regional sharding of large rooms

For extremely large events, the single logical “room” is often internally partitioned across multiple pub-sub topics or partitions (for example, by geographic region or by a hash of viewer ID), each handled by a subset of the gateway fleet, with a lightweight cross-partition merge step for globally-aggregated numbers like total reaction counts. This avoids any single topic or partition becoming a bottleneck on its own.

9.5 Batching at every layer

Beyond reaction aggregation, batching is applied wherever possible: gateways batch multiple queued outbound messages into a single network write to reduce per-message system-call overhead, and the message bus itself batches records for more efficient disk and network I/O internally.

9.6 Memory footprint per connection

A single idle WebSocket connection, including its associated buffers, session metadata, and subscription state, typically consumes somewhere in the range of tens of kilobytes of server-side memory, depending on implementation details. That number sounds trivial in isolation, but multiplied across the hundreds of thousands of connections a single gateway instance might hold, it becomes the dominant factor in how many connections that instance can safely support before memory pressure — not CPU — becomes the binding constraint. This is why gateway capacity planning tends to focus heavily on per-connection memory efficiency: trimming a few kilobytes of unnecessary per-connection state can translate into meaningfully higher connection density per instance, which directly reduces the total fleet size, and therefore cost, needed to support a given audience size.

9.7 Applying Little’s Law to the fan-out pipeline

The same Little’s Law relationship used for general request-serving capacity planning — average items in flight equals arrival rate multiplied by average time in system — applies directly to the message fan-out pipeline here. If the aggregator, message bus, and gateway layer together need to sustain an effective published-message rate of 5,000 batched updates per second system-wide, and the target end-to-end time from publish to client render is 150 milliseconds, then the pipeline must be able to comfortably hold roughly 750 messages “in flight” across its stages at any given moment without falling behind. Sizing internal queue depths, thread pool sizes, and consumer parallelism around this kind of concrete, derived number, rather than an arbitrary guess, is what keeps the pipeline’s latency stable and predictable even as load fluctuates during the natural ebb and flow of a long live event.

A live event has no “try again later” — an outage during the event’s peak moment is a total, unrecoverable failure of the feature for that moment. That constraint changes what “reliable” even means for this system.

10.1 Gateway instance failure and reconnection

If a gateway instance crashes or is taken out of service, its connected clients lose their connection and must reconnect, ideally to a different healthy instance. Client-side reconnection logic with exponential backoff and jitter (to avoid a reconnect storm hitting the same instant) ensures viewers recover automatically within a few seconds, and load balancers detect and stop routing to unhealthy instances quickly.

10.2 Message bus durability and replication

The pub-sub message bus is run as a replicated cluster (as is standard for systems like Kafka), so the loss of a single broker doesn’t interrupt the flow of chat and reactions for an ongoing live event — a scenario with a much higher cost of failure than most other messaging use cases, given the fixed, unrepeatable nature of a live moment.

10.3 Graceful shedding under extreme overload

If, despite pre-provisioning, load still exceeds capacity during an unprecedented spike, the system is designed to shed load gracefully rather than fail completely: increasing aggregation window sizes temporarily, reducing the chat sampling rate further, or in the most extreme case, temporarily pausing non-essential features (like reaction burst animations) while keeping core video delivery and a reduced chat experience alive. A degraded but functioning experience is far better than a full outage during the exact moment engagement is highest.

10.4 Multi-region failover

For globally significant events, infrastructure is typically deployed across multiple regions, with the ability to redirect traffic away from a struggling or failed region toward healthy ones, at the cost of some added latency for affected viewers during the failover window.

10.5 Quorum-based durability for the message bus

As with other replicated systems discussed elsewhere in this style of tutorial, the message bus typically requires a write to be acknowledged by a majority (a quorum) of its replicas before being considered durable, rather than waiting for every single replica or accepting acknowledgment from just one. This tolerates the failure of a minority of broker nodes without any interruption to the live event’s message flow, which matters enormously here given that a live event, unlike most other workloads, offers no opportunity to simply retry later — a moment that isn’t captured and delivered as it happens is lost for good, since replaying it after the fact defeats the entire purpose of “live.”

10.6 Chaos testing specific to bursty, time-boxed workloads

Because this system’s highest-risk moments are compressed into short, unpredictable windows during scheduled events, chaos testing here is particularly valuable when it combines two things at once: injected failures (killed instances, added network latency, simulated broker outages) and injected load spikes that resemble the sharp, correlated bursts real events produce, rather than testing either dimension in isolation. A gateway fleet that survives a instance failure under smooth, average load might behave completely differently — and fail — when that same failure happens during a ten-times-normal traffic spike, which is exactly the combination that matters most in production and therefore exactly the combination worth deliberately testing for ahead of time.

A real-time, high-volume, user-generated text stream is a natural target for abuse — moderation here has to be fast, not just accurate. Latency budgets and safety budgets have to coexist inside the same pipeline.

11.1 Automated moderation pipeline

Given the volume and speed involved, moderation cannot rely primarily on human review for the initial publish decision. A layered automated pipeline typically combines a fast keyword/pattern filter for clearly disallowed content, a lightweight machine-learning classifier scoring messages for spam, harassment, or policy violations, and per-user reputation signals (new accounts or accounts with recent violations may be held to stricter automated thresholds) — all evaluated within the tight latency budget of the ingest path.

11.2 Human-in-the-loop for borderline cases

Messages the automated pipeline scores as borderline, rather than clearly acceptable or clearly disallowed, can be allowed to publish immediately (favoring low latency for the common case) while being queued for asynchronous human moderator review, with the ability to retroactively remove content and apply consequences to the account if review confirms a violation.

11.3 Abuse and spam prevention

Beyond content moderation, the system defends against volumetric abuse: per-user rate limiting (Section 7.2) prevents any single account from flooding a room, CAPTCHA or step-up verification can be triggered for accounts exhibiting bot-like posting patterns, and IP- and device-level throttling helps mitigate coordinated abuse from many low-reputation accounts acting in concert.

11.4 Authentication and impersonation prevention

Every chat message and reaction is tied to an authenticated identity validated at connection time, and gateways reject any attempt to spoof a sender identity on messages arriving over an established connection, since a viewer’s connection is already bound to their verified session from the handshake.

11.5 Data protection

Chat content, while often less sensitive than data like private messages, still typically includes personally identifiable patterns and is protected with encryption in transit (TLS for all client and internal service connections) and access controls limiting who internally can query raw, unaggregated chat and reaction data outside of the authorized moderation and analytics pipelines.

11.6 Least-privilege access to raw chat data

The full, unsampled chat and reaction history retained for moderation and analytics is highly sensitive in aggregate, even if any individual message seems mundane in isolation, since patterns across many messages can reveal things about individual users that no single message would. Internal access to query this raw data directly is restricted to specific, audited tools and roles — content moderators, safety investigators, and authorized analytics pipelines — rather than being broadly queryable by any engineer, following the same least-privilege principle applied to other categories of sensitive user data across a responsible platform. Every access to raw, non-aggregated chat content through these tools is logged, and access patterns are periodically reviewed to catch both accidental over-exposure and deliberate misuse.

11.7 Encryption and secure transport

All client-to-gateway traffic runs over encrypted WebSocket connections (WSS, the secure variant of the WebSocket protocol, layered over TLS), and every internal service-to-service hop — ingest to moderation, moderation to the message bus, the message bus to gateways — is similarly encrypted in transit. Data retained in the durable message store is encrypted at rest, with encryption keys managed through a dedicated key management service rather than being embedded in application configuration, consistent with standard practice for any system handling data at this scale and sensitivity.

With traffic this bursty, by the time a human notices a problem by eye, it’s often already too late — the system needs to detect and often self-correct automatically.

12.1 Real-time dashboards for live events

Because live events are time-boxed and highly visible, teams typically staff a live “war room” or on-call rotation specifically during major scheduled events, watching dashboards purpose-built to surface the handful of metrics above at a glance, refreshed on the order of seconds rather than the minute-level granularity often acceptable for less time-sensitive systems.

12.2 Automated alerting and self-healing

Given how quickly conditions change during a live event, alerting thresholds are tuned to fire fast, and where possible, paired with automated remediation — for example, automatically widening the aggregation window or lowering the chat sampling rate if fan-out latency crosses a threshold, rather than waiting for a human to manually intervene during a fast-moving spike.

12.3 Post-event analysis

After each major event, engineering teams review the full metric history against capacity plans to refine pre-provisioning estimates, aggregation window tuning, and sampling rate defaults for future events, since each large live event is effectively a full-scale, high-stakes load test that produces valuable real-world data unavailable from synthetic testing alone.

12.4 Setting a realistic SLO for a bursty, time-boxed feature

Defining a Service Level Objective for this feature requires some care, because unlike a steady, continuously-running service, live-event traffic is concentrated into short, high-stakes windows where the cost of underperformance is unusually high relative to the total time the feature is even in heavy use. A reasonable approach separates the SLO into two distinct regimes: a looser target for ordinary, lower-traffic live events (for example, “99% of messages delivered within 500 milliseconds”), and a stricter, more heavily-resourced target specifically for known major events where both audience size and business stakes are highest. Treating every live event as equally important, and applying one single blanket SLO regardless of scale, tends to either over-invest in infrastructure for small, low-stakes streams or under-prepare for the rare, enormous events that actually generate the most user attention and the most risk.

12.5 Distributed tracing across the pipeline

Given how many independent services a single chat message or reaction passes through — gateway, ingest, moderation, aggregator, message bus, and back out through gateways again — distributed tracing (tagging each event with a trace identifier that’s propagated and logged at every stage) is essential for diagnosing where time is actually being spent when end-to-end latency degrades. Without tracing, an engineer investigating a latency spike is left correlating separate logs and metrics across half a dozen services by timestamp alone, which is slow and error-prone during exactly the kind of fast-moving incident where speed of diagnosis matters most. With tracing, the same investigation becomes a matter of pulling up a handful of representative traces and immediately seeing which specific stage of the pipeline is contributing the most latency.

How this system is packaged, scaled, and rolled out safely around real, scheduled, high-stakes events — where a bad deploy is a much bigger deal than usual.

13.1 Auto-scaling with pre-warmed capacity

The gateway and aggregation layers run on auto-scaling infrastructure (commonly Kubernetes-based), but for known major events, capacity is explicitly pre-warmed ahead of the scheduled start time rather than relying purely on reactive scale-out triggers, since new instances take time to boot, register, and become fully ready to accept connections — time the system may not have during a traffic ramp that goes from idle to peak within minutes.

13.2 Freezing risky changes around live events

Deployment of new code to the gateway, ingest, and moderation services is typically frozen or heavily restricted in a window immediately before and during major scheduled events, since even a well-tested change carries some risk, and that risk is far less acceptable during an irreplaceable live moment than during ordinary, lower-stakes traffic.

13.3 Multi-region deployment

Given a global audience for major events, gateway and edge infrastructure is deployed across multiple regions, routing each viewer to the nearest healthy region to minimize latency, with cross-region message bus replication ensuring viewers in different regions still see a consistent, synchronized chat and reaction stream for the same event.

13.4 Infrastructure as code and rehearsals

Scaling policies, topic/partition topology, and capacity plans for major events are defined declaratively and version-controlled, and large events are frequently preceded by a full rehearsal — a dry run against production infrastructure using synthetic traffic shaped like the expected real audience — to catch capacity or configuration issues before they can affect real viewers.

13.5 Coordinating a rehearsal across teams

A meaningful rehearsal for a major event is rarely just a load-testing exercise run in isolation by the infrastructure team; it typically involves coordinated participation from the on-call engineers, moderation operations staff, and product stakeholders who would all be involved in a real incident during the actual event. Running the rehearsal as a full dress-run — including simulated failures the on-call team must detect and respond to in real time, using the same dashboards and alerting they’d rely on during the genuine event — surfaces gaps that a purely technical capacity test alone would miss, such as an alert that’s misconfigured, a runbook step that’s outdated, or an escalation path that nobody actually remembers how to use under pressure. The goal of a rehearsal is not just to confirm the infrastructure can handle the expected load, but to confirm the humans operating that infrastructure are equally ready.

Choosing storage and traffic-distribution strategies suited to a workload that is overwhelmingly write-heavy and ephemeral in nature. “Database” in this system is more like “river” than “warehouse.”

14.1 The message bus as the primary real-time data path

Unlike many systems where a database is the primary source of truth on the hot path, here the pub-sub message bus itself is the primary real-time data path — gateways read directly from it to serve live traffic, rather than querying a database for every update. A durable, replicated log-based system (Kafka, or a similar technology) is preferred over a simple in-memory pub-sub precisely because it retains recent history, allowing late-joining or reconnecting gateways to catch up rather than permanently missing messages published while they were briefly disconnected.

14.2 The message store as a secondary, asynchronous path

A separate, more traditional durable store (commonly a distributed wide-column or document store optimized for high write throughput) persists the full, unsampled history of chat and reaction events asynchronously, off the real-time hot path, supporting moderation review, compliance retention requirements, and post-event analytics — none of which need to affect the latency of the live experience itself.

14.3 Caching the session/room registry

The room registry, tracking which gateway instances currently hold connections for which event, is a natural candidate for a fast, in-memory distributed cache (such as Redis), since it’s read and written extremely frequently (on every connect/disconnect) but doesn’t need the stronger durability guarantees of the primary message path — losing a small window of registry updates during a rare failure is a recoverable inconvenience, not a correctness-critical event.

14.4 Load balancing new connections

New connection requests are distributed across the gateway fleet using a load-aware algorithm (such as least-connections, favoring the gateway instance currently holding the fewest active connections) rather than simple round robin, since gateway instances can end up holding meaningfully different connection counts over time as viewers connect and disconnect at different rates.

14.5 Partition count and topic design

Choosing how many partitions to allocate to a given live event’s pub-sub topic is a genuine design decision with real trade-offs, not a value that can simply be set arbitrarily high. Too few partitions concentrate too much of the event’s total message volume onto too few broker resources, creating a bottleneck regardless of how much overall cluster capacity exists. Too many partitions, on the other hand, add coordination and metadata overhead, and can actually hurt latency for a smaller event that doesn’t need that much parallelism. Because major events are known well ahead of time, partition count is typically chosen deliberately based on the expected audience size and message rate for that specific event, rather than using one fixed default for every live stream on the platform regardless of its expected scale.

14.6 Retention policy for the real-time log

The message bus does not need to retain every event’s data forever — its job is to support real-time fan-out and short-term reconnect catch-up, not long-term storage, which is the durable message store’s responsibility instead. A retention policy of a few minutes to a few hours on the pub-sub topic is typically sufficient to cover realistic reconnection scenarios, after which older data can be safely discarded from the bus itself, keeping its storage footprint and operational overhead proportional to recent activity rather than growing without bound across the platform’s entire history of live events.

How clients connect and communicate, and how the backend is decomposed into independently-scalable services with clear contracts between them.

15.1 Client connection protocol

// Client -> Server: send a chat message
{
  "type": "chat.send",
  "eventId": "evt_9f21",
  "text": "what a save!!",
  "clientTimestamp": 1719000123456
}

// Client -> Server: send a reaction tap
{
  "type": "reaction.tap",
  "eventId": "evt_9f21",
  "reactionType": "heart"
}

// Server -> Client: sampled chat message
{
  "type": "chat.receive",
  "userDisplayName": "Priya K.",
  "text": "what a save!!",
  "serverTimestamp": 1719000123510
}

// Server -> Client: batched reaction update
{
  "type": "reaction.batch",
  "reactionType": "heart",
  "windowCount": 38204,
  "totalCount": 4210983
}

A lightweight, well-defined message envelope keeps client-side parsing simple and cheap, which matters when a client may be processing many incoming messages per second during a busy moment.

15.2 Internal service boundaries

The system is decomposed so each piece can scale and evolve independently: the Connection Gateway layer owns only connection lifecycle and local fan-out; the Ingest Service owns validation and rate limiting; the Moderation Pipeline owns content policy decisions; the Reaction Aggregator owns windowed counting; and the Room Registry owns presence and routing metadata. None of these services share a database directly — they communicate exclusively through the message bus and well-defined internal APIs, so any one of them can be scaled, redeployed, or replaced without requiring coordinated changes to the others.

15.3 Internal gRPC between ingest and moderation

service ModerationService {
  rpc ScreenMessage(ScreenRequest) returns (ScreenResponse);
}

message ScreenRequest {
  string event_id = 1;
  string user_id = 2;
  string text = 3;
}

message ScreenResponse {
  enum Decision { ALLOW = 0; BLOCK = 1; FLAG_FOR_REVIEW = 2; }
  Decision decision = 1;
  double confidence_score = 2;
}

15.4 Client-facing error handling for throttled messages

When a client’s own message is rejected due to rate limiting or moderation, the API contract matters for a good user experience: the server responds with an explicit, structured error (distinguishing, for example, between “you’re sending messages too quickly” and “this message was not allowed”) rather than silently dropping the message with no client-visible feedback at all. A well-designed client uses this distinction to show an appropriate, specific message to the user — a brief cooldown indicator for rate limiting versus a content-policy notice for a moderation rejection — rather than leaving the user confused about why their message never appeared in the chat view, which otherwise reads to the user as a bug rather than an intentional platform decision.

Recurring solutions that show up across nearly every large-scale real-time fan-out system, and the mistakes that show up nearly as often when teams try to skip them.

16.1 Anti-patterns to avoid

16.2 Observer pattern generalized across the pipeline

The relationship between the message bus and the many gateway instances subscribed to it is a distributed instantiation of the classic Observer pattern: the bus, as the subject, publishes an event once without needing any awareness of how many gateways are subscribed or what each one intends to do with the message. Each gateway, as an independent observer, reacts in whatever way is appropriate to its own currently-connected viewers. This decoupling is what allows the platform to add entirely new categories of consumers later — for example, a live analytics dashboard for content creators showing real-time engagement trends — without requiring any change whatsoever to the publishing side of the pipeline, since new observers can simply subscribe to the existing stream independently.

16.3 Debounce and coalesce patterns on the client

The same batching philosophy applied server-side in the aggregator is often mirrored on the client for further efficiency and smoothness. Rather than triggering a full UI re-render for every single incoming chat message or reaction update the instant it arrives, well-built clients debounce or coalesce a short burst of near-simultaneous incoming updates into a single rendering pass, typically aligned to the device’s natural rendering frame rate. This avoids wasted rendering work and visual jank on the viewer’s device during a high-traffic moment, complementing the server-side aggregation described earlier rather than duplicating it — server-side aggregation reduces network message volume, while client-side coalescing reduces local rendering overhead for whatever volume of messages does arrive.

Lessons that separate a system that survives its first major event from one that doesn’t. Each of these has been paid for, somewhere, in a real post-mortem.

17.1 Best practices

• Design for correlated spikes, not average load. Live-event traffic is defined by its bursts; a system that only handles smooth growth will fail exactly when it matters most.
• Pre-provision for known major events rather than relying solely on reactive auto-scaling, which may not react fast enough to a near-vertical traffic ramp.
• Make aggregation window size and sampling rate configurable and adjustable live, so the system can be manually or automatically tuned in the middle of an event without a redeploy.
• Keep moderation’s hot-path check fast and push heavier analysis asynchronous, so legitimate messages are never held up by the system’s slowest possible check.
• Freeze risky deployments around scheduled high-stakes events, accepting slower iteration speed during those windows in exchange for stability when it matters most.
• Rehearse major events with realistic, bursty synthetic load, including simulated failures, before they happen for real.
• Preserve the full, unsampled data server-side even while sampling what’s shown to any individual client, so moderation and analytics never lose data that the real-time path intentionally didn’t deliver.

17.2 Common mistakes

• Assuming steady-state load testing is sufficient. Smooth ramp tests hide exactly the failure modes that sudden correlated spikes produce.
• Coupling reaction delivery directly to raw tap volume, recreating the fan-out explosion aggregation is specifically meant to prevent.
• Under-investing in reconnection logic, leaving viewers stuck after a brief network hiccup or gateway failover instead of recovering automatically within seconds.
• Treating moderation as a purely synchronous gate, adding latency to every message rather than reserving synchronous checks for only the fastest, cheapest filters.
• Ignoring the cost of connection churn at event start and end, when a huge fraction of the total audience connects or disconnects within a short window.

17.3 A pragmatic rollout sequence

Teams that have successfully shipped systems like this tend to build them up in a deliberate sequence rather than attempting the fully-scaled version on day one. Start with a correct, simple implementation validated against a small live audience, with straightforward, unaggregated delivery, to confirm the core connection-handling, message-routing, and moderation logic work correctly before introducing any scale-oriented complexity. Add windowed reaction aggregation and chat sampling once real traffic data reveals the actual message and reaction rates the platform’s typical and largest events produce, tuning window sizes and sampling rates against that real data rather than guessing at reasonable defaults in advance. Layer in sharding of large rooms across multiple pub-sub partitions, pre-provisioned capacity for known major events, and full chaos-tested failover only once the platform has outgrown what a simpler, single-partition design can comfortably support. Throughout, invest early and continuously in observability — end-to-end latency tracing, connection and throughput dashboards — since a system this bursty and time-boxed is unusually hard to safely operate, tune, or improve without strong, real-time visibility into exactly what it’s doing at any given moment.

How major platforms have actually engineered this capability. Each has slightly different tuning, but the underlying shape is remarkably consistent.

18.1 What differs across these platforms, and why

While the core architecture converges on similar patterns, the specific tuning differs meaningfully based on each platform’s typical content shape. A platform dominated by long-running, many-hours streams (typical of gaming live streams) tends to optimize its chat infrastructure for sustained, steady-state high throughput over long durations, with capacity planning built around the statistics of an ongoing stream’s typical audience rather than a single, sharply time-boxed spike. A platform built around short, appointment-viewing events — a single live sports match, a one-time keynote — instead optimizes heavily for the specific challenge of an enormous, near-simultaneous connection ramp at a known start time, followed by an equally sharp drop-off once the event concludes, which is a different capacity-planning problem even though the steady-state message-handling machinery looks broadly similar in both cases. Recognizing which of these two traffic shapes a given product actually needs to support — sustained versus sharply time-boxed — is one of the first and most consequential design decisions when adapting this architecture to a specific platform’s real usage pattern.

The questions that come up again and again in interviews, design reviews, and Slack threads five minutes before a launch. Each answer traces back to a decision made earlier in this tutorial.

Zoom back out. If you carry just a handful of ideas from this tutorial into an interview or a design review, make them these.

What makes this problem such a rich one to study is how directly its constraints shape its architecture: an audience that can be ten million strong, a latency budget measured in a few hundred milliseconds, and traffic that spikes not gradually but instantly and in near-perfect unison across the entire audience the moment something exciting happens on screen. Every major technique covered here — aggregation, sampling, sharding, pre-provisioning, graceful degradation — exists specifically because of that combination of constraints, and each would look different, or be unnecessary altogether, for a system without one of those three properties. Understanding not just what these techniques are, but precisely which constraint each one is responding to, is what separates a memorized checklist of scaling tricks from a genuine, transferable understanding of real-time, massive-fan-out system design.

20.1 Where to go from here

If you are preparing this design for an interview, practice explaining not just the final architecture but the arithmetic behind each choice: why aggregation windows are on the order of hundreds of milliseconds, why chat sampling is uniform-random-with-optional-weights rather than first-in-first-out, why the message bus is durable and log-based rather than in-memory, and why moderation is split into a fast synchronous check and a deeper asynchronous one. Being able to derive each of those decisions from constraints on the fly, rather than reciting them, is what turns a memorized design into a defensible one under interviewer follow-up questions. If you are building this system for real, resist the temptation to skip straight to the fully-sharded, multi-region, aggressively-sampled version on day one — a correct, observable single-region system with basic rate limiting and simple aggregation will teach you far more about your actual traffic than any amount of premature scaling architecture ever could, and you will earn the right to add each subsequent layer of complexity precisely when real production data proves you need it.