Real-time chat looks deceptively simple from the outside. Two people type words and see them appear on each other’s screens almost instantly. But underneath that simplicity sits one of the hardest problems in distributed systems: keeping millions of long-lived, stateful connections open at once, routing every message to the right device within a few hundred milliseconds, and never losing a message even when servers crash, networks partition, or an entire data center goes dark.

Chat systems have gone through several generations. The earliest widely used protocol was IRC (Internet Relay Chat) in the late 1980s, a simple text protocol built for channels and rooms rather than private, persistent conversations. In the 2000s, XMPP (Extensible Messaging and Presence Protocol) became the backbone of many chat products, including early versions of Google Talk and Facebook Chat, because it standardized presence (who is online) and message delivery over XML streams. XMPP worked, but XML parsing at scale is expensive, and the protocol was not designed with mobile networks, battery life, or hundreds of millions of simultaneous connections in mind.

The smartphone era changed the requirements completely. WhatsApp, launched in 2009, proved that a small engineering team could serve hundreds of millions of users using Erlang and a heavily customized version of the ejabberd XMPP server, prioritizing connection efficiency over feature richness. Around the same time, Facebook Messenger, WeChat, Telegram, Signal, Slack, and later Discord each took different architectural paths, but they converged on a shared set of building blocks: persistent bidirectional connections (WebSockets or custom binary protocols), a message broker for fan-out, a highly available storage layer tuned for write-heavy, append-only workloads, and push notification integration for offline delivery.

This tutorial designs a chat system for an extreme but increasingly realistic scale target: 500 million concurrent users, each capable of sending and receiving messages with sub-second end-to-end latency. This is roughly the scale of WhatsApp’s or WeChat’s daily active connection load. We will walk through the requirements, the architecture, the internal mechanics, the trade-offs, and the operational concerns a senior engineer or system design interviewer expects you to reason about.

Before drawing a single box on a whiteboard, a good system design answer starts by nailing down what the system must do and how well it must do it. Interviewers pay close attention to whether a candidate asks these questions before diving into architecture.

2.1 Functional Requirements

• Users can send one-to-one messages and group messages (up to a few hundred members per group).
• Messages must be delivered in real time to online recipients, and stored for delivery when the recipient comes back online.
• The system must show delivery and read receipts (sent, delivered, read).
• The system must show typing indicators and online/offline presence.
• Users can send text, images, short videos, voice notes, and files.
• Message history must be retrievable, with pagination, when a user opens a conversation.
• Messages must arrive in the correct order within a conversation.

2.2 Non-Functional Requirements

• Scale: 500 million concurrent connections, with a comparable or larger number of daily active users, since not everyone is online at the same instant.
• Latency: Sub-second (target under 300 ms at p99) end-to-end delivery for online recipients within the same region, and under 1 second for cross-region delivery.
• Availability: 99.99% or higher; a chat outage is highly visible and erodes trust quickly.
• Durability: A message that the server acknowledged must never be lost, even during node or region failure.
• Consistency: Eventual consistency is acceptable for presence and read receipts, but message ordering within a single conversation must be strongly guaranteed.
• Security: End-to-end encryption for one-to-one and group chats, protection against spam and abuse, and strict authentication.

2.3 Back-of-the-Envelope Estimation

Rough capacity planning grounds every later architectural decision. Assume 500 million concurrent connections and that an average user sends one message roughly every 30 seconds while actively chatting, with maybe 10% of connected users actively composing at any given moment.

These numbers tell us immediately that a single database, a single message queue partition, or a single load balancer cannot handle this load. Everything from the edge to the storage layer must be horizontally sharded, and connection state itself — 500 million open sockets — becomes a first-class scaling problem, not just message throughput.

2.4 Choosing a Consistency Model Up Front

A subtle but critical requirements decision is deciding, piece of data by piece of data, how much consistency it actually needs. Not every field in a chat system needs the same guarantee, and treating them all identically is one of the most common design mistakes at this scale.

Deciding this early shapes every subsequent architectural choice: it is precisely why the design in this tutorial uses a durable, quorum-replicated store for messages but a fast, best-effort, in-memory store for presence and typing indicators.

At 500 million concurrent users, the system must be decomposed into independently scalable services, each responsible for one concern. Below is the high-level architecture.

3.1 WebSocket Gateway Fleet

This is the entry point for every connected client. Each gateway node holds tens to hundreds of thousands of persistent WebSocket connections. Its jobs are narrow and fast: authenticate the connection once, keep it alive with heartbeats, deserialize incoming frames, forward them to the Message Service, and push outbound messages back down the socket the moment they arrive. Gateways are intentionally kept stateless with respect to message content — the only state they hold is “which user is connected to which gateway node,” which is registered in the Presence Service.

3.2 Presence Service

Presence answers one question extremely fast: for a given user ID, which gateway node (and therefore which server and region) currently holds their live connection, if any. This is backed by an in-memory store like Redis, sharded by user ID, with a short time-to-live so that a crashed gateway’s stale entries expire quickly. Presence is also what powers “online/offline” and “last seen” features exposed to users.

3.3 Message Service

The Message Service owns the business logic of sending a message: validating the sender, assigning a globally unique, time-ordered message ID, persisting the message, and handing it off to the fan-out pipeline that decides who needs to receive it and how. It is deliberately decoupled from the gateway layer so that message logic can be scaled, deployed, and reasoned about independently of raw connection handling.

3.4 Message Queue (Kafka or similar)

A distributed log-based queue decouples message ingestion from message delivery. Once the Message Service durably appends a message to Kafka, it can acknowledge the sender immediately, while downstream consumers handle fan-out, persistence, search indexing, and push notifications independently and in parallel. This also gives natural replay capability if a downstream consumer fails.

3.5 Notification Service

For recipients who are not currently connected (no live WebSocket), the Notification Service pushes a message through Apple Push Notification Service (APNs) or Firebase Cloud Messaging (FCM) so the OS can wake the app or show a system notification, even though the chat app itself is not running.

3.6 Message Store

A wide-column, horizontally partitioned database such as Cassandra or DynamoDB stores the durable message history, partitioned by conversation ID, and sorted by a time-ordered message ID within each partition. This gives cheap, fast retrieval of “the last N messages in this conversation” without scanning unrelated data.

3.7 Media Service and CDN

Images, voice notes, and videos never travel through the WebSocket message path itself. The client uploads media directly to object storage through the Media Service, receives back a reference URL, and sends that lightweight reference as the “message.” Recipients fetch the actual bytes from a CDN edge node close to them.

3.8 Snowflake-style ID Generator

Every message needs a unique ID that is also roughly time-sortable, so that message history returns in correct order without an expensive secondary sort. A Twitter Snowflake-style generator — combining a timestamp, a machine/shard ID, and a per-millisecond sequence counter — produces such IDs without needing a centralized counter that would become a bottleneck at millions of messages per second.

4.1 Connection Handling at Massive Scale

A single modern server, tuned correctly (increased file descriptor limits, an efficient event loop such as epoll on Linux, and a lightweight per-connection memory footprint), can hold somewhere between 500,000 and a few million idle WebSocket connections. To reach 500 million concurrent connections, the gateway tier needs on the order of a few thousand such nodes, spread across many regions, behind connection-aware load balancers.

Each gateway node runs an event-driven architecture rather than a thread-per-connection model. Thread-per-connection does not scale past a few thousand sockets per machine because of context-switching overhead and per-thread memory cost; an event loop can multiplex hundreds of thousands of sockets on a handful of OS threads by reacting to readiness events rather than blocking.

4.2 Heartbeats and Connection Liveness

Mobile networks silently drop idle connections, and NAT devices time out mappings after a few minutes of inactivity. The gateway and client exchange small heartbeat (ping/pong) frames every 15 to 30 seconds. If a heartbeat is missed twice in a row, the gateway proactively closes the connection and updates Presence, so the system does not keep routing messages to a dead socket.

4.3 Fan-out: Push Model vs Pull Model vs Hybrid

When a message is sent, especially to a group, the system must decide how to get it to every recipient’s device.

• Fan-out on write (push): The moment a message arrives, the server immediately writes a copy into every recipient’s inbox/queue and pushes it down any open WebSocket. This gives the lowest read latency because recipients simply receive the message; it costs more write amplification for very large groups.
• Fan-out on read (pull): The message is stored once, and each recipient’s client pulls new messages when it reconnects or polls. This is cheap to write but adds latency and complexity for the “who read what” tracking, so it is a poor fit for real-time expectations.
• Hybrid approach: Push immediately to recipients who are online (a cheap, fast, in-memory operation through the Presence-routed WebSocket), and fall back to a durable per-user inbox plus a push notification for recipients who are offline. Very large groups (broadcast channels with millions of members) use a separate “fan-out on read” path so a single message does not trigger millions of synchronous writes.

4.4 Message Ordering and Idempotency

Within a single conversation, messages must appear in the order they were sent, even if the underlying infrastructure delivers them out of order due to retries or network jitter. This is solved by attaching the Snowflake-style, monotonically increasing message ID at creation time and having clients render messages sorted by that ID rather than by arrival order. Because networks can duplicate a message during a retry, every message also carries a client-generated idempotency key; the Message Service deduplicates on that key before persisting, so a retried “send” never creates two visible messages.

4.5 Sharding and Partitioning Strategy

Both connection state and message data are partitioned. Presence and Gateway routing are sharded by user ID using consistent hashing, so that a given user’s connection is always looked up on a predictable shard, and adding or removing gateway nodes only reshuffles a small fraction of the keyspace instead of all of it. Message storage is partitioned by conversation ID, so that all messages belonging to one conversation live together and can be paginated efficiently, while different conversations spread evenly across the storage cluster.

4.6 Consistent Hashing for Connection Routing

A naive approach to routing a user to a gateway node — such as user_id modulo number_of_nodes — falls apart the moment a node is added or removed, because the modulo changes for almost every user simultaneously, forcing a massive, disruptive reshuffle of live connections. Consistent hashing solves this by mapping both nodes and keys (user IDs) onto a logical ring using a hash function; a user is routed to the next node clockwise on the ring from their hashed position. When a node joins or leaves, only the small slice of keyspace immediately adjacent to it on the ring is affected, leaving the vast majority of users’ routing untouched. Production systems typically add virtual nodes (many points on the ring per physical node) to keep the load distribution even, since a small number of raw hash points can otherwise land unevenly.

4.7 Rate Limiting Algorithms

Protecting the system from abusive or misbehaving clients requires rate limiting at the connection and account level. A token bucket algorithm is the most common choice: each user (or connection) has a bucket that refills at a steady rate and drains by one token per message sent; if the bucket is empty, further messages are rejected or delayed until it refills. This allows normal bursty human typing behavior while still capping sustained abuse, in contrast to a simpler fixed-window counter, which allows a burst of double the intended rate right at a window boundary. Rate limit state itself lives in the same fast, in-memory cache tier as presence, since it must be checked on every single message with minimal added latency.

4.8 Handling Group Typing Indicators and Presence Broadcasts Efficiently

A group of a few hundred members generating frequent typing indicators can itself become a small-scale fan-out problem. These signals are deliberately treated as lossy and low-priority: they are coalesced (only the latest “is typing” state per user is kept), throttled to at most one update every few seconds per user, and dropped entirely under backpressure rather than being queued durably like message content, since losing one typing indicator has zero user-visible impact.

Walking through the full lifecycle of a single message clarifies how all the components cooperate.

5.1 Send Path

The client opens (or reuses) a WebSocket connection, authenticated once at connect time with a short-lived token so subsequent messages do not repeat a full auth handshake. When the user sends a message, it travels to the nearest gateway, which forwards it to the Message Service. The Message Service does the minimum work necessary to acknowledge quickly: assign an ID, append to the durable queue, and return an acknowledgment. Everything else — persistence, fan-out, indexing — happens asynchronously, off the critical path of the sender’s perceived latency.

5.2 Delivery Path

A consumer of the message queue looks up the recipient’s presence. If the recipient has a live connection, the message is pushed immediately through that specific gateway node — this is the sub-second path. If the recipient is offline, the message waits durably in their per-user inbox in the message store, and a push notification is fired so the OS can alert the user even with the app closed.

5.3 Read Path (History and Pagination)

When a user opens a conversation, the client requests the most recent page of messages by conversation ID, sorted by the time-ordered message ID, typically the last 30 to 50 messages. Scrolling further back issues another paginated query using the oldest message ID seen so far as a cursor. Because storage is partitioned by conversation ID, this is a fast, targeted read rather than a broad scan.

5.4 Acknowledgment and Read Receipts

Three states are tracked per message per recipient: sent (accepted by the server), delivered (reached the recipient’s device), and read (the recipient opened the conversation). Each state transition is itself a small, asynchronous event flowing back through the same pipeline, updated in the message store and pushed back to the sender’s device if they are online.

The overarching theme across every trade-off in this system is the classic tension in distributed systems between strong consistency, high availability, and tolerance of network partitions — the CAP theorem. A chat system almost always chooses availability and partition tolerance over strict consistency, because a user would rather see a message arrive slightly out of order or with a brief presence delay than have the entire app become unusable during a network hiccup.

7.1 Horizontal Scaling of the Gateway Tier

Because each gateway node’s capacity is bounded by memory and file descriptors rather than CPU, scaling out is simply a matter of adding more gateway nodes and letting consistent hashing redistribute new connections. Auto-scaling policies typically watch open connection count and CPU utilization per node, adding capacity well before nodes approach their connection ceiling, since rebalancing hundreds of thousands of live connections mid-flight is disruptive.

7.2 Load Balancing WebSocket Traffic

Standard round-robin HTTP load balancing does not work well for long-lived connections, because a naive balancer would keep sending new connections to the same “first available” node until it fills up, then dump the next thousand onto whichever node is next, creating uneven load. Production systems instead use connection-aware balancing: an L4 load balancer with least-connections or consistent-hashing algorithms, combined with a control plane that tracks per-node connection counts and steers new connections toward underloaded nodes in real time.

7.3 Caching Strategy

Redis (or a similar in-memory store) caches three hot categories of data: presence/routing information (which gateway a user is on), the most recent messages of very active conversations to avoid hitting the message store for every scroll-to-top action, and rate-limiting counters. Cache entries for presence carry short TTLs so stale routing information self-heals quickly after a node failure.

7.4 Reducing Tail Latency

At this scale, average latency is a misleading metric; p99 and p99.9 latency determine user-perceived quality, because with hundreds of millions of messages a “rare” slow path still affects millions of people every day. Techniques used to control tail latency include request hedging (issuing a duplicate read to a second replica if the first hasn’t responded within a few milliseconds), keeping connection pools warm to downstream services, and isolating noisy tenants (very large groups or bot accounts) onto separate infrastructure so they cannot degrade latency for everyone else.

7.5 Concurrency and Backpressure

Every layer must apply backpressure rather than accepting unbounded work. Gateways cap the number of in-flight unacknowledged messages per connection; the Message Service applies per-user rate limits; and Kafka consumers use bounded prefetch so a slow downstream database cannot cause unbounded memory growth upstream. Without backpressure, a slowdown in one component cascades into an out-of-memory crash in another — a very common cause of real-world chat outages.

7.6 CAP Theorem in Practice

The CAP theorem states that a distributed system experiencing a network partition must choose between consistency and availability; it cannot guarantee both. For the message store, this design leans toward availability with tunable, quorum-based consistency: a write succeeds once a majority of replicas acknowledge it, so the system stays available and durable even if a minority of replicas are unreachable, at the cost of a small window where a reader hitting a lagging replica might see slightly stale data. For presence and typing indicators, the system leans even further toward availability, accepting eventual consistency outright, because perfect real-time accuracy of “who is online” is not worth the coordination cost it would require. This is a concrete, worked example of why CAP is not an abstract theorem in this design — it directly explains why two different subsystems within the same product make opposite trade-offs.

7.7 Network Protocol Choices: TCP, TLS, and QUIC

WebSockets run over TCP, which guarantees ordered, reliable delivery at the transport layer, but that reliability comes with a cost on lossy mobile networks: a single lost packet stalls the entire TCP connection until it is retransmitted, a problem known as head-of-line blocking. Some modern chat and real-time systems are migrating parts of their transport to QUIC (the protocol underlying HTTP/3), which runs over UDP and multiplexes independent streams so that a lost packet on one stream does not stall others, and which also collapses the TLS and transport handshake into fewer round trips, shaving meaningful latency off connection setup — a real benefit when hundreds of millions of mobile clients are constantly reconnecting as they move between WiFi and cellular networks.

8.1 Multi-Region, Active-Active Deployment

The system runs in multiple geographic regions simultaneously, each capable of serving traffic independently. Users are routed to their nearest healthy region by GeoDNS or an anycast load balancer, which both minimizes latency and provides natural failure isolation: if one region degrades, traffic shifts to the next-nearest region.

8.2 Replication and Consensus

The message store replicates every write to multiple nodes, typically three, spread across availability zones, using a quorum-based approach: a write is considered successful once it has been acknowledged by a majority of replicas (for example, 2 of 3), and a read likewise consults a majority. This gives strong durability guarantees without requiring every single replica to be available, and it tolerates the loss of one replica without any data loss or downtime. For any component that needs strict agreement on a single value across nodes — such as leader election for a partition — a consensus protocol like Raft is used, since it is easier to reason about than older approaches like Paxos while providing equivalent safety guarantees.

8.3 Failure Recovery

Because the message queue is the durability boundary, any consumer (a fan-out worker, a persistence worker, a notification worker) can crash and restart without losing data: it simply resumes reading the queue from its last committed offset. Gateway node failures are handled by the heartbeat mechanism — a client whose gateway disappears simply reconnects, gets routed to a healthy node by the load balancer, and Presence is updated within seconds.

8.4 Disaster Recovery

Beyond single-node or single-zone failures, the system must survive an entire region going offline. Cross-region asynchronous replication of the message store ensures that even if one region is lost entirely, a very recent (typically seconds-old) copy of all conversations exists elsewhere. Regular backup snapshots and periodic restore drills validate that recovery actually works, not just that it exists on paper.

8.5 Graceful Degradation

Not every feature needs the same availability guarantee. If the search-indexing pipeline or the “last seen” presence feature is temporarily degraded, the core function — sending and receiving messages — must keep working. Designing services so that non-critical features fail independently, without taking down message delivery, is a deliberate architectural choice, not an accident.

8.6 Consensus in Depth: Why Raft

Certain narrow pieces of this system need every node to agree on a single truth even while machines fail and messages get delayed — for example, which replica is currently the write-leader for a given storage partition. Raft solves this by electing a single leader through randomized election timeouts (so two nodes rarely trigger an election simultaneously) and by requiring the leader to replicate every change to a majority of followers before considering it committed. If the leader crashes, the remaining nodes detect the missing heartbeats, hold a new election, and a new leader takes over, having been guaranteed by the majority-write rule to already hold every previously committed change. This is preferred over the older Paxos protocol for internal tooling specifically because Raft was designed for understandability: its leader-based model maps clearly onto “one node is in charge until it fails,” which is far easier for an engineering team to reason about, implement correctly, and debug during an incident than Paxos’s more decentralized formulation.

8.7 Replica Placement and Failure Domains

Replicas are not just spread across different machines; they are deliberately spread across different failure domains — separate racks, separate power circuits, and separate availability zones — so that a single correlated failure (a rack losing power, a network switch failing) cannot take out a majority of replicas for the same partition at once. This is what allows the quorum-based durability guarantee from section 8.2 to hold up against real-world, correlated infrastructure failures rather than only theoretical independent ones.

9.1 End-to-End Encryption

Modern chat systems implement end-to-end encryption using protocols descended from the Signal Protocol, combining the Double Ratchet Algorithm with X3DH (Extended Triple Diffie-Hellman) key agreement. Each message is encrypted on the sender’s device with a key that only the recipient’s device holds; the server transports and stores only ciphertext, and even a full server compromise does not expose message content. This has a direct architectural consequence discussed further in section 6: the server cannot search message content, run spam classifiers on plaintext, or generate content previews server-side.

9.2 Transport Security

All connections, from the initial WebSocket handshake onward, are secured with TLS, protecting metadata and connection integrity even though the payload is separately end-to-end encrypted. Certificate pinning on mobile clients further protects against man-in-the-middle attacks on hostile networks.

9.3 Authentication and Session Management

Users authenticate once through an identity service (often backed by phone number verification or an OAuth-style flow), receiving a short-lived access token and a longer-lived refresh token. The WebSocket connection is authenticated at handshake time using the access token; tokens are rotated frequently, and a compromised token has a short blast radius because of its short lifetime.

9.4 Abuse Prevention and Rate Limiting

Even without reading message content, the system can detect abuse from behavioral signals: an account sending messages to thousands of new recipients per minute, unusually high message velocity, or many recipients blocking the same sender in a short window. Rate limiting is enforced per connection and per account at the gateway and Message Service layers, and suspicious accounts can be throttled or challenged with additional verification without needing to inspect encrypted content.

9.5 Data Protection at Rest

Even though message content is end-to-end encrypted, metadata (who messaged whom, when, group membership) still requires protection at rest through disk-level and database-level encryption, strict access controls, and audit logging on any administrative access to the data stores.

10.1 Golden Signals for a Chat System

10.2 Distributed Tracing

A single message touches the gateway, the Message Service, the queue, a fan-out worker, and a second gateway before reaching its recipient. Distributed tracing attaches a unique trace ID to each message at ingestion, propagated through every hop, so engineers can pinpoint exactly which stage introduced latency for a slow message rather than guessing across five separate service logs.

10.3 Structured Logging

Logs are emitted in a structured (typically JSON) format with consistent fields — user ID (hashed for privacy where appropriate), message ID, conversation ID, region, and latency — feeding into a centralized log aggregation pipeline. This makes it possible to run targeted queries during an incident (“show me all failed deliveries in region eu-west in the last five minutes”) instead of grepping raw text across thousands of machines.

10.4 Alerting

Alerts are tied to symptom-based thresholds rather than raw resource metrics alone: a spike in p99 delivery latency, a drop in delivery success rate, or rising queue lag pages an on-call engineer, because these directly reflect user-visible degradation, whereas a CPU spike alone might be entirely harmless.

11.1 Containerization and Orchestration

Every service — gateways, Message Service, Presence Service, Notification Service — runs as containerized workloads managed by an orchestrator such as Kubernetes, which handles scheduling, health checking, and auto-restarting failed instances. Horizontal Pod Autoscaling adjusts the number of running instances based on connection count and CPU/memory utilization.

11.2 Multi-Region, Multi-Cloud Considerations

Regions are deployed as near-identical stamps of the full architecture, so a region can be added or removed as a unit. Some organizations additionally spread across multiple cloud providers to reduce the blast radius of a single provider’s outage, though this adds meaningful operational complexity and is usually reserved for organizations at the very largest scale.

11.3 Progressive Delivery

Given the blast radius of a bug in the Message Service or Gateway fleet, deployments use canary releases — rolling a new version out to a small percentage of traffic first, watching the golden signals from section 10, and only proceeding to full rollout if metrics stay healthy. Blue-green deployment patterns for stateless services allow instant rollback by simply shifting traffic back to the previous stable version.

11.4 Cost Optimization

At this scale, infrastructure cost is a first-class design constraint. Idle connection handling favors memory-efficient languages and runtimes (Erlang/Elixir, Go, and Rust are common choices for gateway layers specifically because of their low per-connection memory overhead and efficient concurrency models). Media storage uses tiered storage classes, moving older, rarely accessed media to cheaper cold storage automatically.

12.1 Choosing the Message Store

The message store is the single highest-throughput, highest-volume component in the system, so its access pattern drives the choice of database. The dominant pattern is “append a new message to a conversation” and “read the most recent N messages of a conversation,” both of which map cleanly onto a wide-column store like Cassandra or a managed equivalent like DynamoDB, where the conversation ID is the partition key and the time-ordered message ID is the clustering/sort key. This avoids the join-heavy, vertically-scaled patterns of a traditional relational database, which struggles to sustain millions of writes per second across a single machine.

12.2 Caching Layers

Redis clusters sit in front of the message store for the hottest data: active conversation tails, presence/session routing, and rate-limit counters. Cache invalidation follows a write-through pattern for presence (updated the instant a connection opens or closes) and a time-boxed TTL pattern for message tails, since a slightly stale cached view of an inactive conversation is harmless.

12.3 Load Balancer Tiers

Traffic passes through at least two load balancing tiers: a global tier (GeoDNS or anycast) that routes users to their nearest healthy region, and a regional tier (L4/L7 load balancers) that distributes connections across gateway nodes within that region using connection-aware algorithms rather than plain round robin, as discussed in section 7.2.

12.4 Search Indexing

Full-text search over message history (where end-to-end encryption allows it, typically via on-device indexing rather than server-side, or for non-E2E channels like public communities) uses a separate search index such as Elasticsearch, updated asynchronously off the main write path so that search indexing load never affects message delivery latency.

12.5 Partitioning Strategy in Depth

Partitioning by conversation ID works extremely well for the overwhelming majority of one-to-one and small-group conversations, but it introduces a specific risk: a single extraordinarily active conversation — a huge public group or a viral broadcast channel — can concentrate far more traffic onto one partition than an average conversation, creating a hot spot even though the overall cluster has plenty of spare capacity elsewhere. Two complementary techniques address this. First, salting: a hot conversation’s partition key is combined with a small random or rotating suffix, splitting its data (and therefore its write load) across several underlying partitions instead of one, at the cost of slightly more complex reads that must merge results from each sub-partition. Second, dedicated capacity: conversations that are detected as unusually hot are automatically migrated onto isolated infrastructure sized specifically for them, so their load never competes with the millions of ordinary, low-traffic conversations sharing the general-purpose cluster.

12.6 Read Replicas and Locality

Because reads (opening a conversation, scrolling history) vastly outnumber writes for most conversations, read replicas are placed close to where users actually are, so a user in Mumbai reading an old conversation does not need a round trip to a primary replica sitting in a different continent. Writes still go through the quorum described in section 8.2, but subsequent reads of already-committed data can be served locally, keeping the read path fast without weakening the durability guarantee on the write path.

13.1 Protocol Choices

Three different protocols typically coexist, each suited to a different interaction pattern:

// Client to Server
{
  "type":              "message.send",
  "conversation_id":   "conv_9f4a…",
  "idempotency_key":   "cli_2026-08-11_a8f7…",
  "content_ciphertext":"AES-GCM base64 blob…",
  "content_type":      "text/plain",
  "client_ts_ms":      1755024251471
}

// Server to Client (ack)
{
  "type":              "message.ack",
  "message_id":        "1885212471471620096",   // Snowflake ID
  "server_ts_ms":      1755024251491,
  "status":            "SENT"
}

// Server to Recipient
{
  "type":              "message.delivery",
  "message_id":        "1885212471471620096",
  "conversation_id":   "conv_9f4a…",
  "sender_id":         "usr_1c02…",
  "content_ciphertext":"AES-GCM base64 blob…"
}

13.2 Service Boundaries

Each microservice owns a single, clearly bounded responsibility: the Gateway owns connections, Presence owns routing state, the Message Service owns send-time business logic, the Notification Service owns offline delivery, and the Media Service owns large binary content. This separation allows each to be scaled, deployed, and even rewritten independently — for example, migrating the gateway fleet from one language to another without touching message storage logic at all.

13.3 Backward Compatibility

With hundreds of millions of client app installations that update on their own schedule, the wire protocol must remain backward and forward compatible for a long time. This is typically achieved with a versioned, schema-based serialization format (such as Protocol Buffers) where new optional fields can be added without breaking older clients that do not understand them.

14.1 Patterns Used

14.2 Anti-Patterns to Avoid

15.1 Best Practices

• Acknowledge the sender as soon as the message is durably queued, not after full end-to-end delivery, to keep perceived latency low.
• Design idempotency into every write path from day one; retries are inevitable at this scale, and duplicate messages destroy user trust quickly.
• Treat p99.9 latency, not average latency, as the metric that defines user experience at hundreds of millions of users.
• Isolate noisy tenants (extremely large groups, bots, spam accounts) onto separate infrastructure paths so they cannot degrade the experience for everyone else.
• Run regular game-day failure drills — killing a region, a database node, or a queue partition in a controlled way — to verify recovery actually works before it is needed for real.

15.2 Common Mistakes

• Underestimating connection-holding cost and only capacity-planning for message throughput, then discovering the gateway fleet runs out of memory or file descriptors long before it runs out of CPU.
• Coupling the real-time delivery path to a slow, synchronous database write, which makes the entire send operation only as fast as the slowest storage write.
• Ignoring cross-region latency until launch day, then discovering that users on opposite sides of the world experience multi-second delays because every message round-trips through a single home region.
• Building presence as a strongly consistent, centrally coordinated system, which becomes a scaling bottleneck for a feature that does not need perfect accuracy.
• Not planning for thundering-herd reconnection storms after a regional failover, when millions of clients simultaneously attempt to reconnect to the newly healthy region at once.

16.7 Common Thread Across All of Them

Despite different languages and internal tooling, every large-scale chat system converges on the same core ideas: an event-driven connection layer decoupled from business logic, asynchronous fan-out through a durable log, storage partitioned by conversation, and aggressive isolation between the “fast path” (send and deliver) and everything else (search, analytics, media processing).

Pulling all of this together, a chat system built for 500 million concurrent users is not one hard problem but many moderate ones, layered carefully on top of each other: an efficient way to hold enormous numbers of idle connections, a fast and durable way to record every message exactly once, a routing layer that can find any user’s live connection in milliseconds, and an operational discipline that assumes failure will happen constantly and plans for it rather than hoping around it. None of the individual pieces covered here — event loops, consistent hashing, quorum writes, Raft-based leader election, token bucket rate limiting — are unique to chat applications; they show up across most large-scale distributed systems. What makes chat a particularly good system design exercise is that it forces all of these ideas to work together under a single, unforgiving constraint that users feel immediately and viscerally: a message either arrives in under a second, in the right order, exactly once, or the product feels broken. That constraint is what should guide every trade-off discussed throughout this tutorial, from the first line of the architecture diagram to the last item in the on-call runbook.