The earliest instant messaging clients, from IRC in the late 1980s to AOL Instant Messenger and MSN Messenger in the late 1990s, were built around a simple assumption that turned out to be wrong for the world we live in today: that a person is logged in from exactly one place at a time. If you opened the same account on a second computer, the first session was usually just kicked out, because nobody had designed for the idea of one person being simultaneously present on a phone, a laptop, a tablet, and a browser tab, all expecting to see the same conversation in the same state.

That assumption broke down as soon as smartphones made it normal to carry a messaging app in your pocket while also keeping it open on a work laptop. Today, a single WhatsApp, Telegram, Slack, or iMessage account routinely maintains active sessions on three, four, or more devices at once, and users expect something that sounds simple but is deceptively hard to build correctly: if they read a message on their phone during a commute, it should show as read on their laptop the moment they open it at their desk, without them lifting a finger. If a message shows as delivered on one device, it should not still show as merely sent on another. Users never see the distributed systems problem underneath this expectation; they only notice when it breaks — when a message they already read still shows an unread badge, or when a reply sent from one device does not appear on another for several minutes.

This tutorial designs the system that makes that seamless experience possible: a multi-device synchronization layer that treats message state, meaning whether a message has been delivered to a device and whether it has been read by the user, as a piece of shared, eventually consistent state that must be kept aligned across every device tied to one account, all while the underlying network connections to those devices are constantly going up and down as people walk in and out of Wi-Fi coverage, lock their phones, or close their laptops.

It is worth being precise about what “consistent” actually means in this context, because it is easy to reach for the wrong mental model. This is not the strict consistency of a bank balance, where every reader must see the exact same value at the exact same instant or the system is considered broken. It is closer to eventual consistency with a strong ordering guarantee: every device is allowed to be briefly behind, especially one that just reconnected after being offline for an hour, but once it catches up, it must arrive at exactly the same final state as every other device, having applied exactly the same sequence of changes in exactly the same order. Getting that distinction right — allowing temporary staleness while forbidding permanent divergence — is what separates a messaging platform that feels reliable from one that occasionally leaves users staring at a message that stubbornly refuses to show as read no matter how many times they reopen the app.

1.1 A Short History of the Problem

1.2 Functional Requirements

• Every device registered to an account must be able to send and receive messages independently.
• When a message is delivered to any one device, that delivery status must be reflected to the sender and, where relevant, to the user’s other devices.
• When a user reads a message on any one device, all of that user’s other devices must reflect the message as read, typically by clearing an unread badge or removing it from an unread list.
• A device that was offline, closed, or newly installed must be able to reconstruct the fully correct current state of every conversation when it reconnects, not just receive new events going forward.
• The system must support an arbitrary and growing number of simultaneously linked devices per account, not just two.
• Read receipts and delivery receipts sent by a recipient must reach the original sender’s devices, respecting the recipient’s privacy settings around whether read receipts are shared at all.

1.3 Non-Functional Requirements

• State changes should propagate to other online devices of the same account within a few hundred milliseconds to feel instantaneous to the user.
• The system must remain correct under network partitions, meaning a temporarily disconnected device must never end up with a permanently wrong or contradictory state once it reconnects.
• The system must be horizontally scalable to support hundreds of millions of accounts, each with multiple simultaneously connected devices — meaning potentially over a billion live connections.
• State storage must be durable; a server crash must never lose a user’s read or delivery state.
• The design must tolerate devices with very different connectivity characteristics, from a fast broadband-connected desktop client to a phone on a spotty cellular connection that sleeps its network radio to save battery.

1.4 Scale Estimation

Consider a platform with five hundred million daily active accounts, each averaging two and a half concurrently connected devices during peak hours. That is over a billion simultaneous persistent connections the system must maintain. If each account sends or reads an average of thirty messages a day, and each of those actions can trigger a state-change event that must fan out to every other connected device on that account, the system needs to handle tens of billions of state synchronization events daily, with fan-out multiplying that further by the average device count per account. The state itself — read and delivery markers per message per device — is small individually but astronomically large in aggregate, requiring a storage design built specifically for compact, fast, high-cardinality state rather than treating it as an afterthought bolted onto the message store.

The system separates three concerns that are easy to accidentally tangle together: sending and storing the messages themselves, tracking per-device state about those messages, and propagating state changes to every online device in near real time. Keeping these as distinct, cooperating components is what allows each to scale and evolve independently.

2.1 Connection Gateway Servers

Every device maintains a persistent connection, typically a WebSocket or a similar long-lived bidirectional channel, to a gateway server. Gateway servers are stateless with respect to message content but hold the live mapping of which device is connected to which server instance, which is essential for knowing where to push a real-time state update.

2.2 Presence Service

The presence service tracks which of an account’s devices are currently online, and on which gateway server each one is connected, functioning as the routing directory the fan-out dispatcher consults when it needs to reach a specific device right now rather than later through a push notification.

2.3 Messaging Service

Handles the core act of sending a message: accepting it from the sending device, persisting it durably, and initiating delivery to the recipient’s devices. This is intentionally kept separate from state tracking, because a message’s content is immutable once sent, while its read and delivery state changes continuously afterward.

2.4 State Sync Service

This is the heart of the system described in this tutorial. It owns the logic for recording that a message was delivered to or read on a specific device, and for determining what that implies for the account’s other devices and for the original sender.

2.5 Ordered Event Log per Account

Rather than pushing raw ad hoc updates to devices, every state change and every new message is appended to a durable, strictly ordered event log scoped to the account. This log is the single source of truth that any device — whether online right now or reconnecting after being offline for days — can replay from to reconstruct the exact correct current state.

2.6 Fan-out / Notification Dispatcher

Reads new entries from the event log and pushes them to every currently connected device belonging to that account through the gateway servers, and falls back to a platform push notification service for any device that is not currently connected, so state changes are never silently lost — only delayed until the device reconnects or is woken by a push.

Tracing what happens when a message is read on one device and must sync everywhere else makes the design concrete — and then the synchronization model itself explains why those steps are structured the way they are.

3.1 Step by Step: Reading a Message on One Device

1. Message delivery. A message is sent and the messaging service persists it, then the fan-out dispatcher pushes it to every currently online device on the recipient’s account and queues push notifications for offline devices.
2. Per-device delivery acknowledgment. Each recipient device, upon receiving the message over its live connection, sends back a delivery acknowledgment tagged with its own device identifier. The state sync service records this acknowledgment against that specific device.
3. Delivered status computation. The state sync service computes an aggregate delivered status for the message as a whole, following a rule such as delivered-to-account being true once at least one device has acknowledged receipt, and pushes that aggregate status back to the sender’s devices.
4. User reads the message. The user opens the conversation on one specific device, say their phone. That device sends a read event to the state sync service, tagged with the message identifier, the device identifier, and a timestamp.
5. Read state recorded per device and per account. The state sync service records that this specific device has read this message, and additionally updates an account-level read marker, since what other devices need to know is not merely that the phone read it, but that the message is now read for the account as a whole and should no longer appear unread anywhere.
6. Event appended to the ordered log. This state change is appended as a new entry to the account’s ordered event log, which is the durable record every device, online or not, will eventually observe.
7. Fan-out to other online devices. The fan-out dispatcher immediately pushes this read-state event to the account’s other currently connected devices — the laptop, the tablet, and the browser session — through their respective gateway connections.
8. Devices apply the update locally. Each receiving device updates its local view: the unread badge on that conversation clears, and if the conversation is currently open on that device too, the message visually shows as read.
9. Offline device catch-up. A device that was offline at the time simply is not pushed the event live. Instead, when it reconnects, it requests all event log entries since the last position it had successfully processed, replays them in order, and arrives at the identical correct state without ever needing a special-cased “catch-up” code path separate from normal operation.

3.2 The Synchronization Model — Why It Is Built This Way

The central design decision in this entire system is how state is represented and propagated, and it deserves its own dedicated treatment because getting it wrong is what causes the frustrating bugs users actually notice, like a message staying stuck as unread forever.

3.2.1 Why a Single Shared Boolean Is Not Enough

A naive design stores one read flag per message and lets any device flip it to true. This looks correct until you consider ordering: if a slow network causes a stale “mark as unread” action — perhaps from a user explicitly marking a message unread again on their laptop — to arrive at the server after a newer “mark as read” event from their phone, a naive last-write-wins approach based purely on arrival order at the server can silently revert the message to unread even though the phone’s action was the user’s actual latest intent. The fix is to attach a logical timestamp, not merely rely on server arrival order, so state transitions can be ordered correctly regardless of network delay variance between devices.

3.2.2 Per-Device State Plus Account-Level Aggregate State

The system tracks two layers of state deliberately. Per-device state records exactly what each individual device has acknowledged and observed, which is essential for accurate delivery receipts, since a sender legitimately wants to know their message reached at least one, or in some product designs every, device of the recipient. Account-level aggregate state is the derived, simplified view — such as overall read or unread — that is what actually drives the UI badge every device displays. Keeping these as two explicit layers, rather than trying to cram both meanings into one field, is what avoids ambiguity about what a given state field actually represents.

3.2.3 Event Sourcing as the Synchronization Backbone

Rather than devices directly mutating a shared state row and hoping every other device notices, the system models every state change — message delivered, message read, message read again on another device — as an immutable event appended to an ordered, per-account log. Current state, for any device or for the account as a whole, is always a deterministic function of replaying that event log in order. This single decision solves several problems simultaneously: offline devices can catch up by replaying missed events instead of needing a special reconciliation protocol, debugging a “why does this message look wrong” support ticket becomes a matter of reading the event history, and adding a new kind of state in the future — such as message reactions — fits the same pattern without redesigning the sync mechanism.

3.2.4 Logical Clocks and Ordering Across Devices

Because different devices’ local clocks cannot be trusted to agree precisely, the server assigns each event in an account’s log a strictly increasing logical sequence number at the moment it is appended, rather than relying on client-supplied wall-clock timestamps for ordering decisions. Every device tracks the last sequence number it has successfully processed, so catching up is simply asking the server for everything after that number, and the strict server-assigned ordering guarantees every device that replays the log arrives at the exact same final state, regardless of the order in which the underlying network delivered events to each device.

3.2.5 Conflict-Free State Transitions

The state transitions this system needs — message becomes delivered, message becomes read — are deliberately modeled so they only ever move state forward, never requiring the system to decide between two genuinely conflicting simultaneous edits the way a collaborative document editor must. Read is monotonic: once true, later events do not need to un-read a message due to a race, since even an explicit “mark as unread” action is itself just a new forward-moving event with a higher sequence number, not a rollback of history. This conflict-free property is precisely why the system does not need heavyweight distributed consensus between devices; it needs only a single ordering authority — the server-assigned sequence number — which is a substantially simpler problem.

3.2.6 Scoping State Per Conversation Rather Than Globally

State is tracked at the level of an individual conversation and message, not as one giant undifferentiated blob for the whole account, which matters for both performance and correctness. A device that only has one specific conversation open only needs to apply the subset of events relevant to that conversation to update its visible UI immediately, while still eventually processing the full account-wide event stream to keep its unread counters and conversation list accurate. This scoping also means a burst of activity in one very active group conversation does not need to block or delay the propagation of a state change in a quiet one-on-one conversation, since consumers can reasonably choose to prioritize applying events for whichever conversation is currently visible to the user first.

3.2.7 Reconciling Client-Side Optimistic Updates

For the interaction to feel instant, a device typically updates its own local UI optimistically the moment the user takes an action, clearing an unread badge locally before waiting for the server’s acknowledgment. This optimistic update must later be reconciled against the authoritative event the server actually appends, since the server may, in rare cases, order events differently than the client expected — for example if another device’s action was appended just before this one. The client’s reconciliation logic is intentionally simple: once the true server-ordered event stream is replayed, it always wins over any local optimistic guess, so the local optimistic update is best understood as a temporary visual prediction, never as a second source of truth competing with the server.

This chapter closes the loop between the “why” of the sync model and the “how” at the level of concrete state transitions, data structures, and concurrency patterns that keep everything correct under load.

4.1 The Full State Lifecycle

Every device, at any point in time, holds a local cursor representing the last sequence number from the account event log it has fully processed. Normal operation is simply this cursor advancing by one as each new pushed event arrives and is applied. Recovery from any disconnection, however long, is exactly the same mechanism: request all events after the current cursor, apply them in order, advance the cursor. There is intentionally no separate “resync” code path, since unifying normal operation and recovery into the same mechanism eliminates an entire class of bugs where the two paths behave subtly differently.

4.2 Server-Assigned Monotonic Sequence Numbers

Each account’s event log is backed by a strictly increasing counter, incremented atomically for every new event appended for that account. Because the counter is scoped per account rather than global, contention is naturally limited to a single user’s own events, and different accounts can be assigned sequence numbers fully in parallel with no shared bottleneck.

4.3 Vector of Per-Device Cursors

The server maintains, per device, the last sequence number that device has acknowledged processing — essentially a small vector of cursors keyed by device identifier. This is what allows the fan-out dispatcher to know precisely which events a reconnecting device is missing, computed as a simple range query rather than a broad table scan.

4.4 Fan-out on Write versus Fan-out on Read

Two general strategies exist for delivering new events to devices. Fan-out on write immediately pushes a new event to every currently connected device the moment it is appended to the log, which minimizes latency and suits accounts with a small, bounded number of devices — typically well under ten. Fan-out on read instead waits for each device to explicitly pull new events, which trades some latency for lower server-side push load, and is more common for accounts with very large device counts or unusual connectivity patterns. Because a single account rarely has more than a handful of linked devices, this system defaults to fan-out on write for online devices, combined with a pull-based catch-up for anything that was offline — capturing the low latency benefit of push without the fan-out cost explosion that would matter for a very different problem like a broadcast channel with millions of subscribers.

4.5 Idempotent Event Application

Every event carries its own sequence number, and devices track the last number applied, so if a network hiccup causes the same event to be delivered twice, the device simply detects that the incoming event’s sequence number is not greater than its current cursor and discards the duplicate — making the entire apply operation naturally idempotent without needing explicit deduplication logic beyond this cursor comparison.

4.6 Batching and Coalescing Rapid State Changes

When a user scrolls through and reads twenty messages in quick succession, generating twenty individual read events one at a time would be wasteful. The state sync service coalesces read events within a short time window into a single batched event covering a range of messages, reducing both log volume and the number of pushes to other devices, while still preserving full correctness since the batched event still carries one strictly increasing sequence number and still represents a forward-only state transition.

4.7 Backpressure and Flow Control on Gateway Connections

A device with a slow or congested network connection can fall behind the rate at which the fan-out dispatcher wants to push events. Gateway servers apply flow control, buffering a bounded number of pending events per connection and, once that buffer is exceeded, switching that device from live push mode into a “needs catch-up” mode where it simply pulls the backlog once it reconnects with more bandwidth — rather than letting one slow device’s buffer grow unboundedly and risk memory pressure on the gateway server.

Every design decision in this system involves a deliberate trade-off. A strong design does not pretend they do not exist — it names them clearly and picks the ones the product can genuinely absorb.

5.1 Advantages

The system must serve hundreds of millions of accounts, each with several linked devices, without any one account’s activity affecting another’s, and with a latency that feels instant to the user in the common case.

6.1 Sharding the Event Log by Account

Since nearly every operation — read, write, and catch-up — is naturally scoped to a single account, sharding the event log store by account identifier gives near-perfect horizontal scalability, since there is no cross-account coordination required for the vast majority of operations.

6.2 Scaling Gateway Servers Independently of Core Services

Because gateway servers only need to hold live connection state, they scale purely based on concurrent connection count, which can be scaled by simply adding more gateway instances behind the connection load balancer, entirely decoupled from how the state sync and messaging services scale based on write throughput.

6.3 Connection Affinity and Rebalancing

A device’s persistent connection sticks to one gateway server for its duration, and the presence service tracks this mapping so the fan-out dispatcher knows exactly where to route a push. When a gateway server is being scaled down or has failed, its connected devices are dropped and reconnect through the load balancer to a healthy instance, immediately followed by a catch-up request — which is why the unified catch-up mechanism described earlier is so central to the whole design; reconnection is not a special case, it is the normal recovery path.

6.4 Read-Heavy Aggregate State Optimizations

Querying “what is my current unread count across all conversations,” a very common operation every time a client app opens, is optimized by maintaining a continuously updated per-account unread summary as state events are applied, rather than requiring a scan across every conversation’s individual state on every app launch.

6.5 Batching Push Notifications for Offline Devices

If several events occur for an account while a device is offline, the system coalesces them into a single push notification rather than firing one push per event — both to respect platform push notification rate limits and to avoid needlessly waking a sleeping device’s radio multiple times in quick succession.

In a consumer messaging platform, correctness matters more than raw uptime numbers — a message showing wrong state is worse than a brief delay in showing it at all.

7.1 No Single Point of Failure

Gateway servers are stateless enough that losing one only affects the devices connected to it, which simply reconnect elsewhere. The event log store is replicated across multiple nodes with automatic failover, since it is the one component whose data loss would actually be visible and painful to users — unlike the presence registry, which is safely rebuildable.

7.2 CAP Theorem Trade-off

This system favors availability and partition tolerance for the live push path: if the fan-out dispatcher briefly cannot reach a gateway server, it simply queues the event and retries, rather than blocking the sender’s own experience. For the event log itself, however, durability and ordering correctness are non-negotiable, so writes to the log require a quorum acknowledgment from replicas before being confirmed to the client — trading a small amount of write latency for the guarantee that an acknowledged state change is never silently lost even if the node that first accepted it immediately fails.

7.3 Idempotency and Exactly-Once-Apply Semantics

As covered in the algorithms section, every event’s sequence number makes reapplication safe, which is what allows the system to use simple at-least-once delivery internally between services, relying on idempotent application at the consumer rather than needing complex distributed transaction coordination to guarantee each event is processed exactly once.

7.4 Handling Extended Device Offline Periods

A device that has been offline for weeks — perhaps a tablet left in a drawer — still needs to catch up correctly, which could mean replaying a very long event history. The system handles this gracefully by periodically compacting the event log into snapshots of current aggregate state plus a shorter recent tail of raw events, so a device catching up after a long gap loads the latest snapshot plus only the small remaining tail, rather than replaying months of granular history.

7.5 Disaster Recovery

The event log store is replicated across multiple regions. If an entire region becomes unavailable, devices reconnect through the load balancer to gateway servers in a healthy region, and because the catch-up mechanism is cursor-based rather than session-based, devices resume exactly where they left off with no special cross-region reconciliation logic needed.

A multi-device sync layer sees enormously sensitive per-user data every second — who read what, on what device, at what time — and its security posture must be as deliberate as the sync design itself.

8.1 Device Authentication and Authorization

Each device is registered to an account through its own individually issued credential, rather than all devices sharing one account-wide secret, so that a single compromised or lost device’s access can be revoked independently without forcing every other device to re-authenticate.

8.2 End-to-End Encryption Considerations

When message content is end-to-end encrypted, each device typically holds its own encryption keys, which means a newly linked device must go through a secure key exchange — often mediated by an already-trusted existing device — before it can decrypt message history, and this key provisioning step is a separate concern from, but must be carefully coordinated with, the state synchronization mechanism described in this tutorial, since a device cannot correctly display “read” status for content it cannot yet decrypt.

8.3 Protecting Presence and State Metadata

Even without exposing message content, read receipts and online presence are themselves sensitive metadata that reveal behavioral patterns, so access to another user’s read state must respect the recipient’s privacy settings, and internally this metadata is encrypted in transit and at rest with the same rigor as message content — not treated as low-sensitivity operational data.

8.4 Revoking a Lost or Stolen Device

The device registry supports immediate revocation, which forcibly terminates that device’s live connection, invalidates its credential, and removes it from future fan-out targeting — all without needing any cooperation from the revoked device itself, since a stolen device obviously cannot be trusted to voluntarily disconnect.

8.5 Rate Limiting Abusive Clients

A misbehaving or compromised client rapidly sending malformed or excessive state events is rate-limited at the gateway layer, both to protect shared infrastructure and to prevent one bad actor’s device from degrading the real-time experience of other unrelated accounts sharing the same gateway server.

Because this system’s correctness is about every device eventually agreeing on the same state, monitoring here is less about “is the server up” and more about “are devices actually converging fast enough.”

9.1 Key Metrics

• End-to-end propagation latency, from a state change being recorded to it being reflected on every other online device, since this directly measures whether the system feels instant to users.
• Per-device catch-up lag, meaning how far behind the latest sequence number a reconnecting device typically is, which reveals whether devices are disconnecting more often or for longer than expected.
• Gateway connection counts and churn rate, since a sudden spike in reconnections often signals an underlying network or deployment issue.
• Event log write latency and replication lag, since these directly bound how quickly a state change can even be considered durable, let alone propagated.

9.2 Distributed Tracing Across the Fan-out Path

Every state event carries a trace identifier from the moment it is generated on the originating device through log append, fan-out dispatch, and delivery to every other device — allowing an engineer to answer precisely why a specific user’s laptop took eight seconds to show a message as read, rather than guessing at which of several services introduced the delay.

9.3 Consistency Auditing

Because correctness here is about every device eventually agreeing on the same state, the system periodically runs background consistency checks comparing each active device’s last-known state against the authoritative event log, surfacing and alerting on any device that appears stuck or diverged, which is often the earliest signal of a bug in the sync logic before it generates a flood of user complaints.

9.4 Alerting

Sustained growth in average catch-up lag across many devices, or a rising rate of consistency audit mismatches, triggers an on-call alert, since these patterns typically indicate a systemic problem in the fan-out or log replication path rather than an isolated device issue.

Deploying a multi-device sync system means deploying two very different tiers together — long-lived stateful connections at the edge, and durable ordered writes at the core.

10.1 Multi-Region Gateway Deployment

Gateway servers are deployed across regions close to where users actually are, so devices connect to the nearest healthy region, minimizing connection latency, while the account event log itself may live in a specific primary region per account with cross-region replication for resilience.

10.2 Containerized, Auto-Scaled Services

Gateway, presence, messaging, and state sync services all run as independently scaled containerized workloads, with gateway server scaling driven primarily by live connection count and state sync service scaling driven primarily by event write throughput — reflecting how differently these two layers behave under load.

10.3 Rolling Deployments Without Dropping Connections Unsafely

Because gateway servers hold live connections, deployments use a graceful drain pattern: a server being replaced stops accepting new connections, notifies its currently connected devices to reconnect elsewhere, and only shuts down once those devices have migrated — rather than abruptly severing thousands of live connections at once during a routine deployment.

10.4 Canary Releases for Sync Logic Changes

Because a subtle bug in state transition logic could cause silent, hard-to-notice inconsistencies rather than a loud failure, changes to the state sync service are rolled out to a small percentage of accounts first, with the consistency auditing described in the monitoring section specifically watching that canary population for any increase in divergence before a full rollout proceeds.

Storage choices here are shaped less by size and more by access pattern: many small, high-velocity mutations plus a very long tail of durable, append-only history.

11.1 Account Event Log Storage

The event log is the most performance-critical store in the system, since it is both written to on every state change and read from during every device catch-up. An append-only, log-structured storage engine, partitioned by account identifier, fits this access pattern well, since writes are always appends and reads are always sequential range scans from a given sequence number onward — both of which log-structured storage is specifically optimized for.

11.2 Per-Device State Store

The mapping of each device’s last-processed sequence number, along with basic device metadata like platform type and last-seen timestamp, is stored in a fast key-value store keyed by device identifier, since this needs to be read and updated on nearly every event delivery and every reconnect, making low single-digit-millisecond latency essential.

11.3 Message Content Store

Kept separate from the state and event log stores, since message content is comparatively large, written once, and read far less frequently than state is updated. This separation means the extremely hot, small, frequently mutated state data is never competing for the same storage engine’s resources as the larger, cooler message bodies.

11.4 Presence and Connection Registry

An in-memory distributed store tracks which gateway server instance each currently connected device is attached to. This is inherently ephemeral and does not need the same durability guarantees as message state, so it is optimized purely for extremely fast lookups and writes, with the understanding that if this data is lost, it simply gets rebuilt as devices reconnect and re-register.

11.5 Caching the Latest Aggregate State

While the event log is authoritative, recomputing an account’s current aggregate read and delivered state by replaying its entire history on every request would be wasteful. A cache holding the current computed state per conversation, invalidated and updated incrementally as new events are appended, avoids that recomputation cost while the event log remains available as the ground truth for rebuilding the cache from scratch if it is ever lost or found inconsistent.

11.6 Snapshot Storage for Compaction

Periodically computed snapshots, described further in the reliability section, are stored separately from both the raw event log and the live cache, since they serve a different purpose: a durable, versioned checkpoint of aggregate state at a specific sequence number that a long-offline device can load as its starting point instead of replaying the entire history from the very beginning, followed only by the shorter tail of events since that snapshot.

The API surface here has two very different halves — a persistent bidirectional channel to devices, and a set of tightly bounded internal service boundaries.

12.1 Client-Facing Protocol

Devices maintain a persistent bidirectional connection to a gateway server, over which the server pushes new messages and state events, and the device sends its own outgoing messages, read acknowledgments, and delivery acknowledgments. A lightweight request-response API layered on top of that same connection lets a device explicitly request catch-up: give me every event since sequence number N.

12.2 Internal Service Boundaries

• Gateway Service: owns the live connection to each device, purely a transport concern, stateless beyond knowing which devices are attached to it right now.
• Presence Service: the directory of which device is connected to which gateway instance, consulted by the fan-out dispatcher on every push.
• Messaging Service: owns sending and storing message content, independent of how its delivery and read state later evolves.
• State Sync Service: owns the event log, sequence number assignment, and the logic for computing aggregate account-level state from per-device events.
• Fan-out Dispatcher: consumes new log entries and routes them to the right gateway connections or to the push notification service for offline devices.

12.3 Communication Patterns

Device-to-server communication happens over the persistent connection for low latency. Internally, the state sync service publishes new events to the fan-out dispatcher through an asynchronous, ordered messaging backbone rather than direct synchronous calls, which allows the dispatcher to be scaled and restarted independently without ever blocking the critical path of recording a state change durably.

The patterns worth applying, and the ones worth explicitly avoiding, in a multi-device sync platform.

13.1 Patterns Used

• Event Sourcing: the entire synchronization model is built on an immutable, ordered log of state-change events as the single source of truth, described throughout the sync model section.
• CQRS (Command Query Responsibility Segregation): writes go through the state sync service and append to the log, while reads of current aggregate state are served from a separately maintained, continuously updated cached projection — keeping the two paths independently optimized.
• Publish-Subscribe: the fan-out dispatcher distributes new events to any number of interested gateway connections without the state sync service needing to know which specific devices exist or where they are connected.
• Cursor-Based Pagination for Catch-up: reconnecting devices request events after a specific sequence number, the same well-understood pattern used for paginating large result sets, applied here to event replay instead.
• Idempotent Receiver: every device safely ignores events at or below its already-processed cursor, making duplicate delivery harmless rather than something that must be prevented entirely.

13.2 Anti-patterns to Avoid

Practical wisdom that separates a sync system that works in a demo from one that works on a subway train with intermittent LTE.

14.1 Best Practices

• Model every state change as a forward-only, idempotent event rather than a direct mutation, even when it feels like overkill for a simple boolean flag.
• Unify the live push path and the reconnect catch-up path into the same underlying mechanism wherever possible, since divergent paths are where the hardest-to-reproduce bugs live.
• Invest early in consistency auditing tooling, since sync bugs are often silent and only surface as vague, hard-to-reproduce user complaints long after the underlying bug shipped.
• Keep per-device state and account-aggregate state explicitly separate concepts in both the data model and the code, rather than conflating the two.
• Design for devices that disconnect constantly and unpredictably as the normal case, not an edge case, since mobile network behavior makes this the majority of real-world device behavior.

It also helps to think about testing strategy specifically for this class of problem, since the usual approach of writing unit tests for individual functions catches far fewer real bugs here than deliberately simulating the messy conditions the system must survive: randomly dropping and reordering network messages between simulated devices, forcing devices offline for varying durations before catch-up, and running the consistency auditing logic itself against these simulated scenarios as part of the automated test suite — rather than only running it in production. Many of the subtle bugs in real-world multi-device sync systems — a message that syncs correctly ninety-nine times out of a hundred but occasionally sticks in the wrong state — are the kind that a chaos-style simulation environment surfaces reliably, while conventional testing against a single well-behaved connection almost never does.

14.2 Common Mistakes

• Assuming devices will always process events in the exact order they were pushed over the network, rather than relying on the server-assigned sequence number for correctness.
• Forgetting that a device can receive the same event more than once, and building state application logic that is not safe to run twice.
• Under-provisioning gateway servers for connection count while over-focusing scaling effort on message throughput, when in a real deployment the sheer number of long-lived idle connections is often the binding constraint.
• Not accounting for platform-specific push notification rate limits, leading to important state changes never reaching a device that has gone to sleep and is relying entirely on push to wake it.

Variations of this pattern show up across every major consumer messaging platform, each shaped by their own scale and platform constraints.

The questions that come up most often in interviews, design reviews, and internal debates for a multi-device sync platform.

It is treated exactly like a device recovering from an extremely long offline period: it starts from sequence number zero, or from the account’s latest compacted snapshot plus the short recent tail, and replays forward — using precisely the same catch-up mechanism as any reconnecting device rather than a separate first-time-setup code path.

Periodic compaction, described in the reliability section, collapses old raw events into a compact snapshot of current aggregate state, after which the original events behind that snapshot can be safely archived or discarded, since replaying from the snapshot forward produces the identical result as replaying the full original history.

The per-device and account-aggregate delivery state still tracks accurately internally, since that is needed for core functionality like knowing a message reached the recipient, but the read-state event specifically is filtered out before it is exposed to the sender’s devices if the recipient has disabled read receipts — keeping the privacy control enforced at the exposure boundary rather than by simply not tracking the state at all.

Polling trades implementation simplicity for either high latency, if polling infrequently, or high wasted load, if polling frequently enough to feel responsive, and at the scale of hundreds of millions of accounts that wasted load becomes enormous. Persistent connections with server-initiated push deliver near-instant propagation with dramatically less overall network chatter, which is why virtually every production messaging platform at this scale uses persistent connections rather than polling.

The core per-device, event-sourced synchronization mechanism stays the same, but the fan-out target expands from one account’s own devices to every participant’s devices across every one of their own linked devices, and delivery and read status become per-participant rather than a single account-level aggregate — since in a group each member’s read status is independently meaningful rather than being collapsed into one shared flag.

Because the event log is authoritative and every device is expected to be a deterministic function of replaying it, a divergent device is, by definition, either running buggy client logic or has a corrupted local cache. The consistency auditing process described in the monitoring section is designed to catch this pattern, and the standard remedy is instructing the affected device to discard its local state entirely and rebuild it from a fresh full replay of the snapshot plus event tail — which is always guaranteed to produce the correct result since it does not depend on that device’s previous, possibly incorrect, local state at all.

The user-facing action itself — marking a message read on the device where they are actually reading it — always applies instantly and locally regardless of connectivity, since that device obviously has the message in front of the user already. What requires connectivity is only the propagation of that state to the account’s other devices, and if the device is offline when the user reads a message, the read event is simply queued locally and sent to the server as soon as connectivity is restored, at which point normal fan-out to the other devices proceeds exactly as described throughout this tutorial.

Designing multi-device message state synchronization is fundamentally an ordering and propagation problem, not a storage problem. The hard part is guaranteeing every device eventually converges on the same correct state despite constantly connecting and disconnecting.