Picture sending a message to a friend and watching a small checkmark change color the instant they open it.

That tiny visual signal — “read” — carries an enormous amount of social meaning: relief, anxiety, impatience, reassurance. It is one of the most emotionally loaded features in any messaging product, and also, underneath the surface, one of the more deceptively hard systems to build correctly at scale.

The feature sounds trivial: store a timestamp when a message is read, show it to the sender. The difficulty is not in that single sentence — it’s in what happens when you multiply it by billions of messages, by group chats with hundreds of participants each generating their own individual read event, by users who explicitly do not want their read status shared, and by mobile devices with unreliable connectivity that come online in unpredictable bursts. Read receipts sit exactly at the intersection of three hard problems: real-time distributed systems, fan-out at scale, and privacy engineering.

1.1 A Short History of Read Receipts

1.2 Why This Is a Genuinely Interesting System Design Problem

At first glance, this looks like “write a timestamp to a database.” What makes it interesting is the combination of constraints that all apply simultaneously: the write volume is a multiple of message volume (one read event per recipient per message, and group chats multiply this further); the read side needs to be near-instant, because a delayed read receipt feels broken to users watching for it; the data is inherently sensitive, since “when someone read something” is intimate behavioral information; and the whole feature must degrade gracefully, because losing a read receipt is a minor annoyance, but losing a message is unacceptable — the two must never share a failure domain.

Requirements first, then why every naive design collapses under real load.

2.1 Functional Requirements

• Delivery tracking. The system must track when a message reaches a recipient’s device (delivered), separately from when it is actually seen (read).
• Read tracking, per recipient. For one-on-one chats, track a single read timestamp per message. For group chats, track a per-member read state, since different members read messages at different times.
• Real-time notification to the sender. When a recipient reads a message, the sender’s client should reflect that (the ticks turning blue) within roughly a second, while they are actively using the app.
• Privacy controls. A user must be able to disable sending read receipts, which — following the fairness convention established across the industry — also disables their ability to see others’ read receipts.
• Group read summaries. For group chats, provide an aggregate view (“Read by 12 of 40”) as well as a detailed per-member breakdown, without either view leaking data to a member who has opted out of sharing their own read status.
• Offline and multi-device support. A message must be correctly marked as read even if the recipient reads it on a phone while offline, or across multiple linked devices, with the read event syncing once connectivity resumes.

These requirements interact in ways that shape the architecture from the outset. Supporting per-user privacy control means read state cannot simply be baked into a message as a fixed attribute at write time — it must be evaluated dynamically against the reader’s current privacy setting every time it’s about to be exposed to someone else, which pushes the design firmly toward a model where raw read events pass through an explicit, centrally-enforced filtering step before they ever reach another person’s screen. Similarly, supporting both a real-time aggregate view and an on-demand detailed view for group chats means the system needs two distinct read paths with different performance characteristics, rather than one path serving both needs adequately.

2.2 Non-Functional Requirements

2.3 Why a Naive Approach Fails

The naive design: whenever a client opens a message, immediately call an API that writes a row to a shared “reads” table and pushes a real-time notification directly to every other participant in the conversation. Let’s see why this collapses under real conditions.

Every major design decision in this tutorial — batching, fan-out strategy, privacy enforcement at the data layer rather than the UI layer, and strict isolation from the message-delivery path — exists specifically to address one or more of these four failure modes.

The vocabulary we’ll rely on for the rest of the tutorial, each with an analogy and a concrete example.

3.1 Delivered vs. Read State

What it is: Two distinct states in a message’s lifecycle. “Delivered” means the message successfully reached the recipient’s device (arrived in their app, even if unopened). “Read” means the recipient actually opened the conversation and viewed the message.

Why it exists: These carry very different meaning and are produced by different events — delivery is a network/infrastructure event (did the device receive the payload), while read is a genuine user-behavior event (did a person look at it). Conflating them would either under-report (“delivered” shown as “read” when nobody actually looked) or be impossible to implement consistently.

Practical example: A single grey tick means sent from the sender’s device successfully; two grey ticks mean delivered to the recipient’s device; two blue ticks mean the recipient has actually viewed it — three genuinely distinct signals stacked into one compact visual.

3.2 Read Cursor / High-Water Mark

What it is: Instead of storing an individual “read” flag for every single message a user has ever received, many systems track a single pointer per conversation, per user — the ID or timestamp of the most recent message that user has read. Everything at or before that point is implicitly read.

Why it exists: Storing a discrete boolean flag per message per recipient is extremely storage-inefficient at scale, especially for group chats. A single cursor value achieves the same practical outcome (knowing what’s been read) with vastly less storage and far simpler updates.

Production example: When a user opens a conversation and scrolls to the latest message, the client sends a single “read up to message ID 48213” update, rather than 40 separate “message X is read” events for each message currently visible on screen.

3.3 Fan-Out on Write vs. Fan-Out on Read

What it is: Two opposite strategies for delivering an update (like a read receipt) to multiple recipients. Fan-out on write pushes the update proactively to every interested recipient’s inbox or connection the moment it happens. Fan-out on read has recipients pull or query for the update only when they next look at the relevant screen.

Why it exists: Neither strategy is universally better — the right choice depends on how many people need the update and how real-time it truly needs to be for each of them.

Practical example: The message sender, actively watching their screen, needs the read receipt pushed in real time (fan-out on write, to one specific recipient). A group member who is not currently viewing the conversation does not need a live push at all — the read summary can simply be computed on demand the next time they open the chat (fan-out on read).

3.4 Presence and Connection State

What it is: Whether a given user’s client currently has an active, live connection to the messaging backend (typically via a persistent WebSocket or similar long-lived connection), versus being offline or backgrounded.

Why it exists: Real-time push only works for currently-connected clients. Knowing connection state lets the system decide whether to push a read receipt update immediately over an open connection, or queue it for later delivery via a different mechanism (a mobile push notification, or simply waiting for the client’s next sync).

3.5 Idempotency

What it is: A property of an operation where performing it multiple times has the same effect as performing it once.

Why it exists: Mobile networks are unreliable; a client might send the same “mark as read” event twice due to a retry after a timeout, even though the first attempt actually succeeded. The system must handle this without recording a duplicate or double-counting.

Practical example: Because the read cursor is a “read up to message X” pointer rather than a counter, applying the same update twice naturally has no additional effect — moving a bookmark to page 200 a second time doesn’t change anything if it’s already there.

3.6 Eventual Consistency for Read State

What it is: A consistency model where, after an update stops happening, all parts of the system will eventually converge on the same value, but there may be a brief window where different parts of the system see slightly different, temporarily stale values.

Why it exists: Requiring every single component (cache, replicas across regions, every connected client) to agree on the exact read state at every instant would demand expensive, slow coordination for a feature where a delay of a second or two is genuinely harmless. Accepting eventual consistency here is a deliberate, well-justified trade for speed and availability.

Where strict consistency would be wrong to demand: Message content itself typically needs much stronger delivery guarantees than read state does — losing or duplicating a message is a real problem, while a read tick that updates half a second later than theoretically possible is imperceptible to virtually everyone.

3.7 Debouncing

What it is: A technique where rapid, repeated triggers of the same logical action are collapsed into a single effective action, typically by waiting for a short pause in activity before actually acting.

Why it exists: A user scrolling quickly through fifty messages doesn’t need fifty separate “read up to” calls fired as their eyes pass over each one — only the final scroll position, once scrolling settles, is meaningful.

Practical example: A client waits roughly 300–500 milliseconds after the last scroll movement before sending a single cursor-update call reflecting wherever the user ultimately stopped, rather than firing a request on every intermediate frame of the scroll.

3.8 Backpressure

What it is: A mechanism by which a system signals upstream producers to slow down when a downstream consumer cannot keep up with incoming volume, rather than silently dropping data or building an unbounded backlog.

Why it exists: During an extreme traffic spike (many large groups all becoming highly active at once), the Batching Worker Pool could theoretically fall behind the event stream’s incoming rate. Backpressure — implemented through the event stream’s own consumer-lag mechanics and bounded in-flight processing limits — ensures the system degrades predictably (a growing but bounded, monitorable lag) instead of failing catastrophically or silently losing events.

Six zones: edge, real-time connection, core services, async processing, storage, and observability wrapped around everything.

The system separates into six zones: the edge layer (load balancer and API gateway handling every inbound connection), the real-time connection layer (WebSocket gateway tracking presence), the core services (read-receipt service, message service, privacy/settings service), the async processing layer (event stream and batching workers), the storage layer, and observability wrapped around everything.

4.1 Component Breakdown

4.2 Walking Through the Diagram Step by Step

It’s worth tracing Fig. 1 as a narrative, since the sequencing carries most of the design intent. A recipient opens a conversation on their client, and the client sends a cursor-update request over HTTPS. That request first hits the Load Balancer, which routes it to a healthy API Gateway instance; the gateway authenticates the request and applies rate limiting before forwarding it to the Read Receipt Service.

The Read Receipt Service’s very first action is to check the requesting user’s privacy setting with the Privacy Settings Service — a fast, cache-backed lookup, not a slow synchronous round trip on every call. It then writes the new cursor position to the Read Cursor Store, and — assuming read receipts are enabled for this user — emits a read event onto the Event Stream. Crucially, the original HTTPS request returns success to the client at this point; the client never waits for what happens next.

The Batching Worker Pool consumes events from the stream on its own schedule, accumulating them over a short batching window per conversation. For each batch, it consults the Presence Service to find out which participants are currently connected and to which specific WebSocket Gateway node, filters out anyone who has opted out of read receipts, computes a coalesced summary, and pushes that summary through the WebSocket Gateway to each currently-connected, interested recipient — most commonly, the original sender, watching their screen for exactly this update. Meanwhile, the sender’s own client maintains a persistent connection through the same Load Balancer and API Gateway into the WebSocket Gateway, registered and kept alive via the Presence Service, ready to receive that push the moment it arrives.

Zoom in on how the Read Receipt Service and Batching Worker Pool actually process a “mark as read” call.

5.1 How the Read Receipt Service Processes a “Mark as Read” Call

1. Receive the request. The client sends “I have read up to message ID 48213 in conversation C” when the user opens or scrolls through a conversation — a single cursor update, not one call per message.
2. Validate and check privacy settings. The service checks with the Privacy Settings Service (typically via a fast cached lookup, not a synchronous call every single time) whether this user has read receipts enabled at all.
3. Write the cursor. The new read position is written to the Read Cursor Store, keyed by (user ID, conversation ID), overwriting the previous value — this is a simple, fast, idempotent update.
4. Emit an event. A ReadCursorUpdated event is published to the event stream, and the API call returns success to the client immediately — the client does not wait for the fan-out to anyone else.
5. Downstream fan-out happens asynchronously, handled entirely by the Batching Worker Pool, decoupled from the client’s request/response cycle.

5.2 How the Batching Worker Pool Prevents Fan-Out Explosion

This is the component doing the heaviest lifting for scalability. Instead of reacting to every single read event immediately and individually, workers accumulate events over a short time window (for example, 500 milliseconds to a few seconds, tunable per conversation size) and then compute a single, coalesced update to push.

5.3 Presence-Aware Routing

Before pushing a real-time update, the Batching Worker asks the Presence Service whether the target user (the sender, in a 1:1 chat) is currently connected, and if so, to which WebSocket Gateway node. If the user is offline, the real-time push step is skipped entirely — there’s no point pushing to a closed connection — and the updated read state simply becomes available the next time that user’s client reconnects and syncs, via a normal API fetch rather than a push.

5.4 Concurrency Inside the Batching Worker

A single worker instance processes many conversations concurrently, since the bulk of the time per unit of work is spent waiting on I/O (the presence lookup, the privacy check, the push call to the gateway) rather than local computation. Workers use a bounded pool of concurrent batch-processing tasks, keyed by conversation ID, so that one extremely large, very active group conversation being processed does not starve the processing of many smaller, quieter conversations sharing the same worker instance — a classic fairness concern in any shared worker pool design.

5.5 Partitioning Strategy for the Event Stream

The read-events topic is partitioned by conversation ID, ensuring every event for a given conversation is processed in order by the same consumer, while entirely unrelated conversations are processed fully in parallel across the worker fleet. This ordering guarantee matters specifically for the deduplication step in batch processing — the Batching Worker needs to reliably see a given user’s cursor updates for one conversation in the order they actually happened, so that “keep only the latest position per user” logic behaves correctly, while there is no need for any ordering relationship between events belonging to different conversations at all.

5.6 Consistency Model for the Read Cursor Store

Because a stale or slightly-delayed cursor value is a minor, self-correcting inconvenience rather than a correctness violation — the true worst case is a read tick updating a moment later than it ideally would — the Read Cursor Store can safely use a quorum-based, eventually-consistent replication model rather than paying the latency cost of strict linearizable consistency across replicas. This mirrors a broader system design principle: match the consistency guarantee to the actual cost of being briefly wrong, rather than defaulting to the strongest (and most expensive) guarantee everywhere out of caution.

5.7 Handling Clock Skew Across Clients

Client-reported timestamps (used, for example, to determine which of several near-simultaneous cursor updates from the same user is genuinely the latest) cannot be fully trusted, since a device’s local clock can be skewed by seconds or more relative to server time. Rather than relying purely on the client-supplied timestamp to resolve ordering, the server additionally stamps each incoming event with its own receipt time, and the deduplication logic in the Batching Worker primarily orders events by monotonically increasing message ID or sequence position within the conversation — a value the server controls and can guarantee is strictly ordered — falling back to server-side receipt time only as a tiebreaker, rather than trusting client clocks as the primary ordering signal.

5.8 Networking Considerations

Because the WebSocket Gateway holds a very large number of concurrent long-lived connections, and each incoming push needs to find its way to the correct node quickly, gateway nodes and the Presence Service they depend on are typically deployed within the same availability zone to minimize the network hops on this latency-sensitive path. Internal calls between the Batching Worker, Presence Service, and WebSocket Gateway use a binary, connection-multiplexing protocol (gRPC over HTTP/2) rather than opening a fresh connection per call, which matters given how frequently these internal hops occur relative to the comparatively rarer client-facing API calls.

Trace one read event from a recipient’s tap all the way to the sender’s blue ticks — first for 1:1, then for group.

6.1 Lifecycle of a Read Receipt in a 1:1 Chat

1. Recipient opens the conversation and scrolls to the latest message.
2. Client sends a single cursor-update call through the Load Balancer and API Gateway to the Read Receipt Service.
3. Read Receipt Service checks the recipient’s privacy setting. If read receipts are disabled for this user, the cursor is still stored internally (the app itself needs to know what the user has seen, for its own unread-count badge), but no event is emitted for external fan-out.
4. If enabled, the cursor is written and an event is emitted to the stream.
5. The Batching Worker picks up the event, checks the sender’s presence, and — if the sender is actively connected — pushes a real-time “read” update through the WebSocket Gateway.
6. The sender’s client updates the two ticks from grey to blue, typically within a second of the recipient actually reading the message.

6.2 Lifecycle of a Read Receipt in a Large Group Chat

This is the key design decision for group chats: the real-time push carries only a lightweight aggregate (“read by 35 of 40”), not the full list of 35 names, to every connected member. The detailed breakdown — which specific members have read it — is computed on demand (fan-out on read) only when a member actually opens that detail view, since most members never do, and pushing the full detailed list to everyone on every update would be pure waste.

6.3 Multi-Device Synchronization

When a user reads a message on their phone, other devices logged into the same account (a tablet, a linked web client) need to reflect that the message is no longer unread, without generating a redundant read-receipt event visible to the sender. This is handled by distinguishing between an account-level read cursor (used purely for that user’s own unread-count and badge state, synced silently across their own devices) and the externally-visible read event (emitted only once, the first time any of that user’s devices crosses a given cursor position), so the sender sees exactly one “read” transition, never one per linked device.

6.4 Muted and Archived Conversations

A conversation a user has muted or archived still needs correct read-cursor tracking internally — their own client still needs to know what’s unread for badge-count purposes if they ever revisit it — but the product experience typically suppresses real-time push notifications related to that conversation entirely, including any read-receipt-related traffic that would otherwise wake up their connection or consume gateway push capacity for a conversation they’ve explicitly indicated they aren’t actively watching. The Batching Worker’s presence check can incorporate this preference alongside raw connectivity status, treating a muted conversation similarly to how it treats an offline recipient — skip the real-time push, and let the client pick up the current state on its own next sync — which further reduces unnecessary push volume beyond what pure online/offline presence alone would achieve.

Every design choice above trades something away — here they are, side by side.

7.1 Key Trade-off: Per-Message Flags vs. Read Cursor

Most production messaging systems use the cursor model as the default, sometimes supplementing it with sparse per-message tracking for group chats where a detailed “who exactly has read this specific message” view is a required feature, rather than applying dense per-message tracking to every single message universally.

7.2 Key Trade-off: Push-Heavy vs. Pull-Heavy Group Read Summaries

Pushing every detail to every member in real time (push-heavy) gives the most immediate experience but does not scale to large groups. Waiting for members to manually request detail (pull-heavy) scales perfectly but can feel sluggish. The hybrid used in this design — push a lightweight aggregate, pull the detail on demand — captures most of the perceived responsiveness at a small fraction of the fan-out cost, which is why it’s the standard approach across large-scale messaging platforms.

7.3 Key Trade-off: Strict vs. Eventual Consistency for Read State

This system deliberately chooses eventual consistency for read state, reserving strict consistency guarantees for the Message Service’s handling of actual message content, where the cost of being briefly wrong is far higher.

7.4 Key Trade-off: Self-Managed vs. Managed Infrastructure

As with several trade-offs discussed throughout this tutorial, there’s no universally correct answer — the right choice depends on scale, existing operational expertise, and how much of the engineering organization’s attention should reasonably be spent on infrastructure operations versus product-facing logic.

If caching is the central lever for read-heavy systems, batching is the equivalent lever here — because the risk is not read volume but write and fan-out volume.

8.1 Why Batching Is the Primary Scalability Lever

Just as caching is the central scalability strategy for a read-heavy system, batching is the central scalability strategy here, because the core risk in this system is not read volume but write and fan-out volume — specifically, the way group chat size multiplies the number of read events and potential push targets. Tuning the batching window is the single highest-leverage scalability knob in the whole design.

8.2 Handling Thundering-Herd Reconnection Spikes

A significant, often underestimated scaling challenge in this system is not steady-state traffic at all, but the reconnection spike that follows a widespread event — a brief regional network blip, an app update, or a WebSocket Gateway rolling deployment — during which a large fraction of previously-connected clients simultaneously attempt to reconnect and immediately request a full read-state sync for every open conversation. Left unmitigated, this “thundering herd” can produce a sudden, extreme load spike on the Read Cursor Store and Presence Service, exactly the moment those components are least equipped to absorb it if they were also affected by whatever triggered the disconnection in the first place. Mitigations include client-side jittered backoff (each client waits a small, randomized delay before reconnecting, spreading the herd out over a few seconds instead of one instant), and treating the initial post-reconnect sync request as a lower-priority, rate-limited class of traffic distinct from live, real-time updates, so it can be smoothed out without starving genuinely live interactions of capacity.

8.3 Sharding the Read Cursor Store

The Read Cursor Store is sharded by conversation ID (or a combination of user ID and conversation ID), so that the extremely high write volume of cursor updates is spread horizontally across many storage nodes, and a single very active, very large group conversation does not become a hotspot capable of overwhelming a single shard. Consistent hashing is used so that adding shard capacity as the platform grows only requires reshuffling a small fraction of existing data.

8.4 Scaling the WebSocket Gateway and Presence Layer

• Horizontal scaling with sticky-enough routing. A load balancer routes a client’s initial connection to a specific gateway node, and that connection persists on that node for its lifetime; new connections are distributed across the fleet to balance load, rather than every client always landing on one fixed node.
• Presence lookups via a fast, shared registry. Since any gateway node might need to know which other node holds a given user’s connection (for cross-node routing of a push), presence state is kept in a shared, low-latency store (backed by the cache layer) rather than only living in each gateway node’s local memory.
• Connection count as the primary scaling signal for the gateway fleet, since a single node’s capacity is fundamentally bounded by how many concurrent persistent connections it can hold open, which is a different constraint than typical CPU-bound autoscaling triggers.

8.5 Reducing Load at the Source

• Client-side debouncing: a client that scrolls rapidly through many messages sends one cursor update after scrolling settles, not one per message that scrolled past the viewport.
• Skip fan-out entirely when the sender is offline — there’s nobody to push to, so the Batching Worker’s presence check short-circuits this case cheaply before doing any coalescing work.
• Conflate rapid repeated cursor updates from the same user within a batching window into just the final, furthest position, since only the latest cursor value is meaningful.

8.6 Cost Optimization

Because this system’s dominant cost driver is write and fan-out volume rather than expensive per-request computation, cost optimization here looks different from a machine-learning-heavy system, but is no less important at platform scale. A few concrete levers matter most:

• Right-sizing the batching window against actual push volume savings. Measuring the coalescing ratio (raw events in versus pushes out) per conversation-size tier makes it possible to find the smallest batching window that still delivers most of the fan-out reduction, rather than over-widening it and sacrificing responsiveness for savings that were already captured by a shorter window.
• Tiered storage for the cursor store. Extremely hot, actively-updated cursors stay in the in-memory cache layer; cursors for conversations with no recent activity age out of cache and fall back to the underlying durable store on the rare occasion they’re needed, avoiding the cost of keeping every cursor for every conversation permanently resident in expensive memory.
• Connection cost awareness in the WebSocket Gateway fleet. Since gateway capacity is bounded by concurrent connections rather than raw compute, right-sizing this fleet against actual concurrent active-user counts (rather than total registered users, the overwhelming majority of whom are not connected at any given moment) avoids paying for far more standing capacity than is ever actually used.

Read receipts are a nice-to-have layered on top of a must-have. Every reliability decision here flows from that ordering.

9.1 Isolating Read-Receipt Failures from Message Delivery

The single most important reliability principle in this design is that the Read Receipt Service and Message Service are entirely separate services with separate infrastructure, separate datastores, and separate failure domains. If the Read Cursor Store, the event stream, or the Batching Worker Pool becomes degraded or fully unavailable, message sending and receiving must continue completely unaffected — the worst-case user-visible impact should be that the ticks stop updating, never that messages stop arriving.

9.2 Multi-Region Deployment

9.3 Graceful Degradation Ladder

1. Full real-time push — normal case, sender sees the tick change within about a second.
2. Delayed batch push — event stream or batching workers under heavy load; updates still arrive, just with a longer-than-usual delay.
3. Pull-on-open fallback — if real-time push is fully unavailable, the client falls back to fetching current read state via a normal API call whenever the conversation is opened or the app is foregrounded.
4. Stale-but-present state — worst case for the read-receipt subsystem specifically, the sender simply sees the last known state until the subsystem recovers, with message sending and receiving completely unaffected throughout.

9.4 Retry Semantics and Idempotency

Because cursor updates are idempotent (re-applying “read up to message X” has no additional effect if already applied), clients can safely retry a failed or timed-out cursor-update call without any special deduplication logic on the server, and the event stream’s consumers can safely process the same event more than once under an at-least-once delivery guarantee without producing an incorrect result.

9.5 Disaster Recovery and Backup

Read cursor data is, in principle, always recoverable in a degraded but honest way even after a serious data loss event: worst case, every user’s read state simply resets to “everything before now is unread,” which is a poor but recoverable user experience, never a silent correctness failure. Even so, production systems maintain regular snapshots and cross-region replication of the Read Cursor Store to avoid forcing that painful reset scenario. This stands in useful contrast to the Message Store, owned entirely separately by the Message Service, which holds genuinely irreplaceable data — actual message content — and therefore warrants a much stricter recovery point objective and recovery time objective than the read-receipt subsystem does. Distinguishing between data that is a source of truth and data that is a recoverable, derived signal shapes how much disaster-recovery investment each piece of the system actually deserves.

Read state is inherently personal behavioral data. Getting privacy enforcement wrong isn’t a minor bug — it’s a direct breach of user trust.

This section carries unusual weight in this particular system, since read state is inherently personal behavioral data, and getting privacy enforcement wrong is not a minor bug — it’s a direct breach of user trust.

10.1 Enforcing Privacy at the Data Layer, Not the UI Layer

The single most important security decision in this design is that the Batching Worker Pool checks the Privacy Settings Service and filters out opted-out users before any fan-out happens — never relying on the client application to simply hide data it already received. If a client received the raw, unfiltered read event for a user who has opted out, that data has already left the server’s control, and any modified or inspected client could recover it, fully defeating the privacy setting.

10.2 The Fairness Rule: Symmetric Visibility

The industry-standard convention — disabling read receipts also disables your ability to see others’ read receipts — is itself a privacy design decision, not just a product choice. It removes the incentive to disable the setting purely to gain a one-sided information advantage (seeing when others read your messages while hiding when you read theirs), and this rule needs to be enforced at the same central point as the opt-out itself, applied consistently whether the requester is asking about a 1:1 chat or a large group.

10.3 Data Minimization and Retention

Read cursor data, being a single pointer per conversation per user rather than a detailed log of every read event, is naturally minimal by design. Detailed per-event history (useful for debugging or analytics) should be retained only as long as genuinely necessary and with clear limits, rather than kept indefinitely simply because storage is cheap — the sensitivity of the data, not just its cost, should drive retention policy.

10.4 Authentication and Authorization

Every read-receipt API call is authenticated through the API Gateway, and the Read Receipt Service additionally verifies that the requesting user is actually a legitimate member of the conversation they’re claiming to update a cursor for, or querying read state on — preventing any possibility of one user probing another’s read state for a conversation they don’t belong to.

10.5 Audit and Compliance

Because read state is sensitive behavioral data, changes to privacy settings and access to detailed read-event data warrant their own audit trail — a record of when a user changed their read-receipt setting, and, for internal tooling, a record of which internal systems or personnel accessed detailed read-event history, distinct from the read-event data itself. This audit trail supports two distinct needs: giving the platform confidence that its own privacy guarantees are actually being enforced correctly over time (rather than trusting the design on paper alone), and satisfying external data-protection obligations that increasingly require platforms to demonstrate, not just assert, how sensitive personal data is handled. Access to this audit trail should itself be tightly restricted, since a log of “who looked at whose read-status history” is, in its own right, sensitive information.

10.6 Small-Group Deanonymization in More Depth

The small-group deanonymization risk mentioned earlier deserves a concrete mitigation strategy, not just acknowledgment. One practical approach: aggregate counts in conversations below a configurable member-count threshold are either suppressed entirely in favor of a simple binary “seen” or “not yet seen” indicator with no count attached, or the underlying calculation deliberately treats very small groups the same way it treats 1:1 chats — never surfacing a number that, combined with the reader’s own knowledge of who’s in the conversation, could only correspond to one possible person. This is a case where doing the mathematically “more informative” thing (an exact count) is actually the wrong product and privacy decision, and it’s a subtlety that’s easy to miss if privacy review happens only at the level of “is there a toggle” rather than at the level of “does this specific number leak something we promised not to.”

Observability tells you the ticks are actually behaving correctly, not just that the service is “up.”

11.1 Alerting Strategy

Alerts are tiered by user impact. Critical alerts are reserved for anything that risks bleeding into message delivery reliability, or for a detected privacy-filter failure (any exclusion-rate anomaly gets escalated immediately, given the sensitivity of the data involved) — both page an on-call engineer right away. Warning-level alerts cover rising consumer lag or degraded push latency, which affect feature quality but not correctness or privacy, and route to a ticket rather than a page. Dashboards track longer-term trends like coalescing ratio and connection growth for capacity planning, reviewed on a regular cadence rather than actively alerted.

11.2 Defining a Service-Level Objective for This Feature

Because read receipts are explicitly a secondary feature layered on top of message delivery, it’s worth defining their reliability targets separately and somewhat more leniently than the Message Service’s own targets — for example, a target that 99% of read-state updates reach an actively-connected sender within two seconds, with a defined, monitored error budget for the remainder, rather than holding this subsystem to the same strict five-nines availability bar as core message delivery. Having an explicit, separate objective avoids two opposite mistakes: over-investing engineering effort in read-receipt reliability at the expense of higher-priority work, or under-investing to the point that the feature feels broken often enough to genuinely frustrate users.

Independent deployability — and never terminating a live WebSocket the way you’d terminate a stateless HTTP node.

12.1 Independent Deployability

The Read Receipt Service, Batching Worker Pool, Privacy Settings Service, and WebSocket Gateway are deployed and versioned independently of the Message Service and of each other, consistent with the reliability principle of keeping read-receipt infrastructure changes from ever risking message-delivery infrastructure. A rollout of a new Batching Worker version can proceed, and be rolled back if needed, without touching message delivery at all.

12.2 Rolling Deployment for the WebSocket Gateway

Because the WebSocket Gateway holds long-lived, stateful connections, deploying a new version can’t simply terminate old instances outright — that would drop every connection those nodes were holding at once, causing a visible reconnection spike. Instead, new instances are brought up behind the Load Balancer, new connections are gradually routed to them, and old instances are drained — allowed to finish serving their existing connections naturally (or given a bounded grace period with a “please reconnect” signal sent to clients) — before being terminated, spreading the reconnection load out smoothly rather than all at once.

12.3 Infrastructure as Code and Repeatable Regional Rollout

As with the earlier multi-region deployment diagram, every region’s stack — Load Balancer, API Gateway, WebSocket Gateway cluster, Read Cursor Store replica — is defined declaratively so that expanding into a new geographic region is a matter of applying the same infrastructure template with region-specific parameters, rather than manual, error-prone recreation of configuration.

12.4 Connection Capacity Planning

Unlike typical stateless service capacity, which scales predictably with request rate, the WebSocket Gateway’s capacity is bounded by concurrent open connections, and that number can behave quite differently from request-rate-based traffic — a large fraction of a platform’s active user base might hold an open connection simultaneously during a peak usage window even if the actual message and read-event rate during that same window is comparatively modest. Capacity planning for this tier therefore tracks concurrent-connection trends specifically (daily and weekly peak concurrency, growth trajectory, and the added concurrency expected from planned feature launches or marketing pushes) as its own dedicated forecasting input, separate from the request-rate-based forecasting used for the stateless API Gateway and Read Receipt Service tiers.

One store built for very high-write simple key access, one cache for the hottest lookups, and two distinct load-balancing problems — not one.

13.1 Choosing the Read Cursor Store Technology

The access pattern here is simple key lookups and overwrites — get and set a single value keyed by (user ID, conversation ID) — with an extremely high write-to-read ratio during active conversations and a need for very low write latency. A wide-column or key-value store built for high write throughput and horizontal scalability (such as Cassandra or a managed equivalent like DynamoDB) fits this pattern well, in the same way it suits any workload dominated by simple key-based access rather than complex relational queries.

13.2 Cache Layer Design

An in-memory store (Redis, sharded and replicated) sits in front of the Read Cursor Store, caching the most recently accessed cursor values and presence lookups, since these are read extremely frequently — every time a sender’s client needs to display current read state, or a batching worker needs to check whether a target user is online.

13.3 Load Balancing Considerations

Two distinct load balancing problems exist. The Load Balancer at the edge distributes stateless HTTPS API traffic (cursor-update calls, privacy setting changes) using standard health-checked routing across the API Gateway fleet. WebSocket connections need connection-aware routing instead — once a client establishes a persistent connection to a specific WebSocket Gateway node, that mapping needs to persist for the connection’s lifetime, which is a different balancing problem than routing independent, short-lived requests, and is why the WebSocket Gateway relies on the Presence Service to route cross-node updates rather than expecting the Load Balancer to solve this alone.

13.4 Avoiding Hot Partitions

Sharding the Read Cursor Store by conversation ID works well for the overwhelming majority of conversations, but an extremely large, extremely active group (a very large broadcast-style channel, for instance) can still generate disproportionate write load concentrated on a single shard, since every member of that one conversation maps to the same partition key. Production systems address this with a secondary sharding dimension for known hot conversations — splitting a single very large conversation’s cursor writes across multiple sub-shards (for example, by hashing user ID within that conversation as a secondary key) — combined with monitoring that specifically watches for per-shard write-rate imbalance so an emerging hot partition can be identified and mitigated before it becomes a bottleneck for every other, unrelated conversation sharing that shard.

Internal service contracts and the code paths that actually enforce privacy on the server, not on the client.

14.1 Internal API Design

Internal calls (Read Receipt Service to Privacy Settings Service, Batching Worker to Presence Service) use gRPC for its lower serialization overhead and strongly typed contracts, which matters given how frequently these internal calls happen relative to external client-facing traffic.

// Core service contract (conceptual Java interface)
public interface ReadReceiptService {

    // Called when a client reads up to a given message in a conversation
    ReadCursorUpdateResult markRead(ReadCursorUpdateRequest request);

    // Called by a client to fetch current read state for a conversation
    ReadStateSnapshot getReadState(String conversationId, String requestingUserId);
}

public class ReadCursorUpdateRequest {
    private final String userId;
    private final String conversationId;
    private final String upToMessageId;
    private final long clientTimestamp;
}

public class ReadCursorUpdateResult {
    private final boolean accepted;
    private final boolean privacyRestricted; // true if this user has read
                                             // receipts disabled, so no
                                             // external event was emitted
}

14.2 Cursor Update Handler with Privacy Enforcement

public class DefaultReadReceiptService implements ReadReceiptService {

    private final CursorStore cursorStore;
    private final PrivacySettingsClient privacySettings;
    private final EventPublisher eventPublisher;

    @Override
    public ReadCursorUpdateResult markRead(ReadCursorUpdateRequest request) {
        // 1. Always update the user's own cursor -- needed for their own
        //    unread badge regardless of privacy setting
        cursorStore.upsert(request.getUserId(), request.getConversationId(),
                           request.getUpToMessageId());

        // 2. Check privacy setting before emitting anything externally visible
        boolean receiptsEnabled =
            privacySettings.isReadReceiptsEnabled(request.getUserId());

        if (!receiptsEnabled) {
            // Cursor is stored for the user's own experience, but no event
            // is published -- nothing for anyone else to ever see.
            return new ReadCursorUpdateResult(true, true);
        }

        // 3. Emit event for async, batched fan-out
        eventPublisher.publish(new ReadCursorUpdatedEvent(
            request.getUserId(),
            request.getConversationId(),
            request.getUpToMessageId(),
            System.currentTimeMillis()
        ));

        return new ReadCursorUpdateResult(true, false);
    }
}

14.3 Batching Worker with Privacy Filter and Coalescing

public class ReadEventBatchingWorker {

    private final PrivacySettingsClient privacySettings;
    private final PresenceClient presence;
    private final WebSocketPusher pusher;
    private final Duration batchWindow;

    // Called periodically per conversation with all events collected
    // during the current batching window
    public void processBatch(String conversationId,
                             List<ReadCursorUpdatedEvent> events) {

        // Deduplicate to each user's latest cursor position only
        Map<String, ReadCursorUpdatedEvent> latestPerUser =
            events.stream().collect(Collectors.toMap(
                ReadCursorUpdatedEvent::getUserId,
                e -> e,
                (a, b) -> a.getClientTimestamp() > b.getClientTimestamp() ? a : b
            ));

        // Filter out anyone who has since disabled read receipts
        List<ReadCursorUpdatedEvent> visibleEvents = latestPerUser.values()
            .stream()
            .filter(e -> privacySettings.isReadReceiptsEnabled(e.getUserId()))
            .collect(Collectors.toList());

        ReadSummary summary = ReadSummary.aggregate(conversationId, visibleEvents);

        // Push only to currently-connected, interested participants
        for (String memberId : summary.getInterestedMemberIds()) {
            PresenceInfo p = presence.lookup(memberId);
            if (p != null && p.isConnected()) {
                pusher.pushToNode(p.getGatewayNodeId(), memberId, summary);
            }
            // Offline members simply fetch current state next time they
            // open the conversation -- no push queued for them.
        }
    }
}

14.4 Presence Lookup and WebSocket Push Handler

public class PresenceServiceClient {

    private final CacheClient cache; // backed by the shared Redis cache layer

    public PresenceInfo lookup(String userId) {
        String raw = cache.get("presence:" + userId);
        if (raw == null) {
            return PresenceInfo.offline(); // no entry means not connected
        }
        return PresenceInfo.fromCacheValue(raw);
    }

    // Called by a WebSocket Gateway node on connect, and periodically
    // as a heartbeat while the connection stays open
    public void registerConnection(String userId, String gatewayNodeId) {
        PresenceInfo info = new PresenceInfo(true, gatewayNodeId,
                                             System.currentTimeMillis());
        cache.setWithTtl("presence:" + userId, info.toCacheValue(),
                         Duration.ofSeconds(45)); // short TTL; heartbeats
                                                  // refresh it, so a crashed
                                                  // node's stale entries
                                                  // expire on their own
    }
}

public class WebSocketPusher {

    private final Map<String, GatewayNodeClient> nodeClients; // one client
                                                              // per known
                                                              // gateway node

    public void pushToNode(String gatewayNodeId, String userId,
                           ReadSummary summary) {
        GatewayNodeClient client = nodeClients.get(gatewayNodeId);
        if (client == null) {
            log.warn("Unknown gateway node {}, dropping push for user {}",
                     gatewayNodeId, userId);
            return; // presence was stale; the client will simply pull
                    // current state on next reconnect or app foreground
        }
        client.sendToConnection(userId, summary.toWirePayload());
    }
}

14.5 Microservice Boundaries

Each service owns a narrow responsibility: the Privacy Settings Service can evolve its rules (adding, say, per-conversation overrides in the future) without any change to the Batching Worker’s coalescing logic; the Presence Service can be rearchitected or rescaled independently of the Read Cursor Store; the WebSocket Gateway fleet can be scaled purely based on connection count, entirely decoupled from cursor-store write load. This separation is what makes the strict reliability isolation from the Message Service practically enforceable — there is no shared code path or shared datastore that could let a read-receipt failure ripple into message delivery.

Patterns worth reusing, and mistakes worth naming so you can avoid them.

15.1 Patterns Used

15.2 Anti-Patterns to Avoid

Habits to build in, and the specific failure modes each one prevents.

16.1 Best Practices

• Treat privacy enforcement as a server-side, data-layer responsibility, checked once centrally before any fan-out, never delegated to client-side rendering logic.
• Keep the read-receipt subsystem fully separable from message delivery, in code, in deployment, and in storage, so it can degrade independently.
• Default to cursor-based tracking and reserve per-message granularity for the specific cases (small groups, or explicit “who has read this” detail views) that actually need it.
• Tune batching windows by conversation size and activity level, rather than using one fixed window for every conversation regardless of scale.
• Special-case very small groups in aggregate-count logic, since a naive “read by N of M” count can inadvertently reveal a specific individual’s read status when M is small.
• Make privacy setting changes take effect immediately, invalidating relevant caches rather than waiting on a normal TTL, since even a short window of incorrect exposure is a real trust issue for this kind of data.
• Define a separate, explicitly more lenient reliability target for read receipts than for message delivery, and monitor against it directly, rather than implicitly holding every subsystem to the same bar by default.
• Keep an auditable trail of privacy-setting changes and sensitive data access, distinct from the operational data itself, so the platform can demonstrate — not just assert — that its privacy guarantees hold up over time.
• Test privacy enforcement with dedicated, adversarial-style test cases, not just the happy path — explicitly verify that an opted-out user’s read events never appear in any outbound payload, that small-group aggregate counts never uniquely identify one person, and that a stale cached privacy setting can never linger long enough to leak data after a user disables the feature.

16.2 A Note on Testing This Kind of System

Because so much of what this system needs to get right is an absence — the absence of a leaked read status, the absence of a message-delivery slowdown caused by read-receipt load, the absence of a duplicate push — testing strategy here benefits from explicitly writing negative test cases alongside the usual positive ones. A typical positive test confirms that a read event correctly reaches an eligible recipient; a negative test confirms, with equal rigor, that the exact same event never reaches an opted-out recipient, never reaches a user who isn’t a member of the conversation, and never blocks or slows a concurrent message-send operation running through the entirely separate Message Service. Load and chaos testing — deliberately degrading the Read Cursor Store, the event stream, or the Presence Service in a staging environment and confirming that message delivery remains fully unaffected while only the read-receipt feature degrades — validates the core reliability-isolation principle this entire design is built around, rather than assuming the architecture diagram alone guarantees that isolation in practice.

16.3 Common Mistakes

The same skeleton reappears across mobile messaging, workplace chat, social platforms, and even email — a good sign it captures a reusable pattern.

17.1 Lessons Drawn From Real Deployments

Storage and write cost. A cursor requires one small, overwritable value per (user, conversation) pair, while per-message tracking requires a row per (message, recipient) pair — a number that grows without bound as message history accumulates, for information a single pointer already captures in the vast majority of real conversations.

Through batching: the Batching Worker Pool coalesces many individual read events arriving within a short window into a single summarized update per interested recipient, and pushes only a lightweight aggregate count rather than a full per-member list to everyone, reserving the detailed breakdown for on-demand fetch.

The Privacy Settings Service change takes effect immediately, with relevant caches invalidated right away rather than waiting for a TTL. From that point forward, this user’s read events are excluded from any external fan-out and from aggregate counts, and — per the symmetric fairness rule — this user also stops seeing other participants’ read status in every conversation, not just the one they were in when they changed the setting.

By design, no. The Message Service and Read Receipt Service are deliberately isolated at the code, deployment, and storage level specifically so that a read-receipt outage degrades only the read-receipt feature — the ticks simply stop updating — while message sending and receiving continue completely unaffected.

Each gateway node registers and heartbeats a connected user’s presence, including which specific node holds their connection, into a shared, low-latency presence registry. Any other component needing to push to that user looks up this registry first, rather than needing to broadcast to every gateway node and hope one of them has the connection.

Because most group members never open that detailed view, computing and pushing it to everyone on every update would waste the majority of that work. Pushing a lightweight aggregate count and computing the detailed list only on demand (fan-out on read) captures nearly all of the perceived responsiveness at a fraction of the cost.

By treating small conversations specially in the aggregation logic — suppressing exact counts that could only correspond to one identifiable person, and falling back to a simpler binary indicator, or treating them with the same conservative privacy handling used for one-on-one chats, rather than exposing a precise number derived from a small, easily-deduced population.

No — delivery is a network/infrastructure signal about whether a message reached a device, not a behavioral signal about whether a person looked at it, and most platforms treat it as outside the scope of the read-receipt privacy toggle. Only the transition to “read” is gated by the user’s privacy preference; delivery confirmation continues to function normally regardless of that setting.

The batching and fan-out machinery built for hundreds-of-members groups is not wasted on small groups — it still functions correctly — but the deanonymization safeguards discussed in the privacy section become the dominant design concern rather than raw fan-out volume, since small groups are exactly where naive aggregate counts most easily collapse into revealing one specific individual’s read status.

We designed a system that tracks delivery and read state for every message across one-on-one and large group conversations, pushes that state to an actively-watching sender within about a second, respects an explicit, symmetric privacy opt-out at every layer, and does all of this without letting read-receipt load or failures ever threaten the platform’s far more important guarantee: reliably delivering the messages themselves.

The techniques used here — cursor/checkpoint tracking, event-driven batching, presence-aware routing, and strict bulkhead isolation between a core guarantee and a layered-on feature — generalize well beyond read receipts. Typing indicators, live “last seen” status, and collaborative-editing presence cursors all face the same fundamental tension between real-time responsiveness, fan-out cost at scale, and user privacy, and tend to reach for this same architectural toolkit.

If there’s one idea to take away, it’s this: the hardest part of this system was never storing a timestamp — it was deciding, for every single read event, who genuinely needs to know about it right now, who can find out later if they ask, and who should never be told at all. Every architectural choice in this tutorial, from the batching window to the privacy filter’s placement, exists to answer that one question correctly at massive scale.

It’s also worth noting what this design deliberately does not try to be: a single unified “activity tracking” service handling read receipts, typing indicators, and presence all through one shared code path and datastore. Keeping read receipts as its own bounded, independently-reasoned-about subsystem — with its own explicit reliability target, its own privacy enforcement point, and its own clean separation from message delivery — is itself one of the most important architectural decisions in this entire tutorial, even though it never appears as a single box in any of the diagrams. Good system boundaries are often defined as much by what a component deliberately does not take on as by what it does.