Imagine you’re texting a friend, and your phone shows “sending…” for a few seconds too long. You panic and hit send again. A moment later, your friend gets your message twice. That’s a duplicate message — and in everyday texting it’s a mild annoyance. But now imagine that instead of a text message, it’s a bank transfer, an order confirmation, or a “charge this credit card” instruction flowing through a company’s backend systems. Suddenly, a duplicate isn’t annoying — it’s a bug that charges someone twice or ships two identical packages.

Message deduplication is the practice of detecting and eliminating duplicate copies of a message so that a piece of work is processed exactly once (or at least, appears to have been processed exactly once), even though the underlying system may have delivered it more than one time.

This concept isn’t new. It has roots going back to early telecommunication networks, where operators worried about the same telegram being retransmitted due to line noise. As computing moved into networked and distributed systems in the 1970s and 1980s, the same problem resurfaced in a new form: computers talking to each other over unreliable networks, where a message could get lost, delayed, or accidentally sent twice.

1.1 A Short Timeline of the Idea

• 1970sTCP and sequence numbers. TCP, the backbone protocol of the internet, introduced sequence numbers so that a receiving computer could detect and discard duplicate network packets — an early, low‑level form of deduplication.
• 1980s–90sDatabase transactions. Relational databases popularized idempotent operations and unique constraints, letting applications reject a second attempt to insert the “same” record.
• 2000sEnterprise messaging queues. Message brokers like IBM MQ and JMS‑based systems introduced “message IDs” so consumers could recognize a message they’d already seen.
• 2010sDistributed streaming platforms. Apache Kafka, Amazon SQS, and Google Pub/Sub popularized “at‑least‑once delivery,” which pushed the responsibility of deduplication into application design, and later into the platforms themselves (Kafka’s idempotent producer, SQS FIFO’s dedup ID).
• 2020sEvent‑driven microservices at scale. As companies broke monoliths into hundreds of microservices communicating via events, deduplication became a first‑class architectural concern, baked into API design (“idempotency keys”) and payment systems.

Today, message deduplication is a foundational building block anywhere messages, events, or requests travel across an unreliable network — which, in a distributed system, is basically everywhere. What started as a niche concern for telecom engineers is now something every backend developer runs into within their first few months on the job, whether they realize it by that name or not.

To understand why deduplication exists, you first need to understand why duplicates happen in the first place. The short answer: networks are unreliable, and systems that want to be reliable have to compensate by retrying. Retrying is exactly what creates duplicates.

2.1 Why Duplicates Happen

• Network timeouts: A client sends a request, the server processes it successfully, but the response gets lost on the way back. The client, having received no confirmation, assumes failure and retries — the server now does the same work twice.
• Producer retries: A message producer (like a service publishing an event) doesn’t get an acknowledgment from the broker in time, so it resends the message “just in case,” even though the broker actually received it the first time.
• Consumer crash before committing an offset: A consumer reads and processes a message, but crashes right before it tells the broker “I’m done with this one.” When it restarts, the broker — not knowing the message was processed — redelivers it.
• Load balancer / proxy retries: Infrastructure components sitting between services sometimes automatically retry failed calls without the application ever knowing.
• User‑driven duplication: A frustrated user double‑clicks “Submit” or “Pay Now,” firing off two nearly identical requests.
• Message broker redelivery guarantees: Many systems intentionally favor “at‑least‑once” delivery over “at‑most‑once,” because losing a message is usually worse than duplicating one — but this means duplicates are baked into the design by default.

2.2 The Core Motivation

The goal of deduplication is to give downstream systems the illusion of “exactly‑once processing,” even though the underlying transport only guarantees “at‑least‑once delivery.” Achieving true exactly‑once delivery at the network level is famously difficult (some say provably impossible in the general case), so instead, engineers build deduplication logic at the application or infrastructure layer to simulate that guarantee where it matters most.

Before diving deeper, let’s build a shared vocabulary. Each of these terms will come up again and again.

3.9 A Simple Analogy

“Deduplication isn’t about preventing duplicates from being sent — it’s about preventing duplicates from being acted upon.”

A deduplication system, whether it lives inside a message broker, an API gateway, or a custom service, is generally made of the same handful of building blocks.

4.1 The Building Blocks

• Producer — The component that creates and sends the message. It may attach a unique key to help downstream systems detect duplicates.
• Dedup Key Generator — Logic (often on the producer, sometimes on the client) that derives a stable, unique identifier for each logical message.
• Dedup Store — A fast‑lookup data store (in‑memory cache, database, or specialized structure) that remembers which keys have already been seen.
• Dedup Checker / Filter — The piece of logic that, upon receiving a message, checks the dedup store: “have I seen this key before?”
• Consumer / Processor — The component that does the actual business work — charging a card, updating inventory — only if the dedup check passes.
• Expiration / Cleanup Policy — A mechanism (TTL, LRU eviction, scheduled job) that removes old keys from the dedup store so it doesn’t grow forever.

Where this logic physically lives varies a lot. It might be baked into the message broker itself (like Kafka’s idempotent producer), enforced at the API layer (an Idempotency-Key HTTP header), or implemented entirely inside application code using a database’s unique constraint.

Let’s zoom into how a dedup checker actually decides whether a message is new or a repeat.

5.1 Step 1 — Generating a Good Deduplication Key

The entire system hinges on this key being both unique (no two different logical messages share it) and deterministic (the same logical message always produces the same key, even when retried).

Think of it this way: a dedup key is like a fingerprint. Two different people should never share a fingerprint (uniqueness), and the same person’s fingerprint should look identical every time you scan it (determinism). If your “fingerprinting” process is sloppy — say, it changes slightly depending on lighting or angle — you’d fail to recognize the same person twice, which is exactly what happens when a dedup key accidentally includes a value that changes between retries, like a fresh timestamp or a random trace ID generated fresh on each attempt.

Common key strategies:

• Client‑generated UUID: The producer generates a random UUID once and reuses it on every retry of the same logical operation.
• Content hash: Compute a hash (e.g. SHA‑256) of the message’s meaningful fields. Identical content produces an identical hash, so accidental duplicates with the same payload are caught even if no explicit ID was attached.
• Business key: Use a naturally unique identifier from the domain, like orderId + paymentAttempt or userId + eventType + timestamp.
• Broker‑assigned sequence number: Some brokers assign a producer ID and monotonically increasing sequence number per message, letting the broker itself detect duplicate sends without any application code.

5.2 Step 2 — Looking Up the Key

When a message arrives, the dedup checker looks up its key in the dedup store. This needs to be fast — usually a single indexed lookup — because it sits directly in the critical path of every message.

5.3 Step 3 — Deciding and Recording

If the key is not found, the message is genuinely new. The system records the key (often atomically, in the same operation as processing) and proceeds. If the key is found, the message is a duplicate — the system discards it, or simply re‑sends the previously computed result without redoing the work.

5.4 A Minimal Java Example

Here’s a simplified example using a database’s unique constraint to achieve atomic, race‑condition‑free deduplication:

public class PaymentProcessor {

    private final JdbcTemplate jdbc;

    public PaymentProcessor(JdbcTemplate jdbc) {
        this.jdbc = jdbc;
    }

    // idempotencyKey is generated once by the client and reused on every retry
    public PaymentResult processPayment(String idempotencyKey, PaymentRequest request) {
        try {
            // Atomic insert - fails with a constraint violation if the key already exists
            jdbc.update(
                "INSERT INTO processed_payments (idempotency_key, order_id, amount, created_at) " +
                "VALUES (?, ?, ?, NOW())",
                idempotencyKey, request.getOrderId(), request.getAmount()
            );
        } catch (DuplicateKeyException e) {
            // Someone already processed this exact request - return the cached result
            return fetchExistingResult(idempotencyKey);
        }

        // Only reached if the insert succeeded - this is genuinely the first attempt
        PaymentResult result = chargeCard(request);
        saveResult(idempotencyKey, result);
        return result;
    }
}

The key insight: the database’s UNIQUE constraint on idempotency_key does the heavy lifting. It turns “check, then act” into a single atomic operation, because the database itself refuses to let two rows share the same key — no matter how many threads or servers race to insert at the same time.

5.5 An In‑Memory Alternative Using an Atomic Cache Command

Not every system has a relational database sitting in the write path. When speed matters more than transactional guarantees, an in‑memory store like Redis offers an equivalent atomic primitive: SET key value NX EX ttl, which sets a key only if it does not already exist (“NX” — not exists), and automatically expires it after a given number of seconds. This single command does the “check and record” step atomically, without a separate read‑then‑write round trip.

public class RedisDedupChecker {

    private final StatefulRedisConnection<String, String> connection;
    private static final long WINDOW_SECONDS = 60 * 60 * 24; // 24-hour dedup window

    public RedisDedupChecker(StatefulRedisConnection<String, String> connection) {
        this.connection = connection;
    }

    // Returns true if this is the FIRST time we've seen this key
    public boolean markIfNew(String dedupKey) {
        RedisCommands<String, String> cmd = connection.sync();
        SetArgs args = SetArgs.Builder.nx().ex(WINDOW_SECONDS);
        String result = cmd.set(dedupKey, "seen", args);
        return "OK".equals(result); // null means the key already existed
    }
}

Notice how the expiration (ex) is baked directly into the same atomic call that records the key — there’s no separate cleanup job needed for this store, because Redis itself evicts the key once the window passes. This is one reason in‑memory caches with native TTL support are such a popular choice for high‑throughput deduplication.

5.6 Concurrency Considerations

Even with an atomic check‑and‑record primitive, concurrency still deserves careful thought. If two threads on the same server race to process the same key microseconds apart, the atomic operation guarantees only one of them “wins” and proceeds to do the actual work — but the losing thread needs a sensible fallback path, usually waiting briefly and then reading the winner’s cached result, rather than simply erroring out. This waiting behavior avoids a subtle problem: if thread B loses the race but the result from thread A isn’t written yet, thread B might return an empty or partial response to its caller. A short retry‑with‑backoff loop when reading the cached result handles this cleanly in practice.

Let’s trace the full lifecycle of a message from creation to (possibly duplicate) delivery, and see exactly where deduplication intervenes.

6.1 The Lifecycle Stages

1. Key assignment: The client or producer attaches a dedup key before the message is ever sent.
2. Transmission: The message travels across the network, possibly retried due to timeouts.
3. Interception: The dedup layer (gateway, broker, or consumer) receives the message and extracts the key.
4. Lookup: The dedup store is checked for that key.
5. Branch — new vs. duplicate: New messages proceed to processing; duplicates are short‑circuited.
6. Recording: The key (and often the result) is stored, atomically with processing.
7. Expiration: After the dedup window passes, the key is evicted to keep the store bounded in size.

7.1 Key Trade‑off — Strictness vs. Cost

A dedup window of 7 days is safer than one of 5 minutes, but it’s also far more expensive to store and search. Systems typically size the window to match the realistic worst‑case retry delay in their environment — for example, “our mobile clients might retry up to 24 hours later if the device was offline,” so a 48‑hour window with margin is chosen.

Because the dedup check sits in the hot path of every single message, its performance directly determines the throughput ceiling of the whole system.

8.1 Where the Bottlenecks Live

• Lookup latency: Every message pays the cost of a lookup. An in‑memory cache (like Redis) might add sub‑millisecond latency; a relational database with a unique index might add a few milliseconds; a poorly indexed table can add tens of milliseconds and become the system’s bottleneck.
• Write contention: Under high concurrency, many messages might race to write keys at once, causing lock contention on hot database pages or cache shards.
• Store growth: As the number of remembered keys grows, lookups can slow down unless the underlying data structure (hash index, B‑tree, bloom filter) scales well with size.

8.2 Scaling Techniques

A Bloom filter deserves a beginner‑friendly explanation since it comes up so often here. Imagine a giant checklist made of light switches. To “remember” a key, you flip a few specific switches (determined by hashing the key) to ON. To check if a key was seen before, you check if all its corresponding switches are ON. If even one is OFF, you know for certain the key was never added. If all are ON, it’s probably seen — but rarely, by coincidence, a totally different key can flip on the same switches, causing a false positive. This trade‑off is usually worth it: Bloom filters use a tiny fraction of the memory a full key list would need, and you only fall back to a slower, precise check when the Bloom filter says “maybe.”

8.3 Data Structure Choices, Compared

Underneath every dedup store sits a fundamental data‑structure decision, and it’s worth understanding the trade‑offs the way a computer science course would present them:

• Hash table (hash map): Average O(1) lookup and insert. The workhorse behind Redis and most in‑memory dedup stores. Its main weakness is memory — every key (often 16–36 bytes for a UUID) plus overhead is stored in full.
• B‑tree index (relational databases): O(log n) lookup, but with strong ordering guarantees, which is why databases use it for unique constraints — it’s slightly slower than a pure hash table but supports range queries and integrates naturally with transactions.
• Bloom filter: O(k) lookup where k is the small, fixed number of hash functions used (commonly 3–7) — essentially constant time, and dramatically more memory‑efficient than storing full keys, at the cost of a tunable false‑positive rate.
• Cuckoo filter: A newer alternative to Bloom filters that additionally supports deleting individual keys (something classic Bloom filters can’t do safely), useful when you want to actively evict specific keys rather than relying purely on TTL‑based expiry.

Most large‑scale systems layer these: a Bloom or Cuckoo filter as a cheap first‑pass “definitely not a duplicate” check, backed by a precise hash table or database for the final confirmation. This two‑tier approach lets the system handle enormous volumes of genuinely new messages extremely cheaply, reserving the more expensive precise check for the rare cases that actually need it.

If the dedup store is a single point of failure, the entire safety guarantee collapses the moment it goes down. Production deduplication systems need to survive node failures without silently losing their memory of “what’s already been processed.”

9.1 Common Reliability Strategies

• Replication: The dedup store’s data is copied across multiple nodes (e.g. a Redis cluster with replicas, or a database with synchronous replication), so a single node failure doesn’t wipe out the memory of recent keys.
• Persistence: Even fast in‑memory caches usually write to disk (e.g. Redis AOF / RDB snapshots) so that a full restart doesn’t forget recent keys.
• Failover: Automated promotion of a replica to primary when the leader fails, minimizing the window where dedup checks might be unavailable.
• Graceful degradation policy: Deciding in advance what happens if the dedup store is temporarily unreachable — do you reject new messages (safe but reduces availability), or let them through unchecked (available but risks a duplicate)? This is a genuine business decision, not just a technical one.

9.2 Disaster Recovery and Backup

Because the dedup store’s whole job is to “remember,” losing it entirely — through a catastrophic outage, a bad deployment, or an accidental data wipe — has a very specific consequence: any in‑flight retries that arrive after the loss will look brand new, and duplicates can slip through undetected until someone notices downstream (for example, a finance team spotting doubled transaction volume). This is different from most caches, where losing the data just costs you a bit of extra latency while it repopulates.

For this reason, production systems treat the dedup store’s backup and recovery plan seriously:

• Point‑in‑time snapshots: Regularly persisted snapshots of the dedup store allow recovery to a recent, known‑good state rather than starting from a completely empty memory.
• Cross‑region replication: For globally distributed systems, replicating the dedup store to a secondary region means a regional outage doesn’t erase the system’s memory of recent operations.
• Reconciliation jobs: Even with good backups, many teams run a periodic batch job that cross‑checks business records (like a payments ledger) for signs of duplicate processing, acting as a safety net underneath the real‑time dedup layer.

Deduplication touches sensitive operations — payments, account changes, order processing — so it has its own security considerations.

• Key predictability: If dedup keys are easy to guess (like sequential integers), an attacker might be able to “poison” the dedup store by pre‑inserting a key, causing a legitimate future request with that same key to be silently dropped as a “duplicate.” Use cryptographically random or sufficiently unpredictable keys.
• Key scoping: A dedup key should typically be scoped per‑user or per‑tenant (e.g. userId + idempotencyKey), so one user’s key can’t collide with — or be spoofed to intercept — another user’s request.
• Replay‑attack confusion: Deduplication is not a substitute for authentication. An attacker replaying a captured, validly‑signed request isn’t stopped by deduplication logic designed for accidental retries — proper request signing, timestamps, and nonce validation are still needed for real replay‑attack protection.
• Sensitive data in the dedup store: If cached results (not just keys) are stored for fast duplicate responses, make sure that store is encrypted and access‑controlled just like any other data store holding sensitive information.
• Resource exhaustion: An attacker could try to flood the dedup store with a huge number of unique fake keys, aiming to exhaust memory or slow down lookups for real traffic — rate limiting and quotas help mitigate this.

You can’t tell if your deduplication system is working — or silently failing — without visibility into it.

Good logging practice: whenever a message is identified as a duplicate, log the dedup key, the original processing timestamp, and the retry timestamp. This turns “duplicate detected” from an invisible internal event into a traceable, auditable fact — extremely useful when investigating a customer complaint like “why does my order history show this ordered twice, but I was only charged once?”

11.1 Tracing a Request Across the Dedup Boundary

In a microservices environment, it’s worth propagating a consistent trace ID (separate from the dedup key itself) through every hop of a request, including duplicate ones. This way, when an engineer looks at a distributed trace during an incident, they can see clearly: “attempt 1 timed out waiting on the payment gateway, attempt 2 arrived 4 seconds later and was correctly short‑circuited by the dedup layer, returning the cached result from attempt 1.” Without this level of tracing, duplicate‑related incidents are notoriously hard to reconstruct after the fact, because by the time someone investigates, the retry and the original request look identical in the logs unless the dedup key and trace ID are both captured.

Where you deploy your dedup store, and how it’s configured, has real consequences.

12.1 Common Deployment Patterns

• Managed in‑memory cache: Cloud‑managed Redis (AWS ElastiCache, GCP Memorystore) is a popular choice — fast, replicated, and someone else handles the operational burden.
• Native broker support: Many managed queues have deduplication built in, so you don’t need to run a separate store at all — for example, Amazon SQS FIFO queues support a MessageDeduplicationId with a built‑in 5‑minute dedup interval.
• Database unique constraints: If you already have a relational database in the request path (e.g. writing an order row), a unique constraint on the idempotency key column is often the simplest, most transactionally‑consistent option — no extra infrastructure needed.
• Multi‑region considerations: If your system spans regions, the dedup store either needs to be globally consistent (harder, slower) or you accept that dedup only works reliably within a region, and design your key routing so retries of the same logical message always land in the same region.

The choice of underlying storage for the dedup layer is one of the most consequential design decisions.

13.1 Load Balancing Interplay

When a system is horizontally scaled behind a load balancer, retries of “the same” logical request might land on different backend instances. This is exactly why the dedup store usually can’t just be an in‑memory map on a single server — it needs to be centrally accessible (or consistently sharded) so that no matter which instance handles the retry, it sees the same memory of “have I processed this before?” Consistent hashing is often used to route a given key reliably to the same shard, minimizing cross‑node lookups.

In a microservices world, deduplication commonly shows up at the API boundary using an idempotency key pattern — a convention popularized by payment APIs like Stripe.

14.1 The Idempotency‑Key HTTP Pattern

POST /v1/charges HTTP/1.1
Host: api.example.com
Idempotency-Key: 8f14e45f-ceea-4d59-8f7b-2f5c9c3f4a11
Content-Type: application/json

{
  "amount": 5000,
  "currency": "usd",
  "customer": "cus_9s6XKzkNRiz8i3"
}

The client generates the Idempotency-Key once, and reuses the exact same value on every retry attempt of this logical charge. The server:

1. Looks up the key in its dedup store.
2. If found, returns the previously computed response verbatim (same HTTP status, same body) without redoing the charge.
3. If not found, processes the request, stores the key and response together, then returns the response.

14.2 Java Example — A Spring‑Based Idempotency Filter

@Component
public class IdempotencyFilter extends OncePerRequestFilter {

    private final IdempotencyStore store; // backed by Redis or a DB table

    public IdempotencyFilter(IdempotencyStore store) {
        this.store = store;
    }

    @Override
    protected void doFilterInternal(HttpServletRequest req,
                                     HttpServletResponse res,
                                     FilterChain chain) throws IOException, ServletException {

        String key = req.getHeader("Idempotency-Key");
        if (key == null || !"POST".equalsIgnoreCase(req.getMethod())) {
            chain.doFilter(req, res); // no key, or safe method - nothing to dedup
            return;
        }

        Optional<CachedResponse> cached = store.get(key);
        if (cached.isPresent()) {
            writeCachedResponse(res, cached.get()); // short-circuit, no reprocessing
            return;
        }

        // Wrap response so we can capture it after downstream processing
        ContentCachingResponseWrapper wrapped = new ContentCachingResponseWrapper(res);
        chain.doFilter(req, wrapped);

        store.saveIfAbsent(key, new CachedResponse(wrapped.getStatus(), wrapped.getContentAsByteArray()));
        wrapped.copyBodyToResponse();
    }
}

14.3 What Should the Client Do While Waiting?

A subtlety that trips up a lot of client implementations: what should happen if a retry with the same idempotency key arrives while the original request is still being processed, not after it finished? A naive dedup check might find no existing record yet (because the first attempt hasn’t finished), and mistakenly let the second attempt through as if it were new — causing the exact duplicate the whole system was built to prevent. Robust implementations solve this by recording an “in‑progress” marker for the key as soon as processing starts, before the actual business logic runs. A concurrent retry that finds an “in‑progress” marker can then wait briefly and poll for the final result, or return a distinct “still processing, please retry shortly” response, rather than racing ahead and duplicating the work.

14.4 Microservice Event Dedup

In event‑driven microservices, each service that consumes an event should treat that event’s ID as a dedup key — even if the message broker already tries to guarantee “exactly‑once” delivery, defense in depth at the consumer level protects against edge cases like consumer restarts mid‑processing.

15.1 Good Patterns

• Idempotent consumer pattern: Design the consumer’s business logic itself to be naturally idempotent (e.g. “set inventory to X” instead of “decrement inventory by 1”), reducing reliance on a separate dedup layer entirely.
• Outbox pattern: Combine deduplication with the transactional outbox pattern to ensure a message is published exactly once relative to a database transaction, reducing duplicate publishing at the source.
• Idempotency key as part of the API contract: Requiring clients to supply their own key (rather than the server generating one) puts control in the hands of whoever knows if a retry is “the same” logical operation.
• Check‑record atomicity: Always perform the “check if seen” and “record as seen” as a single atomic operation.

15.2 Anti‑Patterns to Avoid

Across all these examples, the same underlying idea repeats: give every logical operation a stable identity, remember which identities have already been handled, and treat that memory as a first‑class, monitored piece of infrastructure — not an afterthought.

17.1 A Closer Look — How Kafka’s Idempotent Producer Works Internally

It’s worth walking through one example in more depth, because it illustrates several concepts from earlier sections working together. When a Kafka producer enables idempotence, the broker assigns it a unique Producer ID (PID) when it first connects. From that point on, every message the producer sends to a given partition is tagged with a monotonically increasing sequence number, starting at zero.

When the broker receives a message, it checks the sequence number against the last one it accepted from that same PID and partition. If the incoming sequence number is exactly one greater than the last accepted one, it’s genuinely new, and the broker accepts and stores it. If the sequence number is less than or equal to the last accepted one, the broker recognizes this as a duplicate — most likely caused by the producer not receiving an acknowledgment and retrying — and it simply re‑acknowledges the message without writing it a second time. This is a beautiful example of deduplication achieved through ordering and monotonic counters rather than an explicit key lookup, and it works because Kafka can guarantee message order within a single partition.

The important limitation to internalize: this only protects the producer‑to‑broker hop. If a consumer reads that message, starts processing it, and crashes before recording that it finished, Kafka’s idempotent producer feature does nothing to stop the consumer from re‑reading and reprocessing the same message after restart — that’s a completely separate problem, solved by consumer‑side idempotency or an application‑level dedup table, exactly as described earlier in this guide.

Message deduplication exists because distributed systems favor reliable, retry‑friendly delivery over perfectly‑once delivery — and retries inevitably create duplicates. Rather than trying to eliminate duplicates at the network level (which is extraordinarily hard), well‑designed systems accept that duplicates will happen and build a dedicated layer to recognize and neutralize them before they cause real damage.

19.1 Key Takeaways

• 01Duplicates are a natural consequence of retries in unreliable networks — not a rare edge case, but an expected, everyday occurrence at scale.
• 02A good deduplication key must be unique, deterministic, and generated before the very first attempt — ideally by the client.
• 03The “check, then record” step must be atomic to avoid race conditions letting duplicates slip through.
• 04The dedup store itself needs to be fast, reliable, and monitored — it’s critical infrastructure, not an afterthought.
• 05Sizing the dedup window is a genuine trade‑off between safety and storage cost.
• 06Designing naturally idempotent operations often reduces how much you need to lean on deduplication infrastructure at all.
• 07Real‑world systems — from Stripe’s Idempotency‑Key header to Kafka’s idempotent producer to SQS FIFO queues — all converge on the same core idea: remember what you’ve already done, and don’t do it twice.
• 08Defense in depth wins: broker‑level dedup + consumer‑level dedup + reconciliation jobs together produce systems that keep working even when individual layers occasionally fail.

“Give every logical operation a stable identity, remember which identities you’ve already handled, and treat that memory as a first‑class, monitored piece of infrastructure — not an afterthought.”