Change Data Capture Patterns with Debezium in Kafka

Change Data Capture Patterns with Debezium in Kafka

A complete, plain-English guide to turning every insert, update, and delete in your database into a reliable stream of events: how Debezium reads database logs, how Kafka carries the changes, and the proven patterns that use them.

Imagine a bank branch that keeps its official record in a big paper ledger. Every time a customer deposits or withdraws money, a clerk writes a new line in the ledger. Now imagine that the fraud team, the mobile app, the statement printer, and the accounting office all need to know about each new line the moment it appears. One way is to have each of them walk to the ledger every few minutes and read the whole thing again. That is slow, tiring, and they still miss changes that happened and were corrected between visits.

A much better way is to place a quiet observer beside the clerk, who copies down each new line the instant it is written and sends a note to everyone who cares. That observer is the idea behind Change Data Capture, usually shortened to CDC. In the world of modern software, the ledger is a database, the lines are changes to rows, the notes travel through Apache Kafka, and one of the most popular observers is an open-source tool called Debezium.

This guide starts from zero. Every new word is explained the first time it appears, every big idea comes with an everyday comparison, and each chapter ends with something you can act on. You will learn how the capture works inside, the patterns teams build with it, the traps that catch beginners, and how to run it safely in production.

1The Problem: Databases Are Islands

Almost every business keeps its most important facts in a database. The trouble starts when other systems need those facts too.

A database is a program that stores data safely and lets other programs read and change it. An online store, for example, keeps customers, products, and orders in one. That works well for the store’s own application. But soon other systems want the same information. A search page needs fresh product data. A recommendation engine needs every order. A dashboard needs live sales numbers. A warehouse system needs to know when to ship.

Each of these wants to know two things: what the data looks like now, and what just changed. Databases are very good at the first and, traditionally, quite poor at the second.

Analogy

A database is like a busy town hall that keeps every citizen’s records in locked cabinets. Anyone can ask a clerk for a file. But there is no town crier announcing “a new house was registered on Maple Street just now”. If you want news, you must keep asking, and the clerks grow tired of the same question.

The old answers, and why they hurt

Old way

Nightly batch copies

Copy the whole table every night. Simple, but data is up to a day old, and each copy puts heavy load on the database.

Old way

Polling for changes

Ask “anything new since last time?” every few seconds. It misses deletes and burns database effort even when nothing changed.

Old way

Dual writes

The application writes to the database and then also sends a message. If one step fails, the two disagree, silently and permanently.

Old way

Database triggers

Small programs inside the database record changes into extra tables. They slow down every write and are hard to maintain.

!
The dual write trap

Writing to a database and to a message system in two separate steps can never be made fully safe. If the program crashes between the steps, one has the change and the other does not. CDC and the outbox pattern, both covered later, exist largely to remove this trap.

What we want is a way to learn about every change, in order, without slowing the database, without missing anything, and without asking the application to do extra work. That wish list is exactly what log-based CDC delivers.

2Change Data Capture in Plain Words

CDC is a simple idea with a few very different ways to build it. Knowing the differences helps you see why Debezium chose the approach it did.

Change Data Capture means detecting every change made to data in a database, and turning each change into a message that other systems can use. An insert becomes a “row created” message. An update becomes a “row changed” message that says what it was and what it became. A delete becomes a “row removed” message.

Three ways to capture changes

ApproachHow it worksStrengthWeakness
Query-basedRepeatedly select rows changed since last check, using a timestamp or version columnSimple; no special database featuresMisses deletes and quick changes; adds query load; needs extra columns
Trigger-basedDatabase triggers write each change into a shadow tableCaptures deletes; works on many databasesSlows writes; adds schema clutter; fragile
Log-basedRead the database’s own internal record of changesComplete, ordered, low impact, includes deletesNeeds database configuration and privileges; depends on database internals

Why the log is special

Nearly every serious database writes each change to an internal file before it does anything else. This file is called the transaction log. Different databases call it different things: the binary log in MySQL, the write-ahead log in PostgreSQL, the redo log in Oracle. The database uses it for its own safety, to recover after a crash and to copy changes to replicas. It already holds every committed change, in the exact order they happened.

Log-based CDC simply reads that file. The database does no extra work for the observer, the application does not change at all, and nothing is missed. Debezium is a log-based CDC tool.

Analogy

Think of a restaurant kitchen. Every order the waiters write goes onto a ticket rail, and the cooks work from it. If you want to know every dish sold today, you do not interrupt the cooks to ask. You read the tickets on the rail. The tickets are the transaction log.

Complete
inserts, updates, and deletes, all captured
Ordered
changes arrive in commit order
Low impact
no extra queries against your tables

3Kafka and Kafka Connect: The Road and the Trucks

Debezium does not work alone. It rides on two other pieces, and understanding them makes everything that follows clearer.

Kafka in one minute

Apache Kafka is a system that stores and moves streams of events. Events are written to named channels called topics. A topic is split into partitions, which are ordered logs. Programs that write are producers. Programs that read are consumers, and they remember how far they have read by keeping an offset, a bookmark. Kafka keeps the events for as long as you configure, so many consumers can read the same events at their own pace, and new consumers can go back and read from the start.

That last point is what makes Kafka a natural home for database changes. A stream of changes can be read by the search team, the analytics team, and the fraud team, each independently, each at its own speed.

Kafka Connect in one minute

Kafka Connect is a framework that runs ready-made programs called connectors to move data between Kafka and other systems. A source connector brings data into Kafka. A sink connector sends data from Kafka out to another system. Connect handles the boring parts: running the connectors, restarting them after failures, storing their progress, and spreading the work over several machines.

Analogy

Kafka is a highway with many lanes. Kafka Connect is a fleet of trucking companies. A source connector is a truck that picks up goods at a factory and drives them onto the highway. A sink connector is a truck that takes goods off the highway and delivers them to a shop. Debezium provides very clever trucks for one kind of factory: databases.

Piece

Worker

A running Connect process. Several workers together form a Connect cluster.

Piece

Connector

The configured job, such as “capture changes from this database”.

Piece

Task

The unit of work a connector splits into. Debezium connectors normally use a single task, because one database log is a single ordered story.

Piece

Converter

Turns records into bytes for Kafka, using a format such as JSON, Avro, or Protobuf.

Piece

Transform

A small step that reshapes each record on the way through. Called a single message transform, or SMT.

Piece

Offset storage

Where Connect saves the connector’s progress, so it can resume after a restart. It is kept in a Kafka topic.

4What Debezium Is, and How You Can Run It

Debezium is a family of tools around one job: turning database changes into event streams.

Debezium is an open-source platform for change data capture. Its main product is a set of Kafka Connect source connectors, one for each supported database. There are connectors for MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Db2, and several others. Each knows how to read that database’s own log format and translate it into a common event shape.

Three ways to run it

1

On Kafka Connect

The most common way. You install the Debezium connector into a Connect cluster and configure it. Events flow into Kafka topics, and you gain everything Connect offers: scaling, restarts, and shared management.

2

Debezium Server

A standalone program that runs the connectors without Kafka Connect and can send changes to other systems, such as cloud messaging services, instead of Kafka. Useful when Kafka is not part of the picture.

3

Debezium Engine

A library that embeds the capture logic inside your own application. It gives the most control and the least infrastructure, but you must handle progress storage and restarts yourself.

This guide focuses on the first way, Debezium on Kafka Connect, because it is what most teams use with Kafka.

Why teams choose it

Reason

No application changes

Existing programs keep writing to their database as before. Capture is added from the outside.

Reason

Consistent event shape

Different databases produce the same style of events, so downstream code looks alike.

Reason

Snapshots and streaming together

It first reads existing data, then follows live changes, with no gap between the two.

Reason

Open source and widely used

A large community, many connectors, and plenty of production experience.

i
Good to know

Debezium reports what changed in the database. It does not know why. A price update is just “price went from 10 to 12”. If you need business meaning, such as “a discount was applied”, you must add it yourself, and the outbox pattern in a later chapter shows how.

5Architecture: The Pieces and How They Connect

Study the picture first. Then read the roles below it.

flowchart LR
  subgraph App["Application Side"]
    A["Application"] -->|"writes rows"| DB[("Source Database")]
    DB --> LOG["Transaction Log"]
  end
  subgraph Connect["Kafka Connect Cluster"]
    W1["Worker 1 with Debezium Connector"]
    W2["Worker 2 standby"]
    OFF["Offsets and Status Topics"]
  end
  subgraph K["Kafka Cluster"]
    T1["Topic: orders"]
    T2["Topic: customers"]
    SH["Schema History Topic"]
  end
  subgraph Consumers["Consumers"]
    C1["Search Indexer"]
    C2["Data Warehouse Sink"]
    C3["Microservice"]
  end
  LOG -->|"reads changes"| W1
  W1 --> T1
  W1 --> T2
  W1 --> SH
  W1 --- OFF
  W2 --- OFF
  T1 --> C1
  T1 --> C2
  T2 --> C3
    
Figure 1: The connector reads the database log, writes one topic per table, and records its progress in Kafka.

Reading the diagram

The application on the left writes to the database as usual. The database appends each change to its transaction log. The Debezium connector, running inside a Kafka Connect worker, reads that log like a very patient reader following a long scroll. For each change it creates an event and writes it to a Kafka topic, usually one topic per table. Consumers on the right read the topics and do whatever they need to do.

Component roles

Database

Transaction log

The database’s own record of changes. Debezium reads it but never writes to it.

Connect

Debezium connector

Reads the log, converts changes to events, and sends them to Kafka.

Connect

Offset topic

Holds the connector’s bookmark in the log, so it resumes exactly where it stopped.

Kafka

Table topics

One topic per captured table, holding that table’s change events.

Kafka

Schema history topic

For some databases, a record of table structure changes, so old log entries can be understood later.

Downstream

Consumers and sinks

The programs that use the events: indexers, warehouses, caches, and services.

i
Key idea

The application and the consumers never talk to each other. The database log and Kafka sit between them. That loose coupling is what lets you add a new consumer next month without touching the application.

6Reading the Log: How Each Database Tells Its Story

Every database keeps its change record differently. Debezium hides the differences, but knowing them explains many setup steps and surprises.

DatabaseWhere changes come fromBookmark typeSetup highlights
MySQL and MariaDBThe binary log (binlog)Log file and position, or a global transaction IDRow-based binlog format; a unique server ID; replication privileges
PostgreSQLWrite-ahead log through logical decodingLog sequence number (LSN)Logical replication level; a replication slot; a publication
SQL ServerChange tables filled by SQL Server’s own CDC featureCommit and change LSNCDC enabled on the database and each table; agent running
OracleRedo log via LogMiner or another readerSystem change number (SCN)Archive log mode; supplemental logging; special user rights
MongoDBChange streams built on the operation logResume tokenReplica set or sharded cluster

The common pattern

Despite the differences, the setup follows the same three steps for every database. First, turn on the database feature that keeps enough detail in the log. Second, create a dedicated user with just the rights the connector needs. Third, tell the connector where the database is, which tables to watch, and a name to use for the topics.

Analogy

Each database writes its diary in a different language and handwriting. The Debezium connector is a skilled translator who reads any of them and always writes the summary in the same standard format, so the readers downstream never need to learn each diary’s style.

!
Watch out

The log is not kept forever. Databases delete old log files to save disk. If a connector is stopped for longer than the log is kept, the entries it needs may be gone. Then the only way back is to take a fresh snapshot. Chapter after chapter, you will see this same theme: the log has a shelf life.

7Anatomy of a Change Event

Everything downstream depends on what a change event contains. Learn the shape once, and every pattern becomes easier to read.

Each change Debezium captures becomes one Kafka message with two main parts: a key and a value. The key identifies the row that changed, using the table’s primary key columns. The value is a wrapper called the envelope, which describes the change itself.

What is in the envelope

FieldWhat it holdsWhy it matters
beforeThe row as it was before the changeLets you see what was overwritten; empty for inserts
afterThe row as it is after the changeThe new state; empty for deletes
opThe kind of change: create, update, delete, or readTells consumers how to react
sourceWhere the change came from: database, table, log position, transaction, timeTraceability and ordering
ts_msWhen the connector processed the eventMeasuring delay
transactionOptional details tying the event to its database transactionRebuilding groups of related changes

The four kinds of operation

Create

A new row appeared

The “before” part is empty. The “after” part shows the new row.

Update

A row changed

Both parts are present. Comparing them shows exactly which columns changed.

Delete

A row was removed

The “before” part shows the removed row. The “after” part is empty.

Read

A row seen during a snapshot

Marks rows copied during the initial snapshot, not real changes at that moment.

Analogy

Picture a courier’s delivery slip. The label on the outside is the key: which parcel this is about. Inside the envelope is a form with “what it looked like before”, “what it looks like now”, the type of change, and a stamp saying which warehouse and shift handled it. Anyone can read the form and know exactly what happened.

Deletes and tombstones

When a row is deleted, Debezium normally sends two messages. The first is a delete event that describes the removed row. The second is a tombstone, a message with the same key and an empty value. Kafka uses tombstones in a feature called log compaction: when compaction runs, a tombstone tells Kafka that all older messages for that key can be thrown away. Without tombstones, deleted rows would live forever in a compacted topic.

Column types and precision

Databases have types with no perfect twin in messaging formats, such as exact decimals, dates with time zones, and large binary values. Debezium offers settings to choose how they arrive, for example decimals as exact numbers, as text, or as floating point numbers. Choose carefully for money. Floating point can quietly round values, while exact or text forms preserve every digit.

!
Watch out

An update event may not carry the full previous row unless the database is set up to record it. On PostgreSQL, for example, the “before” part is thin by default. If your pattern needs it, change the table’s replica identity setting before you start.

8Snapshots: Catching Up on What Already Exists

The log only holds recent changes. To capture a table that already has millions of rows, Debezium first takes a snapshot.

When a connector starts for the first time, it faces a puzzle. Your tables already contain data, but the log only remembers changes from a certain point onward. A new consumer that only sees future changes would be missing the whole starting picture. So Debezium takes an initial snapshot: it reads the current rows of every captured table and emits them as read events. Then it switches to reading the log from exactly the point where the snapshot began, so nothing falls between the two.

Analogy

You join a group chat halfway through the year. To catch up, you first read a summary of everything so far, which is the snapshot. From then on you read every new message as it arrives, which is streaming. The handover between summary and live messages must have no gaps or overlaps.

Snapshot modes

ModeWhat it doesWhen you would use it
InitialSnapshot data and structure on first start, then streamThe usual choice for a new connector
Initial onlySnapshot and stop, with no streamingA one-time data copy
Schema onlyCapture table structure but not existing rows; stream new changesYou only care about changes from now on
When neededSnapshot again if the saved position is gone from the logAutomatic recovery for a connector that fell far behind
NeverNo snapshot; start streaming from the log’s oldest available pointRare; when you manage the starting state yourself

The problem with big snapshots

A classic snapshot reads all tables in one go. For huge tables this can take hours, and if it fails near the end, it starts again. During that time the connector produces no live changes, so consumers see nothing new. Some databases also hold locks or long-running transactions during a snapshot, which can affect other work.

Incremental snapshots

Debezium solved this with incremental snapshots. Instead of pausing streaming to copy everything, it copies a table in small chunks, interleaved with live log reading. A special signal, sent through a small control table or a Kafka topic, starts a chunked snapshot of chosen tables at any time. The connector uses a clever marking technique so that if a live change touches a row inside a chunk, the newer change wins and old copies are dropped.

flowchart TD
  S["Signal: snapshot table X"] --> C1["Open chunk window"]
  C1 --> R["Read next chunk of rows by primary key"]
  R --> L["Keep reading live log changes in parallel"]
  L --> D{"Live change touches a row inside the chunk?"}
  D -- "Yes" --> DROP["Drop the older copy, keep live change"]
  D -- "No" --> KEEP["Keep chunk row"]
  DROP --> EMIT["Emit chunk to topic"]
  KEEP --> EMIT
  EMIT --> MORE{"More chunks left?"}
  MORE -- "Yes" --> R
  MORE -- "No" --> DONE["Snapshot of table complete"]
    
Figure 2: An incremental snapshot copies small chunks while live streaming continues, and live changes always win.
Benefit

Streaming never stops

Live consumers keep getting fresh changes while old data is being copied.

Benefit

Resumable

A failure loses at most one chunk, not the whole snapshot.

Benefit

On demand

Re-snapshot one table after a bug or a new consumer, without restarting anything.

Requirement

Primary keys

Chunking works by walking primary key ranges, so tables should have a stable primary key.

9Internal Working: From Commit to Kafka, Step by Step

Follow one order from the moment an application saves it to the moment a consumer reads it.

sequenceDiagram
  autonumber
  participant App as Application
  participant DB as Source Database
  participant Dbz as Debezium Connector
  participant Conn as Kafka Connect
  participant K as Kafka Topic
  participant Con as Consumer
  App->>DB: Insert order row and commit
  DB->>DB: Append change to transaction log
  Dbz->>DB: Stream log entries from saved position
  DB-->>Dbz: Committed change for the orders table
  Dbz->>Dbz: Decode row, build envelope, choose key
  Dbz->>Conn: Hand over record with source offset
  Conn->>K: Produce record to topic
  K-->>Conn: Acknowledged
  Conn->>Conn: Save source offset to offsets topic
  Con->>K: Poll for new records
  K-->>Con: Order change event
    
Figure 3: The change travels from the application through the database log, the connector, and Kafka to the consumer.

Step 1 and 2: the write and the log

The application saves an order and commits the transaction. Before the database considers the commit final, it has already appended the change to its transaction log. This is the same step the database always performs for its own safety. Debezium adds nothing here.

Step 3 to 5: reading and decoding

The connector holds a connection to the database that behaves like a replica asking for the latest log entries. It receives entries in commit order. For each committed change it decodes the raw bytes, looks up the table structure, builds the envelope with before, after, and source details, and picks the key from the primary key columns.

Step 6 to 8: handing to Kafka

The connector passes the record to the Connect framework together with a note saying which log position it came from. Connect sends the record to Kafka and waits for confirmation. Only after Kafka has confirmed does Connect save the position to its offsets topic. That order is very important: the bookmark only moves after the data is safe.

Step 9 to 10: consumption

Any consumer subscribed to the orders topic reads the event on its own schedule and does its job. The database and the application are not involved at all.

i
Performance note

Because Debezium only reads a file the database is already writing, adding capture to a database usually costs a small amount of extra disk reading and network. Compare that to polling, where every check is a new query competing with your application.

10Offsets, Restarts, and Delivery Guarantees

What happens when the connector crashes at the worst possible moment? The answer shapes how you must write every consumer.

The connector’s bookmark is called an offset. It records the position in the database log up to which changes have been safely sent. If the connector crashes and restarts, it reads its last saved offset and asks the database to continue from there.

The gap that causes duplicates

Look again at the order of steps: send to Kafka, then save the bookmark. Suppose the connector crashes after Kafka confirmed a batch but before the bookmark was saved. On restart, the connector resumes from the older bookmark and sends those changes again. Consumers see the same events twice. This is called at-least-once delivery: nothing is ever lost, but a few events may repeat after a failure.

Analogy

A courier delivers a parcel, then goes to tick it off on the list. If the courier faints just after delivering and before ticking, the replacement courier sees the parcel is not ticked and delivers it again. Better twice than never. The receiver’s answer is to notice “I already have this parcel”.

Making consumers safe

Technique

Idempotent writes

Design the consumer so applying the same event twice gives the same result. Upserts keyed by primary key are naturally safe.

Technique

Deduplication by position

Each event carries its log position. A consumer can remember the highest position it has applied and skip older ones.

Technique

Deduplication by identifier

For outbox events, a unique event ID lets consumers ignore repeats.

Technique

Transactional sinks

Write the data and the consumer’s own progress together in one transaction, so both happen or neither does.

What about exactly-once?

Newer versions of Kafka Connect include support for exactly-once delivery for source connectors, and some Debezium connectors can use it. It reduces duplicates from the connector side but does not remove the need for careful consumers, because they can still crash and reprocess. Treat it as a helpful extra, and check the documentation for your connector and version before depending on it.

When the saved position is gone

If the connector was down so long that the database has already deleted the log section it needs, the bookmark points to nothing. Depending on the snapshot mode you chose, the connector either fails loudly or takes a new snapshot. Failing loudly is safer for data you cannot afford to duplicate or reorder, but it needs a human to decide what to do.

11Topics, Keys, Partitions, and Ordering

Where events land in Kafka decides what order consumers see them in. Get this right and many bugs never appear.

One topic per table

By default, each captured table gets its own topic, named from a prefix you choose, the schema, and the table. For example, a connector with the prefix “shop” watching the orders table in the sales schema writes to a topic named like shop.sales.orders. Consumers subscribe only to the tables they care about.

The key decides the partition

The event key is the row’s primary key. Kafka uses the key to choose the partition: the same key always goes to the same partition. Since a partition is an ordered log, all changes to one row are delivered in the order they happened. That is the ordering promise you can rely on.

i
The ordering promise

Changes to the same row arrive in order. Changes to different rows, or in different tables, may arrive in a different order than they were committed. Design consumers around that rule.

Why cross-table order is not guaranteed

An order and its order lines live in two tables, so they land in two topics, each with its own partitions. A consumer reading both topics may see a line before its parent order. To handle this, consumers should tolerate arrival in any order, or use the transaction details in the event to group related changes. A later chapter shows options.

Log compaction: keeping only the latest

A topic can be set to keep just the newest event for each key, discarding older ones. This is called log compaction. For a CDC topic it means the topic becomes a compact copy of the table’s current state: one event per row. New consumers can read it from the start to rebuild the full table quickly. Tombstones make deleted rows disappear eventually.

Retention styleTopic holdsBest for
Time-basedAll changes for a periodStreaming to live consumers; short-term replay
CompactedThe latest change for every keyRebuilding current state; cache and search loading
Compacted plus timeLatest per key, but not foreverBalance of state recovery and storage cost
Long retentionFull history for a long timeAudit trails and replaying history

The special history topic

Some connectors, such as MySQL, also write table structure changes to a separate schema history topic. It must be a single partition and must keep its data forever, because the connector reads it on restart to understand old log entries. Losing it is serious. Protect it with a high replication factor and no automatic deletion, and never share it between connectors.

12The Pattern Map: What Teams Build with CDC

Before diving into each pattern, here is the whole map in one place. Every pattern answers a different question about moving data.

PatternQuestion it answersTypical consumers
Replicate to a read storeHow do I keep a search index or cache fresh?Search engines, caches
OutboxHow do I publish events safely from my own service?Other microservices
Service data sync and read modelsHow do services share data without sharing databases?Query services, reporting services
Strangler migrationHow do I move from an old system to a new one safely?New system, comparison tools
Audit and historyHow do I keep a record of every change?Audit stores, compliance tools
Analytics feedHow do I feed a warehouse or lake with fresh data?Warehouses, lakes, dashboards
Analogy

Think of CDC as a tap on a water main. What you do with the water is up to you: fill a tank, water a garden, or feed a fountain. The tap is the same. The patterns are the different things people build downstream.

13Pattern 1: Keeping Search Indexes and Caches Fresh

The most popular CDC use is also the easiest to understand: keep a second copy of data, shaped for fast reading, in step with the database.

Databases are built to store and protect data. They are not always the best at searching text or serving millions of quick lookups. So teams copy data into a search engine for full-text queries, or into a cache for lightning-fast reads. The challenge is keeping the copy up to date.

The old way and its pain

Without CDC, the application updates the database and also tries to update the index or cache. Two writes, two chances to fail. When one fails, the search page shows a product that no longer exists, or a price that changed yesterday. Nightly rebuilds patch over the problem but leave hours of wrong data.

The CDC way

1

Application writes only to the database

One write, one source of truth. The application knows nothing about the index.

2

Debezium publishes the change

The change lands on the product topic within a short time.

3

A sink applies it to the index or cache

Creates and updates become upserts. Deletes become removals.

4

The index catches up

Typically within seconds. If it falls behind, it simply reads faster to catch up.

Analogy

A library keeps its books on shelves, which is the database. The card catalogue in the lobby is the search index. Instead of the librarian rewriting cards from memory, a helper watches every book that arrives or leaves and updates the cards automatically. The catalogue always agrees with the shelves.

Details that matter

Detail

Flatten the envelope

Most sinks want the “after” row only, not the full envelope. A standard transform unwraps it for them.

Detail

Handle deletes

Make sure the sink turns delete events into removals, or the index will show ghosts.

Detail

Shape the document

A search document often joins several tables. Build it with a stream processor, or consume several topics and merge.

Detail

Plan a rebuild

Use compacted topics or a fresh snapshot to rebuild the index from scratch whenever mappings change.

i
Tip

Cache invalidation is famously hard. With CDC, you can either update the cached value from the “after” row, or simply delete the cache entry when a change arrives and let the next read refill it. The second option is simpler and avoids stale writes.

14Pattern 2: The Outbox, Publishing Events Without Dual Writes

This is the pattern that solves the dual write trap from the first chapter. It is the most valuable idea in this guide.

A microservice often needs to do two things at once: save data in its own database and announce that something happened to other services. For instance, an order service saves a new order and tells the shipping service. Doing both separately can fail halfway. The outbox pattern fixes this by making the announcement part of the database transaction.

flowchart LR
  subgraph Svc["Order Service"]
    API["Business logic"] --> TX["One database transaction"]
  end
  TX -->|"writes order row"| ORD[("orders table")]
  TX -->|"writes event row"| OUT[("outbox table")]
  OUT --> LOG["Transaction log"]
  LOG --> DBZ["Debezium with Outbox Event Router"]
  DBZ --> T1["Topic: order events"]
  DBZ --> T2["Topic: payment events"]
  T1 --> SHIP["Shipping Service"]
  T1 --> MAIL["Notification Service"]
  T2 --> BILL["Billing Service"]
    
Figure 4: The service writes its data and its event in one transaction. Debezium then publishes the event row.

How it works

1

Add an outbox table

A small table in the service’s own database, with columns for a unique event ID, the type of thing it is about, that thing’s ID, the event type, and the event details.

2

Write both in one transaction

When the service saves an order, it also inserts a row into the outbox table, in the same transaction. Either both happen or neither does. The dual write problem is gone.

3

Debezium captures the outbox inserts

It watches only the outbox table.

4

A transform routes the event

Debezium ships a ready-made transform, the outbox event router. It reads the columns, sends the event to a topic chosen from the row, uses the aggregate ID as the key, and puts the event details in the message body.

5

Other services consume

They see clean business events like “order placed”, not raw table changes.

Analogy

Imagine you write a letter and want to post it, but your mailbox might be closed. Instead, you place the letter in your own outgoing tray together with the ledger entry in a single motion. A reliable postal worker then collects everything from the tray and delivers it. Your action and the letter cannot separate, and the delivery happens eventually.

Why teams like it

Benefit

Atomic

The event exists if and only if the business change was saved.

Benefit

Business meaning

Events are designed by the service, not leaked from its table layout. Table changes stay private.

Benefit

Stable contract

The service can reshape its tables freely while keeping events steady.

Benefit

Ordered per entity

Using the aggregate ID as key keeps events for one order in sequence.

Housekeeping and duplicates

Outbox rows pile up. Debezium reads them from the log, so the service can delete each row right after inserting it, in the same transaction. The insert is still recorded in the log, but the table stays tiny. Since delivery is at-least-once, consumers may see an event twice, so each event carries a unique ID that consumers use to ignore repeats.

!
Watch out

Do not confuse the outbox with simply capturing your business tables and publishing raw row changes to other teams. That exposes internal table design as a public contract, and every column rename becomes a breaking change for others.

15Pattern 3: Sharing Data Between Services and Building Read Models

Microservices should own their data. But services still need each other’s facts. CDC lets them share without sharing a database.

In a well-designed system of small services, each service has its own private database. The customer service owns customers. The order service owns orders. But the order screen needs to show the customer’s name. Calling the customer service on every request is slow and makes the order service fail whenever the customer service is down.

Local copies from events

With CDC, the order service keeps its own small local copy of the customer fields it needs, such as name and shipping city. It fills and updates this copy by listening to the customer topic. Now showing an order needs no call to the customer service at all. The copy may lag by a moment, but it is always available.

Analogy

Instead of phoning head office every time you need a customer’s address, each branch keeps its own address book and updates it whenever head office sends a change notice. Branches keep working even when the phone line to head office is down.

Read models and CQRS

A read model is a copy of data shaped exactly for one screen or report. The idea of separating writing models from reading models is often called CQRS, short for command query responsibility segregation. CDC provides the feed: changes from several write databases are combined by a stream processor into one read table that answers a specific question quickly, like “show every order with its customer and delivery status”.

ChoiceGood forCost
Call the other service liveData that must be perfectly currentSlower, and fails when the other service fails
Keep a local copy from CDCData that can be a few seconds oldMore storage; must handle late or out-of-order updates
Build a joined read modelScreens and reports needing data from many servicesA stream processing job to maintain

Expose events, not tables

Use the outbox pattern for data the owning service wants to share as a contract, and use raw table capture only within a team or for non-critical copies. A raw table capture couples other teams to your table layout.

Example: the shipping cost calculator

The shipping service needs the customer’s country and the product’s weight to compute costs. Instead of calling two services, it keeps small local tables filled from the customer and product topics. Quotes now appear in a few milliseconds, and continue to work even during a customer service outage.

16Pattern 4: Migrating Gradually with the Strangler Approach

Replacing a big old system in one weekend is a famous way to fail. CDC lets you move piece by piece while both systems stay in step.

The strangler fig approach is named after a vine that slowly grows around a tree until it replaces it. In software, you build new components beside the old system and move traffic across gradually, until the old system has nothing left to do.

1

Capture the old system’s database

Debezium streams every change from the legacy database into Kafka, without modifying legacy code.

2

Feed the new system

The new service consumes the changes and builds its own data store. It now mirrors the old system’s data, live.

3

Compare and verify

Run both side by side. Compare results to find bugs before any user depends on the new system.

4

Shift reads first

Send a small share of read traffic to the new system, then more.

5

Shift writes carefully

Move writes to the new system, and reverse the flow so the old database receives changes from the new one until it is switched off.

Analogy

Moving house room by room, while the old house still has the lights on, is easier than throwing everything on a truck in one night. You can go back for a forgotten item, and you can live in both places while you settle in.

The same idea helps with database upgrades and moves between database products or cloud providers. The old and new databases stay in sync through the stream, you switch when ready, and you keep the ability to switch back.

!
Watch out

Two-way sync can create loops, where a change flows from A to B and back to A forever. Mark each change with its origin and refuse to send it back where it came from.

17Pattern 5: Audit Trails and Change History

Databases forget. When a row is updated, the old value is gone. CDC remembers what the database forgets.

Many businesses must answer questions like “who changed this price, and what was it before?” The database only holds the current state. Because every event has both a before and an after, a CDC stream is a natural audit trail. Store the events in a durable place, and you can replay the full history of any row.

Use

Compliance records

Keep an unchangeable list of every change, for regulators or internal review.

Use

Debugging

Reconstruct exactly how a record reached its current, surprising state.

Use

Time travel

Rebuild what the data looked like at a past moment.

Use

Anomaly detection

Watch for suspicious patterns, like many changes to sensitive fields.

Who did it?

The log knows what changed but not who changed it, because the database only sees the application’s shared login. To capture the human, have the application store the user in a column such as “last updated by”, or add context to the transaction. Debezium can then include it in the event. Some setups also write a separate metadata row per transaction with user details.

Where to keep the history

A Kafka topic with long retention works for shorter horizons. For years of history, sink the events into cheap object storage or an append-only table, and partition by date. Protect it with strict access, since it holds every past value, including ones the main database deleted on purpose.

i
Privacy note

A history that keeps everything can clash with rules about deleting personal data. Plan how you will honor deletion requests in the audit store, for example by encrypting each person’s data with a key you can destroy.

18Pattern 6: Feeding Warehouses and Data Lakes

Analytics teams have long lived on yesterday’s data. CDC brings them minutes-old data without hammering the source database.

A data warehouse is a system built to answer big analytical questions over lots of history. A data lake stores raw files cheaply and lets many tools read them. Both traditionally received data through nightly bulk extracts. Those extracts read entire tables, load the source database heavily, and deliver stale numbers.

Streaming replaces the nightly copy

With CDC, the warehouse receives a steady trickle of changes. A sink connector writes them into the warehouse or lake as they come, in small batches. The source database sees no extra queries, and dashboards are minutes old instead of a day old.

The delete and update challenge

Analytical storage is often append-only. It is happy to add rows but reluctant to change or remove them. To keep an accurate current picture, there are two common approaches.

Append everything, resolve later

  • Store each change event as a new row, with its operation and time
  • Build a “latest state” view that picks the newest row per key and hides deleted ones
  • Keeps full history for free
  • Simple and fast to write

Merge into a current table

  • Apply updates and deletes so the table always shows the current state
  • Queries are simple and cheap
  • Requires a storage format that supports row-level changes
  • History is lost unless kept separately

Many teams do both: keep the raw change log as a permanent record, and maintain a merged current-state table on top of it.

Analogy

A shop keeps every receipt in a box, which is the change log, and also keeps a tidy stock sheet that shows how many items are on the shelf right now. The receipts prove the sheet is right, and the sheet saves everyone from counting receipts.

19Single Message Transforms: Reshaping Events on the Way Out

The raw envelope is rich but heavy. Small transforms let you trim, route, and filter events before they reach Kafka.

A single message transform, or SMT, is a small step that Kafka Connect applies to each record, one at a time, as it passes through. You can chain several. They are ideal for light reshaping, but they are not meant for heavy work like joining streams. For that, use a stream processor.

Shape

Unwrap the record

Reduces the envelope to just the new row state, which is what most sinks and simple consumers expect. It can add chosen metadata fields back in, such as the operation type or table name, and it can turn deletes into a flag instead of a missing record.

Route

Outbox event router

Sends outbox rows to topics chosen from their contents, and shapes them into clean business events.

Route

Table router

Merges topics from many similar tables into one topic, for example monthly or sharded tables that share a structure.

Filter

Event filtering

Drops events you do not need, such as changes to internal columns or test data, using conditions on the content.

Protect

Column handling

Removes, masks, or shortens sensitive columns. Some of this is built into the connector itself.

Enrich

Add fields

Adds timestamps, constant labels, or a copy of the key into the value.

When to unwrap, and when not to

Unwrapping is convenient, but it throws away the before state and some source details. If a consumer needs to know what a value was before, or needs the exact log position, keep the full envelope. A common design is to keep raw envelopes in the main topics for careful consumers, and let simple sinks apply the unwrap step on their own side.

Analogy

SMTs are like the small adjustments a mail room does to each letter: put it in the right pigeonhole, remove a wrapper, stamp a label. They are quick, one letter at a time. Sorting a hundred letters into a report is a different job for a different department.

!
Watch out

Each SMT runs on every event, so slow or complex ones hurt throughput. Scripting-based transforms are flexible but heavier. Keep chains short, and test with realistic volume.

20Schemas, Serialization, and Surviving Change

Tables change. Columns are added, renamed, and dropped. Your event streams must survive that without breaking every consumer.

Every change event has a shape, called a schema: which fields exist and what type each is. Consumers rely on that shape. When the source table changes, the shape of new events changes too. Managing that gracefully is called schema evolution.

Choosing a format

FormatSizeSchema handlingNote
JSON with embedded schemaLargeEach message carries its schemaEasy to read; wasteful at scale
JSON without schemaMediumNone enforcedSimple; risky as things change
Avro with a registrySmallSchema stored once in a registry; messages hold an IDPopular; strong compatibility checks
Protobuf with a registrySmallSame idea as AvroGood for teams already using Protobuf

A schema registry is a service that stores every version of every schema and can refuse a new version that would break existing consumers. For production CDC, a compact binary format plus a registry is the usual choice.

How table changes appear

1

Add a column

Usually safe. New events include the field. Old consumers that ignore unknown fields keep working. Give the column a default so compatibility rules are satisfied.

2

Drop a column

The field disappears from new events. Consumers that expected it may break. Announce and phase these changes in.

3

Rename a column

From the log’s view, this looks like one column vanishing and another appearing. It is a breaking change for consumers. Prefer adding the new column, copying data, then dropping the old one later.

4

Change a type

The riskiest change. A widening change, like a small number type to a bigger one, is often fine. Anything else usually needs a planned migration.

The compatibility rulebook

Decide a compatibility mode with your registry. Backward compatibility means new readers can read old data. Forward compatibility means old readers can read new data. Full means both. For CDC topics that consumers replay from the start, backward compatibility is the minimum, because new consumer versions must read events written long ago.

i
Tip

Treat database schema changes like changes to a public interface. Write down who consumes each table, and give them notice before any structural change.

!
Watch out

Some databases let structural changes slip through in ways the connector cannot fully follow, and connector versions differ in what they support. Test your typical changes in a staging environment with a running connector, before doing them in production.

21PostgreSQL Specifics: Slots, Publications, and the Disk Full Risk

PostgreSQL is a favorite for CDC, and it has one famous hazard. Understanding it can save you a midnight outage.

PostgreSQL exposes changes through logical decoding, which turns its internal write-ahead log into readable row changes. Debezium connects using a replication slot, a named marker inside the database that says “this reader has consumed the log up to here, so keep everything after it”. A publication lists which tables are shared with that reader.

Key setup points

Setting

Logical WAL level

The database must be set to record enough detail for logical decoding. Changing it needs a restart, so plan ahead.

Setting

A decoding plugin

Modern setups use the built-in plugin, so nothing extra needs installing on the database server.

Setting

Replica identity

Controls how much of the old row is written for updates and deletes. The default gives only the key. Set it to full for tables where you need the whole “before” row.

Setting

Dedicated user

A user with replication rights and read access to just the captured tables.

The slot that fills the disk

Because the slot promises to keep all log data after its position, a slot whose reader has stopped forces PostgreSQL to hoard log files without limit. Disk space shrinks steadily until the database itself stops. This is the most common serious CDC incident on PostgreSQL.

Analogy

A newspaper delivery stops when you go on holiday, but the newsagent has promised to keep every issue for you. Each day the pile grows. After a month there is no room in the shop for anything else. The promise was kind, but it needs a limit.

1

Monitor slot lag

Watch how far behind each slot is, in bytes of log retained. Alert early.

2

Cap retained log

Newer PostgreSQL versions can limit how much log a slot may hold, at the price of invalidating a slot that falls too far behind.

3

Drop unused slots

Deleting a connector does not always delete its slot. Remove orphans by hand.

4

Use heartbeats

On quiet databases, or when only a few tables are captured, the slot may not advance even though the database is busy elsewhere. A heartbeat setting writes small periodic changes so the slot keeps moving.

Other gotchas

Very large column values stored out of line, known as TOAST values, are omitted from update events when they did not change. Debezium then inserts a placeholder, and consumers must not overwrite real data with it. Failover is another concern: slots have traditionally lived only on the primary server, so after a failover to a replica the slot may be missing. Newer versions add ways to keep slots in step on standby servers, but plan and test your failover carefully.

22MySQL and Others: Binlogs, Server IDs, and Friends

Each database has its own quirks. Here are the ones that trip up teams most often.

MySQL and MariaDB

The MySQL connector reads the binary log. It needs the log in row format with full row images, so each entry holds the complete changed row. The connector poses as a replica, so it needs a server ID unique among all replicas and readers, or two readers will kick each other off. The database user needs replication and read rights.

MySQL

Binlog retention

Binary logs expire after a set number of days. Keep retention comfortably longer than the longest connector outage you can imagine.

MySQL

Global transaction IDs

When enabled, the connector can reconnect to a different server after a failover, since positions are the same everywhere.

MySQL

Schema history topic

Required. Records table structure changes so the connector can read old entries. Never lose it.

MySQL

Read from a replica

Capturing from a replica reduces load on the primary, but check that it records the full binlog and replicates all changes.

SQL Server

SQL Server has its own change data capture feature, which copies changes into special change tables using an agent job. Debezium reads those tables. The feature must be turned on for the database and each table. Because of the extra step through change tables, there is a small extra delay and extra storage. Cleanup jobs delete old change table rows, so make sure they do not remove entries before the connector reads them.

Oracle

Oracle capture reads the redo log, most often via LogMiner. It needs archive logging on, supplemental logging turned on for the captured tables, and a carefully privileged user. Oracle has licensing and performance considerations, and very high volumes may need a different reader. Plan Oracle projects with extra care and read the connector guide closely.

MongoDB

MongoDB capture uses change streams built on its operation log, and needs a replica set or a sharded cluster. Events describe documents rather than rows, and updates can be delivered as changes or as whole documents depending on settings.

DatabaseBiggest hazardGuardrail
PostgreSQLSlot causing the disk to fillSlot lag alerts; retention cap; heartbeats
MySQLBinlog expiring before the connector catches upGenerous retention; lag alerts
SQL ServerCleanup job removing changes too earlyTune cleanup retention; monitor lag
OracleLog mining load and complexityCareful sizing; dedicated mining settings
MongoDBOplog rolling overLarge oplog window; monitor lag

23Transactions and Ordering Across Tables

A database transaction changes several rows at once, all or nothing. In Kafka, those changes scatter across topics. Here is how to put the story back together.

When an order is placed, one transaction might insert an order row, three order line rows, and update a stock count. In the log these are one atomic unit. In Kafka they become five separate events on three topics with no guaranteed arrival order. A consumer that reads them independently may see a line without its order, or a stock change before the order it belongs to.

Three ways to cope

Option 1

Tolerate any order

Design consumers so a missing parent is fine for a moment. Store the child, and fill the link when the parent arrives. Simple and robust.

Option 2

Use transaction metadata

Debezium can emit begin and end markers on a special topic, and tag each event with its transaction ID and position inside it. A consumer can wait until the end marker before acting on the group.

Option 3

Send one aggregate

Use the outbox pattern to publish one event holding the whole order, so there is nothing to reassemble.

Analogy

A moving crew unloads one household into several rooms. The bed may arrive upstairs before the mattress does. You can either accept things show up in any order and tidy up at the end, or ask the crew foreman to announce “all boxes for this house are now delivered”.

The single-partition option

If you must have a strict global order, you can route related tables into one topic with one partition. This preserves order but limits throughput to one consumer, so use it only for small, critical flows.

24Performance and Scalability

A single log is a single story, and a single story is read by one reader. That shapes how CDC scales.

One task per connector

The transaction log is one ordered sequence, so most Debezium connectors run a single task to read it in order. You cannot speed up one connector by adding tasks. This surprises people who are used to Connect connectors scaling out.

How to scale anyway

1

Capture only what you need

List the tables and columns explicitly. Reading and decoding less data is the cheapest speed-up there is.

2

Split by table groups

Run several connectors, each watching a different set of tables. Each has its own bookmark, its own replication slot or server ID, and its own topics. Ordering within a table stays intact.

3

Tune the internal queue

The connector has a queue between reading and sending, with settings for batch size and queue size. Larger values raise throughput and use more memory.

4

Tune Kafka producing

Batching, compression, and acknowledgment settings on the Connect side affect how fast events leave.

5

Scale consumers by partition

Give table topics enough partitions for the parallelism you need downstream. Remember that per-key order holds within a partition.

Things that slow capture down

Cause

Huge transactions

A bulk update touching millions of rows creates millions of events in one transaction. The connector may hold them in memory until commit.

Cause

Wide rows and big values

Large text or binary columns inflate every event.

Cause

Slow Kafka or slow converters

If sending is the bottleneck, reading faster only fills the queue.

Cause

Heavy transforms

Scripted transforms and large chains cost time on every event.

A quick sizing sense

Suppose a database sees 2,000 row changes per second at peak, and each event is about 1 KB with the envelope. That is roughly 2 MB per second, about 170 GB per day before compression, well within what a single connector can handle. Storage in Kafka then depends on retention. With three copies and seven days, the topics hold roughly 3.5 terabytes at that rate uncompressed, so compression and compaction matter.

i
Tip

Bulk data loads and big maintenance updates can flood the stream with events that nobody needs. Coordinate with the teams that run them, or filter those changes out, to avoid an accidental surge.

25High Availability and Failure Scenarios

Everything fails eventually. What matters is what happens next, and whether you lose data, get duplicates, or simply pause.

What keeps running, and what does not

A CDC pipeline has many parts, and each can fail on its own. The good news is that the design is forgiving: the log is durable, the bookmark is stored in Kafka, and Kafka itself is replicated. Most failures mean a pause and a catch-up, not lost data.

flowchart TD
  F["Something fails"] --> Q1{"Which part?"}
  Q1 -->|"Connect worker"| W["Another worker takes over the connector"]
  W --> W2["Resume from saved offset, possible duplicates"]
  Q1 -->|"Kafka broker"| K["Replicas take over leadership"]
  K --> K2["Producer retries, brief delay"]
  Q1 -->|"Source database down"| D["Connector retries connection"]
  D --> D2["Resume from same position when database returns"]
  Q1 -->|"Database failover"| FO{"Same log positions on new primary?"}
  FO -->|"Yes, with global IDs"| FO1["Reconnect and continue"]
  FO -->|"No, slot or position lost"| FO2["Re-snapshot or manual repair"]
  Q1 -->|"Log already purged"| P["Saved position no longer exists"]
  P --> P2["Take new snapshot and rebuild consumers"]
    
Figure 5: How the pipeline recovers from each kind of failure, and which ones need human help.

Scenario walk-throughs

A Connect worker crashes

The Connect cluster notices and hands the connector to another worker, which reads the saved offset and continues. A short pause and a few repeated events are the only signs. Run at least two workers so a spare is always waiting.

The database is restarted for maintenance

The connector loses its connection and retries with waits in between. When the database returns, it resumes from its bookmark. No action needed, but set the retry limits high enough for planned maintenance windows.

The connector is down for a weekend

On Monday, the connector reads all the changes it missed. If the database kept its log for longer than the outage, everything is fine. If not, the position is gone, and a new snapshot is needed. This is why log retention should exceed your worst outage.

The database fails over to a replica

With MySQL global transaction IDs, the connector can find the same position on the new primary. With PostgreSQL, the replication slot may not exist on the new server, so a fresh slot and possibly a snapshot are needed. Rehearse this before it happens for real.

The schema history topic is damaged

For connectors that need it, this is serious, because the connector cannot interpret old log entries. Recovery options exist that rebuild the structure history from the database, but they need care. Prevent the problem with high replication and no deletion.

Design for the gap

Assume the pipeline will sometimes pause. Downstream systems should treat the data as “eventually current” and show or use the age of the data when it matters. Provide a way to rebuild any consumer from a snapshot or a compacted topic, so recovery is a routine action instead of an emergency.

26Security and Privacy: CDC Sees Everything

By design, CDC copies every change to every captured table into a place where many people can read it. That deserves real care.

A CDC stream holds complete rows, including names, emails, and possibly financial or health details. Kafka topics also last longer than a single query would. Treat every CDC topic as if it were the database itself.

Protect the door into the database

Rule

Least privilege user

A dedicated user with only the rights the connector needs, and read access only to the captured tables.

Rule

Encrypted connections

Use TLS between the connector and the database, and between the connector and Kafka.

Rule

Secrets kept outside

Passwords should come from a secrets manager through a configuration provider, not from plain text in a shared connector file.

Rule

Restricted admin access

Whoever can change connector settings can point it at any table. Limit that power.

Protect the data in the stream

1

Capture fewer columns

Exclude columns nobody needs. Data never captured cannot leak.

2

Mask or shorten sensitive values

The connector can replace values with fixed characters, or cut long text down, before the event is even created.

3

Encrypt fields where needed

Sensitive fields can be encrypted before they reach Kafka, with keys held by the owners. Community transforms and client libraries can help.

4

Lock down topics

Use Kafka access rules so only approved consumers can read each topic. Raw table topics deserve stricter access than curated business event topics.

5

Plan for deletion requests

Privacy laws may require erasing a person’s data. Compacted topics with tombstones and sinks that honor deletes help, but backups and history stores need their own plan.

Analogy

A CDC stream is a live photocopier attached to your filing cabinet. It is wonderfully handy, and it means every secret in the cabinet can appear in a second place. You would not leave the copies on a public table.

!
Watch out

The schema history topic and the offsets topic are also sensitive. The history topic can contain the text of structural statements, and the offsets topic reveals what is being captured. Restrict both.

27Monitoring, Logging, and Metrics

The worst CDC failure is a silent one. The connector looks healthy while data quietly stops flowing. Good monitoring catches that first.

The metrics that matter

SignalWhat it tells youAlert when
Milliseconds behind sourceHow far the connector lags the databaseIt grows steadily or passes your freshness target
Time since last eventWhether events are still flowingLong silence on a busy database
Connector and task statusRunning, paused, or failedAnything other than running
Queue remaining capacityWhether the internal queue is fillingIt stays near empty capacity
Snapshot progressRows copied and time takenSnapshot much slower than expected
Events seen and filteredVolume and how much is droppedSudden jumps or drops
Database log retainedHow much log the database keeps for the readerGrowth toward disk limits
Consumer lag downstreamHow far consumers trail the topicsLag grows without recovering

The end-to-end freshness check

Connector metrics can look fine while something else is broken. Add a simple test: a heartbeat table that a small job updates every minute, and a check that the update reaches the far end within an agreed time. This measures the whole path, not just one piece, and catches problems nobody thought to monitor.

Analogy

A factory conveyor belt has sensors at each stage. But the best test of the whole line is to drop a marked item at the start and time how long it takes to reach the packing table. If it never arrives, something is wrong, even if every sensor says green.

Logs

Connector logs explain reconnects, snapshot phases, and structural changes. Keep them, and search them for repeated warnings. Set up dashboards that show per-table event rates, so a table that suddenly goes quiet stands out.

Troubleshooting guide

SymptomLikely causeFirst check
Connector failed at startMissing privileges, wrong log settings, unreachable databaseConnector error message and database setup
Events stopped, connector shows runningStuck on a huge transaction or blocked producerLag metric, queue capacity, Kafka health
Database disk growing (PostgreSQL)Slot not advancingSlot lag and connector status
Duplicates downstreamRestart after failureConsumer idempotency
Missing columns after a changeStructural change not followedSchema history and connector version notes
Connector cannot find positionLog purgedLog retention against outage length
Two connectors fighting (MySQL)Duplicate server IDServer ID uniqueness

28Advantages, Disadvantages, and Trade-offs

Log-based CDC with Debezium is powerful, and it is not free. An honest view of both sides helps you decide well.

Advantages

  • Captures every insert, update, and delete, in commit order
  • Very low impact on the source database
  • No changes to application code required
  • Removes dual write bugs when combined with the outbox
  • Consumers are decoupled and can be added freely
  • Replayable history when topics are retained
  • Consistent event shape across many databases
  • Seconds of latency instead of hours

Disadvantages

  • Needs database configuration and privileges, sometimes a restart
  • Events mirror table structure unless you use the outbox
  • At-least-once delivery means consumers must handle duplicates
  • Ordering holds only per row, not across tables
  • Snapshots of big tables take time and care
  • Operational hazards such as log growth on PostgreSQL
  • Schema changes need coordination
  • More infrastructure to run: Connect, Kafka, registry

Comparing the capture approaches

FeatureQuery-basedTrigger-basedLog-based (Debezium)
Captures deletesNoYesYes
Catches every intermediate changeNoYesYes
Load on sourceGrows with poll frequencySlows every writeSmall
Application changesOften needs extra columnsNone, but adds triggersNone
Setup effortLowMediumMedium to high
LatencyPoll intervalLowLow

Core trade-offs

Trade-off

Raw tables versus outbox

Raw capture is quick to start but couples consumers to table design. Outbox costs a little more design and gives a stable contract.

Trade-off

Full envelope versus unwrapped

The envelope keeps full detail. Unwrapped is simpler for sinks but loses the before state.

Trade-off

One connector versus many

One is simple. Several split the load and the blast radius, with more to manage.

Trade-off

Short versus long retention

Long retention allows replay and audit at storage cost.

29Design Patterns and Anti-Patterns

Shapes that work, and shapes that look fine until they fail.

Patterns worth copying

Pattern

Outbox for shared events

Services publish curated business events, not table internals.

Pattern

Idempotent consumers

Every consumer can safely apply the same event twice.

Pattern

Compacted topics for state

New consumers rebuild current state from a compact topic.

Pattern

Heartbeat monitoring

A canary table proves the full path works.

Pattern

Registry with compatibility rules

Breaking changes are caught before they reach consumers.

Pattern

Connector per table group

Independent readers for independent domains.

Anti-patterns to avoid

ANTI-PATTERN 01Avoid
What it looks like

Publishing raw table changes from a service’s private database as the public interface to all other teams.

Why it hurts

Every column rename or table split breaks other teams. The database schema becomes a frozen public contract.

Better approach

Use the outbox pattern for shared events, and keep raw capture for private or internal use.

ANTI-PATTERN 02Avoid
What it looks like

Assuming consumers will never see the same event twice.

Why it hurts

After a crash, duplicates arrive, and counters, balances, and totals become wrong.

Better approach

Make every write idempotent, or track positions and event IDs.

ANTI-PATTERN 03Avoid
What it looks like

Leaving a PostgreSQL replication slot unmonitored, or forgetting to drop it when a connector is removed.

Why it hurts

The database quietly hoards log files until the disk fills and the production database stops.

Better approach

Alert on slot lag, cap retained log, use heartbeats, and clean up slots as part of connector removal.

ANTI-PATTERN 04Avoid
What it looks like

Relying on order across different tables or topics.

Why it hurts

Children arrive before parents, and joins fail intermittently in ways that are very hard to reproduce.

Better approach

Tolerate any arrival order, use transaction metadata, or publish whole aggregates.

ANTI-PATTERN 05Avoid
What it looks like

Making the CDC stream do heavy business logic through chains of scripted transforms.

Why it hurts

Throughput drops, debugging gets hard, and logic hides in connector settings.

Better approach

Keep transforms light. Put real logic in stream processors or services with tests.

30Best Practices and Common Mistakes

A checklist to walk through before going live, followed by the mistakes that catch most teams on their first attempt.

Before you go live

1

Prepare the database

Turn on the right log level and row detail, create a dedicated user, and confirm log retention is longer than your worst outage.

2

Capture only what you need

List tables and columns explicitly. Avoid “everything” patterns that pull in temporary and internal tables.

3

Set up Kafka properly

Choose replication, retention, and compaction per topic. Protect the schema history and offsets topics.

4

Choose format and registry rules

Pick a compact format with a registry and set a compatibility level.

5

Plan the snapshot

Decide the mode, estimate the duration, and pick a quiet time for large tables.

6

Make consumers idempotent

Test them by replaying the same events twice.

7

Wire up monitoring and runbooks

Lag, silence, slot growth, connector status, and an end-to-end heartbeat check.

Common mistakes

Mistake

Tables without primary keys

Without a key, ordering, compaction, and incremental snapshots all suffer. Add keys before capturing.

Mistake

Log retention too short

A weekend outage becomes a full re-snapshot because the log was already purged.

Mistake

Forgetting deletes downstream

Sinks that ignore delete events leave ghost records in indexes and reports.

Mistake

Treating money as floating point

Precision quietly drifts. Use exact or text representations for currency.

Mistake

Sharing the history topic

Two connectors writing one history topic corrupt each other. One per connector.

Mistake

Skipping a rehearsal

The first failover, the first big schema change, and the first re-snapshot should not happen in production without practice.

31Deployment and Rollout

A careful order of steps turns a risky launch into a boring one.

What the connector needs to know

At its core, a connector needs to know how to reach the database, which user to use, which tables to include, what name prefix to give topics, where to store structure history if required, which format to use for keys and values, and which transforms to apply. Keep these settings in version control and review changes like code.

Running Connect well

Cluster

At least two workers

So a crash moves the connector to a survivor within seconds.

Cluster

Replicated internal topics

Offsets, configs, and status topics need a high replication factor.

Cluster

Separate clusters by criticality

A heavy analytics connector should not share workers with a critical outbox connector.

Platform

Containers and operators

Many teams run Connect on Kubernetes, using an operator to describe connectors as configuration and roll out upgrades safely.

A safe rollout order

1

Try it on a copy

Restore a recent backup to a test database and run the full pipeline there.

2

Start with one small table

Prove the whole path, including consumers and deletes.

3

Watch through a peak

Observe lag and database impact during the busiest hour.

4

Add tables in waves

Use incremental snapshots to add more without disruption.

5

Upgrade carefully

Read release notes, test connector upgrades in staging first, and upgrade one connector at a time.

!
Rollback plan

Know how to stop a connector, what to do with its slot or position, and how to restart from a snapshot. Removing a connector without a plan can leave behind an active slot that quietly fills the database disk.

32Testing Your CDC Pipeline

The interesting bugs appear during failures and structural changes. Test those on purpose.

TestWhat you doWhat you expect
Basic flowInsert, update, and delete a rowThree matching events; a tombstone after the delete
SnapshotStart a connector on a populated tableRead events for each row, then live changes without gaps
Connector crashKill the connector mid-streamResumes; possible duplicates; consumers stay correct
Kafka broker lossStop a brokerBrief delay; no loss
Database restartRestart the source databaseConnector reconnects on its own
Long outagePause the connector longer than log retention in a testBehaves as planned: fails clearly or re-snapshots
Column addedAdd a column, insert dataNew field appears; consumers keep working
Column renamed or droppedRename or drop in testYou see exactly which consumers break
Huge transactionUpdate a million rowsMemory and lag stay within limits
Duplicate replaySend every event twice to a consumerFinal state identical
FailoverSwitch the database to a standbyRecovery follows your written runbook

Test with real containers

It is easy to spin up a database, Kafka, and Connect in containers on a laptop or in a build pipeline. Write automated tests that change rows and check the events that come out. Then wire in your consumers and confirm the end result. Tests like these catch surprises in connector upgrades before your users do.

Example: the surprise rename

A developer renames a column from “cust_name” to “customer_name” in a routine cleanup. The pipeline test fails: the search sink no longer receives a name and indexes blanks. The team adds the new column, copies data, updates consumers, and drops the old column a month later. Without the test, search results would have silently lost names.

33A Worked Example: An Online Bookstore Grows Up

A made-up story to tie everything together. The company, Northwind Books, and its teams are fictional.

Northwind Books started with one application and one PostgreSQL database. As it grows, the search page is slow, the recommendations are stale, and the finance team’s reports arrive a day late. The engineers decide to adopt CDC with Debezium on Kafka.

1

Prepare PostgreSQL

They set the log level to logical, create a dedicated capture user, and set the replica identity to full on the tables where they need the previous row. Log retention alerts are added from day one.

2

Capture books and customers first

One Debezium connector watches only the book and customer tables, with an explicit column list that leaves out password hashes. Topics are compacted so new consumers can rebuild quickly.

3

Fresh search

A search sink unwraps the events and upserts into the search engine. Deletes remove books that were withdrawn. Search results now match the catalogue within seconds.

4

Outbox for orders

The order service adds an outbox table. Placing an order writes the order and an “order placed” event in one transaction. A second connector routes outbox rows to an order events topic. Shipping and email services subscribe.

5

Idempotent consumers

After a test where a connector was killed, the email service sent two confirmations for one order. The team adds a check on the event ID, and duplicates disappear.

6

Warehouse feed

A warehouse sink appends all order changes, and a view resolves the latest state. Finance sees figures that are minutes old, not a day old.

7

The scare

During a holiday, the search connector is accidentally paused. Two days later, an alert on database log growth fires. The team resumes the connector, which catches up in minutes, and adds a stronger alert on connector status and a heartbeat check.

8

Schema change routine

A registry now blocks breaking changes. Renames follow the add, copy, and drop routine with notice to all consumers.

What the story teaches

Lesson

Start small and useful

Two tables and one visible win built trust for the rest.

Lesson

Outbox for contracts

Business events kept other services away from table internals.

Lesson

Test the failure, not just the flow

The duplicate email showed up in a test, not in front of customers.

Lesson

Watch the database side too

The log growth alert caught what connector metrics missed.

34Real-World Usage Patterns

How organizations of different shapes typically use Debezium. These are common industry patterns, described in general terms, not claims about any single company’s private setup.

The microservices company

Teams that split a large application into many services almost always meet the dual write problem, and the outbox pattern with Debezium is a widely used answer. Each service keeps its own database and publishes business events through its own outbox. Other services react to those events rather than reading each other’s data.

The modernizing enterprise

Large organizations with long-lived databases use CDC to escape batch integration. They stream changes from core systems into Kafka, then feed new digital products, analytics platforms, and cloud data stores from the same stream. The old system stays untouched, which is exactly what risk-averse teams want.

The analytics-first team

Data teams use Debezium to replace nightly extracts with continuous feeds into warehouses and lakes. They value that the source database is no longer hammered by big queries, and that fresh numbers arrive within minutes.

The cloud migration project

Teams moving databases between data centers or cloud providers use CDC to keep old and new in sync while they shift applications gradually. When every application has moved, they switch off the old database with confidence.

The platform team

Some companies offer CDC as an internal service. Application teams request a capture, and the platform provides connectors, topics, schemas, and monitoring in a standard way. Standard naming, ownership records, and default security rules turn CDC from a specialist project into an everyday tool.

Outbox
for reliable events between services
Streaming
in place of nightly batch copies
Gradual
migration with old and new in sync

35Quick Glossary

Every important word from this guide in one place.

TermPlain meaning
At-least-once deliveryNothing is lost, but some events may repeat after a failure
BinlogMySQL’s binary log of changes
CDCChange Data Capture: turning database changes into events
CompactionKeeping only the newest event per key in a topic
ConnectorA configured job that moves data in or out of Kafka
CQRSSeparating the model for writing from the models for reading
DebeziumAn open-source log-based CDC platform
EnvelopeThe wrapper holding before, after, operation, and source details
IdempotentSafe to apply more than once with the same result
Incremental snapshotCopying a table in chunks while streaming continues
Kafka ConnectA framework for running connectors
Logical decodingPostgreSQL’s way to turn its log into row changes
OffsetThe saved position that lets a reader resume
Outbox patternWriting events to a table in the same transaction as the business data
PublicationA PostgreSQL list of tables shared with readers
Replication slotA PostgreSQL marker that keeps log data for a reader
Schema history topicA topic recording table structure changes for some connectors
Schema registryA service storing schema versions and checking compatibility
SMTSingle message transform: a small per-record change step
SnapshotA one-time read of existing rows
Strangler figReplacing an old system gradually, piece by piece
TombstoneA message with a key and no value, marking a deletion
Transaction logThe database’s internal record of every committed change
WALWrite-ahead log, PostgreSQL’s transaction log

36Frequently Asked Questions

Short, direct answers to the questions people ask most often.

Q1Does Debezium slow down my database?
Very little. It reads the log the database already writes. The main extra costs are the initial snapshot, which reads tables, and a small amount of log reading and network. Test under peak load.
Q2Can Debezium miss a change?
Not while the log is available. It resumes from its saved position. Changes can only be missed if the log was purged before the connector read it, in which case it needs a fresh snapshot.
Q3Will I get duplicates?
Possibly, after a failure. Delivery is at-least-once, so design consumers to handle repeats.
Q4Are events ordered?
Yes for one row, because the same key goes to the same partition. No across tables or different rows.
Q5Do I need Kafka to use Debezium?
Not strictly. Debezium Server sends changes to other systems, and the Engine embeds capture in an application. Kafka Connect with Kafka is the most common setup.
Q6Why can’t I add more tasks to speed up a connector?
The database log is one ordered sequence, so one task reads it. Split tables across several connectors if you need more parallel capture.
Q7What is the difference between CDC and the outbox pattern?
CDC is the mechanism that reads changes from the log. The outbox is a design that uses CDC to publish deliberate business events from a dedicated table.
Q8How do deletes work?
A delete event carries the removed row, followed by a tombstone with an empty value so compacted topics can drop the key. Sinks must react to both.
Q9What happens if the connector is down for days?
If the database still has the log, it catches up. If not, you need a snapshot. Keep log retention longer than your worst outage.
Q10Why is my PostgreSQL disk filling up?
Probably a replication slot that is not advancing, so the database keeps all log data behind it. Check slot lag, restart the connector, or drop unused slots.
Q11Can I capture tables without primary keys?
Yes, but ordering by key, compaction, and incremental snapshots work poorly. Add primary keys wherever you can.
Q12How do I handle a column rename?
Treat it as breaking. Add the new column, copy data, move consumers over, then drop the old column later.
Q13Should I unwrap events?
For simple sinks, yes. If consumers need the before state or source details, keep the full envelope.
Q14Is CDC safe for sensitive data?
It can be, with care: least-privilege database user, TLS, excluded or masked columns, strict topic access, and a plan for deletion requests.
Q15How do I add a new table to a running connector?
Add it to the table list and use an incremental snapshot signal to copy its existing rows, while live streaming continues.

37Summary and Key Takeaways

You now have the full picture, from the first row written to the last consumer reading it.

Change Data Capture turns a database from a silent island into a source of live events. Debezium reads the database’s own transaction log, turns each change into a structured event, and sends it through Kafka Connect into Kafka, where any number of consumers can use it at their own pace. The patterns built on top, such as fresh search indexes, the outbox, local copies for services, gradual migrations, audit trails, and analytics feeds, all share the same foundation.

Write once to the database, and let the log tell everyone else what changed.

Key Takeaways

  • Log-based CDC is complete and gentle. It captures every insert, update, and delete in commit order without extra queries or application changes.
  • Expect at-least-once delivery. Make every consumer idempotent, and use event IDs or log positions to skip repeats.
  • Ordering holds per row, not across tables. Tolerate any arrival order, use transaction metadata, or publish whole aggregates.
  • Use the outbox for shared events. It removes the dual write trap and keeps table internals private.
  • The log has a shelf life. Keep database log retention longer than any outage, and watch PostgreSQL slots so the disk never fills.
  • Plan for change. Use a schema registry with compatibility rules, and treat renames and type changes as migrations.
  • Secure and monitor the stream. Mask sensitive columns, restrict topic access, and add an end-to-end heartbeat so silent failures are caught first.