Change Data Capture Patterns with Debezium in Kafka
A complete, plain-English guide to turning every insert, update, and delete in your database into a reliable stream of events: how Debezium reads database logs, how Kafka carries the changes, and the proven patterns that use them.
Imagine a bank branch that keeps its official record in a big paper ledger. Every time a customer deposits or withdraws money, a clerk writes a new line in the ledger. Now imagine that the fraud team, the mobile app, the statement printer, and the accounting office all need to know about each new line the moment it appears. One way is to have each of them walk to the ledger every few minutes and read the whole thing again. That is slow, tiring, and they still miss changes that happened and were corrected between visits.
A much better way is to place a quiet observer beside the clerk, who copies down each new line the instant it is written and sends a note to everyone who cares. That observer is the idea behind Change Data Capture, usually shortened to CDC. In the world of modern software, the ledger is a database, the lines are changes to rows, the notes travel through Apache Kafka, and one of the most popular observers is an open-source tool called Debezium.
This guide starts from zero. Every new word is explained the first time it appears, every big idea comes with an everyday comparison, and each chapter ends with something you can act on. You will learn how the capture works inside, the patterns teams build with it, the traps that catch beginners, and how to run it safely in production.
1The Problem: Databases Are Islands
Almost every business keeps its most important facts in a database. The trouble starts when other systems need those facts too.
A database is a program that stores data safely and lets other programs read and change it. An online store, for example, keeps customers, products, and orders in one. That works well for the store’s own application. But soon other systems want the same information. A search page needs fresh product data. A recommendation engine needs every order. A dashboard needs live sales numbers. A warehouse system needs to know when to ship.
Each of these wants to know two things: what the data looks like now, and what just changed. Databases are very good at the first and, traditionally, quite poor at the second.
A database is like a busy town hall that keeps every citizen’s records in locked cabinets. Anyone can ask a clerk for a file. But there is no town crier announcing “a new house was registered on Maple Street just now”. If you want news, you must keep asking, and the clerks grow tired of the same question.
The old answers, and why they hurt
Nightly batch copies
Copy the whole table every night. Simple, but data is up to a day old, and each copy puts heavy load on the database.
Polling for changes
Ask “anything new since last time?” every few seconds. It misses deletes and burns database effort even when nothing changed.
Dual writes
The application writes to the database and then also sends a message. If one step fails, the two disagree, silently and permanently.
Database triggers
Small programs inside the database record changes into extra tables. They slow down every write and are hard to maintain.
Writing to a database and to a message system in two separate steps can never be made fully safe. If the program crashes between the steps, one has the change and the other does not. CDC and the outbox pattern, both covered later, exist largely to remove this trap.
What we want is a way to learn about every change, in order, without slowing the database, without missing anything, and without asking the application to do extra work. That wish list is exactly what log-based CDC delivers.
2Change Data Capture in Plain Words
CDC is a simple idea with a few very different ways to build it. Knowing the differences helps you see why Debezium chose the approach it did.
Change Data Capture means detecting every change made to data in a database, and turning each change into a message that other systems can use. An insert becomes a “row created” message. An update becomes a “row changed” message that says what it was and what it became. A delete becomes a “row removed” message.
Three ways to capture changes
| Approach | How it works | Strength | Weakness |
|---|---|---|---|
| Query-based | Repeatedly select rows changed since last check, using a timestamp or version column | Simple; no special database features | Misses deletes and quick changes; adds query load; needs extra columns |
| Trigger-based | Database triggers write each change into a shadow table | Captures deletes; works on many databases | Slows writes; adds schema clutter; fragile |
| Log-based | Read the database’s own internal record of changes | Complete, ordered, low impact, includes deletes | Needs database configuration and privileges; depends on database internals |
Why the log is special
Nearly every serious database writes each change to an internal file before it does anything else. This file is called the transaction log. Different databases call it different things: the binary log in MySQL, the write-ahead log in PostgreSQL, the redo log in Oracle. The database uses it for its own safety, to recover after a crash and to copy changes to replicas. It already holds every committed change, in the exact order they happened.
Log-based CDC simply reads that file. The database does no extra work for the observer, the application does not change at all, and nothing is missed. Debezium is a log-based CDC tool.
Think of a restaurant kitchen. Every order the waiters write goes onto a ticket rail, and the cooks work from it. If you want to know every dish sold today, you do not interrupt the cooks to ask. You read the tickets on the rail. The tickets are the transaction log.
3Kafka and Kafka Connect: The Road and the Trucks
Debezium does not work alone. It rides on two other pieces, and understanding them makes everything that follows clearer.
Kafka in one minute
Apache Kafka is a system that stores and moves streams of events. Events are written to named channels called topics. A topic is split into partitions, which are ordered logs. Programs that write are producers. Programs that read are consumers, and they remember how far they have read by keeping an offset, a bookmark. Kafka keeps the events for as long as you configure, so many consumers can read the same events at their own pace, and new consumers can go back and read from the start.
That last point is what makes Kafka a natural home for database changes. A stream of changes can be read by the search team, the analytics team, and the fraud team, each independently, each at its own speed.
Kafka Connect in one minute
Kafka Connect is a framework that runs ready-made programs called connectors to move data between Kafka and other systems. A source connector brings data into Kafka. A sink connector sends data from Kafka out to another system. Connect handles the boring parts: running the connectors, restarting them after failures, storing their progress, and spreading the work over several machines.
Kafka is a highway with many lanes. Kafka Connect is a fleet of trucking companies. A source connector is a truck that picks up goods at a factory and drives them onto the highway. A sink connector is a truck that takes goods off the highway and delivers them to a shop. Debezium provides very clever trucks for one kind of factory: databases.
Worker
A running Connect process. Several workers together form a Connect cluster.
Connector
The configured job, such as “capture changes from this database”.
Task
The unit of work a connector splits into. Debezium connectors normally use a single task, because one database log is a single ordered story.
Converter
Turns records into bytes for Kafka, using a format such as JSON, Avro, or Protobuf.
Transform
A small step that reshapes each record on the way through. Called a single message transform, or SMT.
Offset storage
Where Connect saves the connector’s progress, so it can resume after a restart. It is kept in a Kafka topic.
4What Debezium Is, and How You Can Run It
Debezium is a family of tools around one job: turning database changes into event streams.
Debezium is an open-source platform for change data capture. Its main product is a set of Kafka Connect source connectors, one for each supported database. There are connectors for MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Db2, and several others. Each knows how to read that database’s own log format and translate it into a common event shape.
Three ways to run it
On Kafka Connect
The most common way. You install the Debezium connector into a Connect cluster and configure it. Events flow into Kafka topics, and you gain everything Connect offers: scaling, restarts, and shared management.
Debezium Server
A standalone program that runs the connectors without Kafka Connect and can send changes to other systems, such as cloud messaging services, instead of Kafka. Useful when Kafka is not part of the picture.
Debezium Engine
A library that embeds the capture logic inside your own application. It gives the most control and the least infrastructure, but you must handle progress storage and restarts yourself.
This guide focuses on the first way, Debezium on Kafka Connect, because it is what most teams use with Kafka.
Why teams choose it
No application changes
Existing programs keep writing to their database as before. Capture is added from the outside.
Consistent event shape
Different databases produce the same style of events, so downstream code looks alike.
Snapshots and streaming together
It first reads existing data, then follows live changes, with no gap between the two.
Open source and widely used
A large community, many connectors, and plenty of production experience.
Debezium reports what changed in the database. It does not know why. A price update is just “price went from 10 to 12”. If you need business meaning, such as “a discount was applied”, you must add it yourself, and the outbox pattern in a later chapter shows how.
5Architecture: The Pieces and How They Connect
Study the picture first. Then read the roles below it.
flowchart LR
subgraph App["Application Side"]
A["Application"] -->|"writes rows"| DB[("Source Database")]
DB --> LOG["Transaction Log"]
end
subgraph Connect["Kafka Connect Cluster"]
W1["Worker 1 with Debezium Connector"]
W2["Worker 2 standby"]
OFF["Offsets and Status Topics"]
end
subgraph K["Kafka Cluster"]
T1["Topic: orders"]
T2["Topic: customers"]
SH["Schema History Topic"]
end
subgraph Consumers["Consumers"]
C1["Search Indexer"]
C2["Data Warehouse Sink"]
C3["Microservice"]
end
LOG -->|"reads changes"| W1
W1 --> T1
W1 --> T2
W1 --> SH
W1 --- OFF
W2 --- OFF
T1 --> C1
T1 --> C2
T2 --> C3
Reading the diagram
The application on the left writes to the database as usual. The database appends each change to its transaction log. The Debezium connector, running inside a Kafka Connect worker, reads that log like a very patient reader following a long scroll. For each change it creates an event and writes it to a Kafka topic, usually one topic per table. Consumers on the right read the topics and do whatever they need to do.
Component roles
Transaction log
The database’s own record of changes. Debezium reads it but never writes to it.
Debezium connector
Reads the log, converts changes to events, and sends them to Kafka.
Offset topic
Holds the connector’s bookmark in the log, so it resumes exactly where it stopped.
Table topics
One topic per captured table, holding that table’s change events.
Schema history topic
For some databases, a record of table structure changes, so old log entries can be understood later.
Consumers and sinks
The programs that use the events: indexers, warehouses, caches, and services.
The application and the consumers never talk to each other. The database log and Kafka sit between them. That loose coupling is what lets you add a new consumer next month without touching the application.
6Reading the Log: How Each Database Tells Its Story
Every database keeps its change record differently. Debezium hides the differences, but knowing them explains many setup steps and surprises.
| Database | Where changes come from | Bookmark type | Setup highlights |
|---|---|---|---|
| MySQL and MariaDB | The binary log (binlog) | Log file and position, or a global transaction ID | Row-based binlog format; a unique server ID; replication privileges |
| PostgreSQL | Write-ahead log through logical decoding | Log sequence number (LSN) | Logical replication level; a replication slot; a publication |
| SQL Server | Change tables filled by SQL Server’s own CDC feature | Commit and change LSN | CDC enabled on the database and each table; agent running |
| Oracle | Redo log via LogMiner or another reader | System change number (SCN) | Archive log mode; supplemental logging; special user rights |
| MongoDB | Change streams built on the operation log | Resume token | Replica set or sharded cluster |
The common pattern
Despite the differences, the setup follows the same three steps for every database. First, turn on the database feature that keeps enough detail in the log. Second, create a dedicated user with just the rights the connector needs. Third, tell the connector where the database is, which tables to watch, and a name to use for the topics.
Each database writes its diary in a different language and handwriting. The Debezium connector is a skilled translator who reads any of them and always writes the summary in the same standard format, so the readers downstream never need to learn each diary’s style.
The log is not kept forever. Databases delete old log files to save disk. If a connector is stopped for longer than the log is kept, the entries it needs may be gone. Then the only way back is to take a fresh snapshot. Chapter after chapter, you will see this same theme: the log has a shelf life.
7Anatomy of a Change Event
Everything downstream depends on what a change event contains. Learn the shape once, and every pattern becomes easier to read.
Each change Debezium captures becomes one Kafka message with two main parts: a key and a value. The key identifies the row that changed, using the table’s primary key columns. The value is a wrapper called the envelope, which describes the change itself.
What is in the envelope
| Field | What it holds | Why it matters |
|---|---|---|
| before | The row as it was before the change | Lets you see what was overwritten; empty for inserts |
| after | The row as it is after the change | The new state; empty for deletes |
| op | The kind of change: create, update, delete, or read | Tells consumers how to react |
| source | Where the change came from: database, table, log position, transaction, time | Traceability and ordering |
| ts_ms | When the connector processed the event | Measuring delay |
| transaction | Optional details tying the event to its database transaction | Rebuilding groups of related changes |
The four kinds of operation
A new row appeared
The “before” part is empty. The “after” part shows the new row.
A row changed
Both parts are present. Comparing them shows exactly which columns changed.
A row was removed
The “before” part shows the removed row. The “after” part is empty.
A row seen during a snapshot
Marks rows copied during the initial snapshot, not real changes at that moment.
Picture a courier’s delivery slip. The label on the outside is the key: which parcel this is about. Inside the envelope is a form with “what it looked like before”, “what it looks like now”, the type of change, and a stamp saying which warehouse and shift handled it. Anyone can read the form and know exactly what happened.
Deletes and tombstones
When a row is deleted, Debezium normally sends two messages. The first is a delete event that describes the removed row. The second is a tombstone, a message with the same key and an empty value. Kafka uses tombstones in a feature called log compaction: when compaction runs, a tombstone tells Kafka that all older messages for that key can be thrown away. Without tombstones, deleted rows would live forever in a compacted topic.
Column types and precision
Databases have types with no perfect twin in messaging formats, such as exact decimals, dates with time zones, and large binary values. Debezium offers settings to choose how they arrive, for example decimals as exact numbers, as text, or as floating point numbers. Choose carefully for money. Floating point can quietly round values, while exact or text forms preserve every digit.
An update event may not carry the full previous row unless the database is set up to record it. On PostgreSQL, for example, the “before” part is thin by default. If your pattern needs it, change the table’s replica identity setting before you start.
8Snapshots: Catching Up on What Already Exists
The log only holds recent changes. To capture a table that already has millions of rows, Debezium first takes a snapshot.
When a connector starts for the first time, it faces a puzzle. Your tables already contain data, but the log only remembers changes from a certain point onward. A new consumer that only sees future changes would be missing the whole starting picture. So Debezium takes an initial snapshot: it reads the current rows of every captured table and emits them as read events. Then it switches to reading the log from exactly the point where the snapshot began, so nothing falls between the two.
You join a group chat halfway through the year. To catch up, you first read a summary of everything so far, which is the snapshot. From then on you read every new message as it arrives, which is streaming. The handover between summary and live messages must have no gaps or overlaps.
Snapshot modes
| Mode | What it does | When you would use it |
|---|---|---|
| Initial | Snapshot data and structure on first start, then stream | The usual choice for a new connector |
| Initial only | Snapshot and stop, with no streaming | A one-time data copy |
| Schema only | Capture table structure but not existing rows; stream new changes | You only care about changes from now on |
| When needed | Snapshot again if the saved position is gone from the log | Automatic recovery for a connector that fell far behind |
| Never | No snapshot; start streaming from the log’s oldest available point | Rare; when you manage the starting state yourself |
The problem with big snapshots
A classic snapshot reads all tables in one go. For huge tables this can take hours, and if it fails near the end, it starts again. During that time the connector produces no live changes, so consumers see nothing new. Some databases also hold locks or long-running transactions during a snapshot, which can affect other work.
Incremental snapshots
Debezium solved this with incremental snapshots. Instead of pausing streaming to copy everything, it copies a table in small chunks, interleaved with live log reading. A special signal, sent through a small control table or a Kafka topic, starts a chunked snapshot of chosen tables at any time. The connector uses a clever marking technique so that if a live change touches a row inside a chunk, the newer change wins and old copies are dropped.
flowchart TD
S["Signal: snapshot table X"] --> C1["Open chunk window"]
C1 --> R["Read next chunk of rows by primary key"]
R --> L["Keep reading live log changes in parallel"]
L --> D{"Live change touches a row inside the chunk?"}
D -- "Yes" --> DROP["Drop the older copy, keep live change"]
D -- "No" --> KEEP["Keep chunk row"]
DROP --> EMIT["Emit chunk to topic"]
KEEP --> EMIT
EMIT --> MORE{"More chunks left?"}
MORE -- "Yes" --> R
MORE -- "No" --> DONE["Snapshot of table complete"]
Streaming never stops
Live consumers keep getting fresh changes while old data is being copied.
Resumable
A failure loses at most one chunk, not the whole snapshot.
On demand
Re-snapshot one table after a bug or a new consumer, without restarting anything.
Primary keys
Chunking works by walking primary key ranges, so tables should have a stable primary key.
9Internal Working: From Commit to Kafka, Step by Step
Follow one order from the moment an application saves it to the moment a consumer reads it.
sequenceDiagram
autonumber
participant App as Application
participant DB as Source Database
participant Dbz as Debezium Connector
participant Conn as Kafka Connect
participant K as Kafka Topic
participant Con as Consumer
App->>DB: Insert order row and commit
DB->>DB: Append change to transaction log
Dbz->>DB: Stream log entries from saved position
DB-->>Dbz: Committed change for the orders table
Dbz->>Dbz: Decode row, build envelope, choose key
Dbz->>Conn: Hand over record with source offset
Conn->>K: Produce record to topic
K-->>Conn: Acknowledged
Conn->>Conn: Save source offset to offsets topic
Con->>K: Poll for new records
K-->>Con: Order change event
Step 1 and 2: the write and the log
The application saves an order and commits the transaction. Before the database considers the commit final, it has already appended the change to its transaction log. This is the same step the database always performs for its own safety. Debezium adds nothing here.
Step 3 to 5: reading and decoding
The connector holds a connection to the database that behaves like a replica asking for the latest log entries. It receives entries in commit order. For each committed change it decodes the raw bytes, looks up the table structure, builds the envelope with before, after, and source details, and picks the key from the primary key columns.
Step 6 to 8: handing to Kafka
The connector passes the record to the Connect framework together with a note saying which log position it came from. Connect sends the record to Kafka and waits for confirmation. Only after Kafka has confirmed does Connect save the position to its offsets topic. That order is very important: the bookmark only moves after the data is safe.
Step 9 to 10: consumption
Any consumer subscribed to the orders topic reads the event on its own schedule and does its job. The database and the application are not involved at all.
Because Debezium only reads a file the database is already writing, adding capture to a database usually costs a small amount of extra disk reading and network. Compare that to polling, where every check is a new query competing with your application.
10Offsets, Restarts, and Delivery Guarantees
What happens when the connector crashes at the worst possible moment? The answer shapes how you must write every consumer.
The connector’s bookmark is called an offset. It records the position in the database log up to which changes have been safely sent. If the connector crashes and restarts, it reads its last saved offset and asks the database to continue from there.
The gap that causes duplicates
Look again at the order of steps: send to Kafka, then save the bookmark. Suppose the connector crashes after Kafka confirmed a batch but before the bookmark was saved. On restart, the connector resumes from the older bookmark and sends those changes again. Consumers see the same events twice. This is called at-least-once delivery: nothing is ever lost, but a few events may repeat after a failure.
A courier delivers a parcel, then goes to tick it off on the list. If the courier faints just after delivering and before ticking, the replacement courier sees the parcel is not ticked and delivers it again. Better twice than never. The receiver’s answer is to notice “I already have this parcel”.
Making consumers safe
Idempotent writes
Design the consumer so applying the same event twice gives the same result. Upserts keyed by primary key are naturally safe.
Deduplication by position
Each event carries its log position. A consumer can remember the highest position it has applied and skip older ones.
Deduplication by identifier
For outbox events, a unique event ID lets consumers ignore repeats.
Transactional sinks
Write the data and the consumer’s own progress together in one transaction, so both happen or neither does.
What about exactly-once?
Newer versions of Kafka Connect include support for exactly-once delivery for source connectors, and some Debezium connectors can use it. It reduces duplicates from the connector side but does not remove the need for careful consumers, because they can still crash and reprocess. Treat it as a helpful extra, and check the documentation for your connector and version before depending on it.
When the saved position is gone
If the connector was down so long that the database has already deleted the log section it needs, the bookmark points to nothing. Depending on the snapshot mode you chose, the connector either fails loudly or takes a new snapshot. Failing loudly is safer for data you cannot afford to duplicate or reorder, but it needs a human to decide what to do.
11Topics, Keys, Partitions, and Ordering
Where events land in Kafka decides what order consumers see them in. Get this right and many bugs never appear.
One topic per table
By default, each captured table gets its own topic, named from a prefix you choose, the schema, and the table. For example, a connector with the prefix “shop” watching the orders table in the sales schema writes to a topic named like shop.sales.orders. Consumers subscribe only to the tables they care about.
The key decides the partition
The event key is the row’s primary key. Kafka uses the key to choose the partition: the same key always goes to the same partition. Since a partition is an ordered log, all changes to one row are delivered in the order they happened. That is the ordering promise you can rely on.
Changes to the same row arrive in order. Changes to different rows, or in different tables, may arrive in a different order than they were committed. Design consumers around that rule.
Why cross-table order is not guaranteed
An order and its order lines live in two tables, so they land in two topics, each with its own partitions. A consumer reading both topics may see a line before its parent order. To handle this, consumers should tolerate arrival in any order, or use the transaction details in the event to group related changes. A later chapter shows options.
Log compaction: keeping only the latest
A topic can be set to keep just the newest event for each key, discarding older ones. This is called log compaction. For a CDC topic it means the topic becomes a compact copy of the table’s current state: one event per row. New consumers can read it from the start to rebuild the full table quickly. Tombstones make deleted rows disappear eventually.
| Retention style | Topic holds | Best for |
|---|---|---|
| Time-based | All changes for a period | Streaming to live consumers; short-term replay |
| Compacted | The latest change for every key | Rebuilding current state; cache and search loading |
| Compacted plus time | Latest per key, but not forever | Balance of state recovery and storage cost |
| Long retention | Full history for a long time | Audit trails and replaying history |
The special history topic
Some connectors, such as MySQL, also write table structure changes to a separate schema history topic. It must be a single partition and must keep its data forever, because the connector reads it on restart to understand old log entries. Losing it is serious. Protect it with a high replication factor and no automatic deletion, and never share it between connectors.
12The Pattern Map: What Teams Build with CDC
Before diving into each pattern, here is the whole map in one place. Every pattern answers a different question about moving data.
| Pattern | Question it answers | Typical consumers |
|---|---|---|
| Replicate to a read store | How do I keep a search index or cache fresh? | Search engines, caches |
| Outbox | How do I publish events safely from my own service? | Other microservices |
| Service data sync and read models | How do services share data without sharing databases? | Query services, reporting services |
| Strangler migration | How do I move from an old system to a new one safely? | New system, comparison tools |
| Audit and history | How do I keep a record of every change? | Audit stores, compliance tools |
| Analytics feed | How do I feed a warehouse or lake with fresh data? | Warehouses, lakes, dashboards |
Think of CDC as a tap on a water main. What you do with the water is up to you: fill a tank, water a garden, or feed a fountain. The tap is the same. The patterns are the different things people build downstream.
13Pattern 1: Keeping Search Indexes and Caches Fresh
The most popular CDC use is also the easiest to understand: keep a second copy of data, shaped for fast reading, in step with the database.
Databases are built to store and protect data. They are not always the best at searching text or serving millions of quick lookups. So teams copy data into a search engine for full-text queries, or into a cache for lightning-fast reads. The challenge is keeping the copy up to date.
The old way and its pain
Without CDC, the application updates the database and also tries to update the index or cache. Two writes, two chances to fail. When one fails, the search page shows a product that no longer exists, or a price that changed yesterday. Nightly rebuilds patch over the problem but leave hours of wrong data.
The CDC way
Application writes only to the database
One write, one source of truth. The application knows nothing about the index.
Debezium publishes the change
The change lands on the product topic within a short time.
A sink applies it to the index or cache
Creates and updates become upserts. Deletes become removals.
The index catches up
Typically within seconds. If it falls behind, it simply reads faster to catch up.
A library keeps its books on shelves, which is the database. The card catalogue in the lobby is the search index. Instead of the librarian rewriting cards from memory, a helper watches every book that arrives or leaves and updates the cards automatically. The catalogue always agrees with the shelves.
Details that matter
Flatten the envelope
Most sinks want the “after” row only, not the full envelope. A standard transform unwraps it for them.
Handle deletes
Make sure the sink turns delete events into removals, or the index will show ghosts.
Shape the document
A search document often joins several tables. Build it with a stream processor, or consume several topics and merge.
Plan a rebuild
Use compacted topics or a fresh snapshot to rebuild the index from scratch whenever mappings change.
Cache invalidation is famously hard. With CDC, you can either update the cached value from the “after” row, or simply delete the cache entry when a change arrives and let the next read refill it. The second option is simpler and avoids stale writes.
14Pattern 2: The Outbox, Publishing Events Without Dual Writes
This is the pattern that solves the dual write trap from the first chapter. It is the most valuable idea in this guide.
A microservice often needs to do two things at once: save data in its own database and announce that something happened to other services. For instance, an order service saves a new order and tells the shipping service. Doing both separately can fail halfway. The outbox pattern fixes this by making the announcement part of the database transaction.
flowchart LR
subgraph Svc["Order Service"]
API["Business logic"] --> TX["One database transaction"]
end
TX -->|"writes order row"| ORD[("orders table")]
TX -->|"writes event row"| OUT[("outbox table")]
OUT --> LOG["Transaction log"]
LOG --> DBZ["Debezium with Outbox Event Router"]
DBZ --> T1["Topic: order events"]
DBZ --> T2["Topic: payment events"]
T1 --> SHIP["Shipping Service"]
T1 --> MAIL["Notification Service"]
T2 --> BILL["Billing Service"]
How it works
Add an outbox table
A small table in the service’s own database, with columns for a unique event ID, the type of thing it is about, that thing’s ID, the event type, and the event details.
Write both in one transaction
When the service saves an order, it also inserts a row into the outbox table, in the same transaction. Either both happen or neither does. The dual write problem is gone.
Debezium captures the outbox inserts
It watches only the outbox table.
A transform routes the event
Debezium ships a ready-made transform, the outbox event router. It reads the columns, sends the event to a topic chosen from the row, uses the aggregate ID as the key, and puts the event details in the message body.
Other services consume
They see clean business events like “order placed”, not raw table changes.
Imagine you write a letter and want to post it, but your mailbox might be closed. Instead, you place the letter in your own outgoing tray together with the ledger entry in a single motion. A reliable postal worker then collects everything from the tray and delivers it. Your action and the letter cannot separate, and the delivery happens eventually.
Why teams like it
Atomic
The event exists if and only if the business change was saved.
Business meaning
Events are designed by the service, not leaked from its table layout. Table changes stay private.
Stable contract
The service can reshape its tables freely while keeping events steady.
Ordered per entity
Using the aggregate ID as key keeps events for one order in sequence.
Housekeeping and duplicates
Outbox rows pile up. Debezium reads them from the log, so the service can delete each row right after inserting it, in the same transaction. The insert is still recorded in the log, but the table stays tiny. Since delivery is at-least-once, consumers may see an event twice, so each event carries a unique ID that consumers use to ignore repeats.
Do not confuse the outbox with simply capturing your business tables and publishing raw row changes to other teams. That exposes internal table design as a public contract, and every column rename becomes a breaking change for others.
15Pattern 3: Sharing Data Between Services and Building Read Models
Microservices should own their data. But services still need each other’s facts. CDC lets them share without sharing a database.
In a well-designed system of small services, each service has its own private database. The customer service owns customers. The order service owns orders. But the order screen needs to show the customer’s name. Calling the customer service on every request is slow and makes the order service fail whenever the customer service is down.
Local copies from events
With CDC, the order service keeps its own small local copy of the customer fields it needs, such as name and shipping city. It fills and updates this copy by listening to the customer topic. Now showing an order needs no call to the customer service at all. The copy may lag by a moment, but it is always available.
Instead of phoning head office every time you need a customer’s address, each branch keeps its own address book and updates it whenever head office sends a change notice. Branches keep working even when the phone line to head office is down.
Read models and CQRS
A read model is a copy of data shaped exactly for one screen or report. The idea of separating writing models from reading models is often called CQRS, short for command query responsibility segregation. CDC provides the feed: changes from several write databases are combined by a stream processor into one read table that answers a specific question quickly, like “show every order with its customer and delivery status”.
| Choice | Good for | Cost |
|---|---|---|
| Call the other service live | Data that must be perfectly current | Slower, and fails when the other service fails |
| Keep a local copy from CDC | Data that can be a few seconds old | More storage; must handle late or out-of-order updates |
| Build a joined read model | Screens and reports needing data from many services | A stream processing job to maintain |
Expose events, not tables
Use the outbox pattern for data the owning service wants to share as a contract, and use raw table capture only within a team or for non-critical copies. A raw table capture couples other teams to your table layout.
Example: the shipping cost calculator
The shipping service needs the customer’s country and the product’s weight to compute costs. Instead of calling two services, it keeps small local tables filled from the customer and product topics. Quotes now appear in a few milliseconds, and continue to work even during a customer service outage.
16Pattern 4: Migrating Gradually with the Strangler Approach
Replacing a big old system in one weekend is a famous way to fail. CDC lets you move piece by piece while both systems stay in step.
The strangler fig approach is named after a vine that slowly grows around a tree until it replaces it. In software, you build new components beside the old system and move traffic across gradually, until the old system has nothing left to do.
Capture the old system’s database
Debezium streams every change from the legacy database into Kafka, without modifying legacy code.
Feed the new system
The new service consumes the changes and builds its own data store. It now mirrors the old system’s data, live.
Compare and verify
Run both side by side. Compare results to find bugs before any user depends on the new system.
Shift reads first
Send a small share of read traffic to the new system, then more.
Shift writes carefully
Move writes to the new system, and reverse the flow so the old database receives changes from the new one until it is switched off.
Moving house room by room, while the old house still has the lights on, is easier than throwing everything on a truck in one night. You can go back for a forgotten item, and you can live in both places while you settle in.
The same idea helps with database upgrades and moves between database products or cloud providers. The old and new databases stay in sync through the stream, you switch when ready, and you keep the ability to switch back.
Two-way sync can create loops, where a change flows from A to B and back to A forever. Mark each change with its origin and refuse to send it back where it came from.
17Pattern 5: Audit Trails and Change History
Databases forget. When a row is updated, the old value is gone. CDC remembers what the database forgets.
Many businesses must answer questions like “who changed this price, and what was it before?” The database only holds the current state. Because every event has both a before and an after, a CDC stream is a natural audit trail. Store the events in a durable place, and you can replay the full history of any row.
Compliance records
Keep an unchangeable list of every change, for regulators or internal review.
Debugging
Reconstruct exactly how a record reached its current, surprising state.
Time travel
Rebuild what the data looked like at a past moment.
Anomaly detection
Watch for suspicious patterns, like many changes to sensitive fields.
Who did it?
The log knows what changed but not who changed it, because the database only sees the application’s shared login. To capture the human, have the application store the user in a column such as “last updated by”, or add context to the transaction. Debezium can then include it in the event. Some setups also write a separate metadata row per transaction with user details.
Where to keep the history
A Kafka topic with long retention works for shorter horizons. For years of history, sink the events into cheap object storage or an append-only table, and partition by date. Protect it with strict access, since it holds every past value, including ones the main database deleted on purpose.
A history that keeps everything can clash with rules about deleting personal data. Plan how you will honor deletion requests in the audit store, for example by encrypting each person’s data with a key you can destroy.
18Pattern 6: Feeding Warehouses and Data Lakes
Analytics teams have long lived on yesterday’s data. CDC brings them minutes-old data without hammering the source database.
A data warehouse is a system built to answer big analytical questions over lots of history. A data lake stores raw files cheaply and lets many tools read them. Both traditionally received data through nightly bulk extracts. Those extracts read entire tables, load the source database heavily, and deliver stale numbers.
Streaming replaces the nightly copy
With CDC, the warehouse receives a steady trickle of changes. A sink connector writes them into the warehouse or lake as they come, in small batches. The source database sees no extra queries, and dashboards are minutes old instead of a day old.
The delete and update challenge
Analytical storage is often append-only. It is happy to add rows but reluctant to change or remove them. To keep an accurate current picture, there are two common approaches.
Append everything, resolve later
- Store each change event as a new row, with its operation and time
- Build a “latest state” view that picks the newest row per key and hides deleted ones
- Keeps full history for free
- Simple and fast to write
Merge into a current table
- Apply updates and deletes so the table always shows the current state
- Queries are simple and cheap
- Requires a storage format that supports row-level changes
- History is lost unless kept separately
Many teams do both: keep the raw change log as a permanent record, and maintain a merged current-state table on top of it.
A shop keeps every receipt in a box, which is the change log, and also keeps a tidy stock sheet that shows how many items are on the shelf right now. The receipts prove the sheet is right, and the sheet saves everyone from counting receipts.
19Single Message Transforms: Reshaping Events on the Way Out
The raw envelope is rich but heavy. Small transforms let you trim, route, and filter events before they reach Kafka.
A single message transform, or SMT, is a small step that Kafka Connect applies to each record, one at a time, as it passes through. You can chain several. They are ideal for light reshaping, but they are not meant for heavy work like joining streams. For that, use a stream processor.
Unwrap the record
Reduces the envelope to just the new row state, which is what most sinks and simple consumers expect. It can add chosen metadata fields back in, such as the operation type or table name, and it can turn deletes into a flag instead of a missing record.
Outbox event router
Sends outbox rows to topics chosen from their contents, and shapes them into clean business events.
Table router
Merges topics from many similar tables into one topic, for example monthly or sharded tables that share a structure.
Event filtering
Drops events you do not need, such as changes to internal columns or test data, using conditions on the content.
Column handling
Removes, masks, or shortens sensitive columns. Some of this is built into the connector itself.
Add fields
Adds timestamps, constant labels, or a copy of the key into the value.
When to unwrap, and when not to
Unwrapping is convenient, but it throws away the before state and some source details. If a consumer needs to know what a value was before, or needs the exact log position, keep the full envelope. A common design is to keep raw envelopes in the main topics for careful consumers, and let simple sinks apply the unwrap step on their own side.
SMTs are like the small adjustments a mail room does to each letter: put it in the right pigeonhole, remove a wrapper, stamp a label. They are quick, one letter at a time. Sorting a hundred letters into a report is a different job for a different department.
Each SMT runs on every event, so slow or complex ones hurt throughput. Scripting-based transforms are flexible but heavier. Keep chains short, and test with realistic volume.
20Schemas, Serialization, and Surviving Change
Tables change. Columns are added, renamed, and dropped. Your event streams must survive that without breaking every consumer.
Every change event has a shape, called a schema: which fields exist and what type each is. Consumers rely on that shape. When the source table changes, the shape of new events changes too. Managing that gracefully is called schema evolution.
Choosing a format
| Format | Size | Schema handling | Note |
|---|---|---|---|
| JSON with embedded schema | Large | Each message carries its schema | Easy to read; wasteful at scale |
| JSON without schema | Medium | None enforced | Simple; risky as things change |
| Avro with a registry | Small | Schema stored once in a registry; messages hold an ID | Popular; strong compatibility checks |
| Protobuf with a registry | Small | Same idea as Avro | Good for teams already using Protobuf |
A schema registry is a service that stores every version of every schema and can refuse a new version that would break existing consumers. For production CDC, a compact binary format plus a registry is the usual choice.
How table changes appear
Add a column
Usually safe. New events include the field. Old consumers that ignore unknown fields keep working. Give the column a default so compatibility rules are satisfied.
Drop a column
The field disappears from new events. Consumers that expected it may break. Announce and phase these changes in.
Rename a column
From the log’s view, this looks like one column vanishing and another appearing. It is a breaking change for consumers. Prefer adding the new column, copying data, then dropping the old one later.
Change a type
The riskiest change. A widening change, like a small number type to a bigger one, is often fine. Anything else usually needs a planned migration.
The compatibility rulebook
Decide a compatibility mode with your registry. Backward compatibility means new readers can read old data. Forward compatibility means old readers can read new data. Full means both. For CDC topics that consumers replay from the start, backward compatibility is the minimum, because new consumer versions must read events written long ago.
Treat database schema changes like changes to a public interface. Write down who consumes each table, and give them notice before any structural change.
Some databases let structural changes slip through in ways the connector cannot fully follow, and connector versions differ in what they support. Test your typical changes in a staging environment with a running connector, before doing them in production.
21PostgreSQL Specifics: Slots, Publications, and the Disk Full Risk
PostgreSQL is a favorite for CDC, and it has one famous hazard. Understanding it can save you a midnight outage.
PostgreSQL exposes changes through logical decoding, which turns its internal write-ahead log into readable row changes. Debezium connects using a replication slot, a named marker inside the database that says “this reader has consumed the log up to here, so keep everything after it”. A publication lists which tables are shared with that reader.
Key setup points
Logical WAL level
The database must be set to record enough detail for logical decoding. Changing it needs a restart, so plan ahead.
A decoding plugin
Modern setups use the built-in plugin, so nothing extra needs installing on the database server.
Replica identity
Controls how much of the old row is written for updates and deletes. The default gives only the key. Set it to full for tables where you need the whole “before” row.
Dedicated user
A user with replication rights and read access to just the captured tables.
The slot that fills the disk
Because the slot promises to keep all log data after its position, a slot whose reader has stopped forces PostgreSQL to hoard log files without limit. Disk space shrinks steadily until the database itself stops. This is the most common serious CDC incident on PostgreSQL.
A newspaper delivery stops when you go on holiday, but the newsagent has promised to keep every issue for you. Each day the pile grows. After a month there is no room in the shop for anything else. The promise was kind, but it needs a limit.
Monitor slot lag
Watch how far behind each slot is, in bytes of log retained. Alert early.
Cap retained log
Newer PostgreSQL versions can limit how much log a slot may hold, at the price of invalidating a slot that falls too far behind.
Drop unused slots
Deleting a connector does not always delete its slot. Remove orphans by hand.
Use heartbeats
On quiet databases, or when only a few tables are captured, the slot may not advance even though the database is busy elsewhere. A heartbeat setting writes small periodic changes so the slot keeps moving.
Other gotchas
Very large column values stored out of line, known as TOAST values, are omitted from update events when they did not change. Debezium then inserts a placeholder, and consumers must not overwrite real data with it. Failover is another concern: slots have traditionally lived only on the primary server, so after a failover to a replica the slot may be missing. Newer versions add ways to keep slots in step on standby servers, but plan and test your failover carefully.
22MySQL and Others: Binlogs, Server IDs, and Friends
Each database has its own quirks. Here are the ones that trip up teams most often.
MySQL and MariaDB
The MySQL connector reads the binary log. It needs the log in row format with full row images, so each entry holds the complete changed row. The connector poses as a replica, so it needs a server ID unique among all replicas and readers, or two readers will kick each other off. The database user needs replication and read rights.
Binlog retention
Binary logs expire after a set number of days. Keep retention comfortably longer than the longest connector outage you can imagine.
Global transaction IDs
When enabled, the connector can reconnect to a different server after a failover, since positions are the same everywhere.
Schema history topic
Required. Records table structure changes so the connector can read old entries. Never lose it.
Read from a replica
Capturing from a replica reduces load on the primary, but check that it records the full binlog and replicates all changes.
SQL Server
SQL Server has its own change data capture feature, which copies changes into special change tables using an agent job. Debezium reads those tables. The feature must be turned on for the database and each table. Because of the extra step through change tables, there is a small extra delay and extra storage. Cleanup jobs delete old change table rows, so make sure they do not remove entries before the connector reads them.
Oracle
Oracle capture reads the redo log, most often via LogMiner. It needs archive logging on, supplemental logging turned on for the captured tables, and a carefully privileged user. Oracle has licensing and performance considerations, and very high volumes may need a different reader. Plan Oracle projects with extra care and read the connector guide closely.
MongoDB
MongoDB capture uses change streams built on its operation log, and needs a replica set or a sharded cluster. Events describe documents rather than rows, and updates can be delivered as changes or as whole documents depending on settings.
| Database | Biggest hazard | Guardrail |
|---|---|---|
| PostgreSQL | Slot causing the disk to fill | Slot lag alerts; retention cap; heartbeats |
| MySQL | Binlog expiring before the connector catches up | Generous retention; lag alerts |
| SQL Server | Cleanup job removing changes too early | Tune cleanup retention; monitor lag |
| Oracle | Log mining load and complexity | Careful sizing; dedicated mining settings |
| MongoDB | Oplog rolling over | Large oplog window; monitor lag |
23Transactions and Ordering Across Tables
A database transaction changes several rows at once, all or nothing. In Kafka, those changes scatter across topics. Here is how to put the story back together.
When an order is placed, one transaction might insert an order row, three order line rows, and update a stock count. In the log these are one atomic unit. In Kafka they become five separate events on three topics with no guaranteed arrival order. A consumer that reads them independently may see a line without its order, or a stock change before the order it belongs to.
Three ways to cope
Tolerate any order
Design consumers so a missing parent is fine for a moment. Store the child, and fill the link when the parent arrives. Simple and robust.
Use transaction metadata
Debezium can emit begin and end markers on a special topic, and tag each event with its transaction ID and position inside it. A consumer can wait until the end marker before acting on the group.
Send one aggregate
Use the outbox pattern to publish one event holding the whole order, so there is nothing to reassemble.
A moving crew unloads one household into several rooms. The bed may arrive upstairs before the mattress does. You can either accept things show up in any order and tidy up at the end, or ask the crew foreman to announce “all boxes for this house are now delivered”.
The single-partition option
If you must have a strict global order, you can route related tables into one topic with one partition. This preserves order but limits throughput to one consumer, so use it only for small, critical flows.
24Performance and Scalability
A single log is a single story, and a single story is read by one reader. That shapes how CDC scales.
One task per connector
The transaction log is one ordered sequence, so most Debezium connectors run a single task to read it in order. You cannot speed up one connector by adding tasks. This surprises people who are used to Connect connectors scaling out.
How to scale anyway
Capture only what you need
List the tables and columns explicitly. Reading and decoding less data is the cheapest speed-up there is.
Split by table groups
Run several connectors, each watching a different set of tables. Each has its own bookmark, its own replication slot or server ID, and its own topics. Ordering within a table stays intact.
Tune the internal queue
The connector has a queue between reading and sending, with settings for batch size and queue size. Larger values raise throughput and use more memory.
Tune Kafka producing
Batching, compression, and acknowledgment settings on the Connect side affect how fast events leave.
Scale consumers by partition
Give table topics enough partitions for the parallelism you need downstream. Remember that per-key order holds within a partition.
Things that slow capture down
Huge transactions
A bulk update touching millions of rows creates millions of events in one transaction. The connector may hold them in memory until commit.
Wide rows and big values
Large text or binary columns inflate every event.
Slow Kafka or slow converters
If sending is the bottleneck, reading faster only fills the queue.
Heavy transforms
Scripted transforms and large chains cost time on every event.
A quick sizing sense
Suppose a database sees 2,000 row changes per second at peak, and each event is about 1 KB with the envelope. That is roughly 2 MB per second, about 170 GB per day before compression, well within what a single connector can handle. Storage in Kafka then depends on retention. With three copies and seven days, the topics hold roughly 3.5 terabytes at that rate uncompressed, so compression and compaction matter.
Bulk data loads and big maintenance updates can flood the stream with events that nobody needs. Coordinate with the teams that run them, or filter those changes out, to avoid an accidental surge.
25High Availability and Failure Scenarios
Everything fails eventually. What matters is what happens next, and whether you lose data, get duplicates, or simply pause.
What keeps running, and what does not
A CDC pipeline has many parts, and each can fail on its own. The good news is that the design is forgiving: the log is durable, the bookmark is stored in Kafka, and Kafka itself is replicated. Most failures mean a pause and a catch-up, not lost data.
flowchart TD
F["Something fails"] --> Q1{"Which part?"}
Q1 -->|"Connect worker"| W["Another worker takes over the connector"]
W --> W2["Resume from saved offset, possible duplicates"]
Q1 -->|"Kafka broker"| K["Replicas take over leadership"]
K --> K2["Producer retries, brief delay"]
Q1 -->|"Source database down"| D["Connector retries connection"]
D --> D2["Resume from same position when database returns"]
Q1 -->|"Database failover"| FO{"Same log positions on new primary?"}
FO -->|"Yes, with global IDs"| FO1["Reconnect and continue"]
FO -->|"No, slot or position lost"| FO2["Re-snapshot or manual repair"]
Q1 -->|"Log already purged"| P["Saved position no longer exists"]
P --> P2["Take new snapshot and rebuild consumers"]
Scenario walk-throughs
A Connect worker crashes
The Connect cluster notices and hands the connector to another worker, which reads the saved offset and continues. A short pause and a few repeated events are the only signs. Run at least two workers so a spare is always waiting.
The database is restarted for maintenance
The connector loses its connection and retries with waits in between. When the database returns, it resumes from its bookmark. No action needed, but set the retry limits high enough for planned maintenance windows.
The connector is down for a weekend
On Monday, the connector reads all the changes it missed. If the database kept its log for longer than the outage, everything is fine. If not, the position is gone, and a new snapshot is needed. This is why log retention should exceed your worst outage.
The database fails over to a replica
With MySQL global transaction IDs, the connector can find the same position on the new primary. With PostgreSQL, the replication slot may not exist on the new server, so a fresh slot and possibly a snapshot are needed. Rehearse this before it happens for real.
The schema history topic is damaged
For connectors that need it, this is serious, because the connector cannot interpret old log entries. Recovery options exist that rebuild the structure history from the database, but they need care. Prevent the problem with high replication and no deletion.
Design for the gap
Assume the pipeline will sometimes pause. Downstream systems should treat the data as “eventually current” and show or use the age of the data when it matters. Provide a way to rebuild any consumer from a snapshot or a compacted topic, so recovery is a routine action instead of an emergency.
26Security and Privacy: CDC Sees Everything
By design, CDC copies every change to every captured table into a place where many people can read it. That deserves real care.
A CDC stream holds complete rows, including names, emails, and possibly financial or health details. Kafka topics also last longer than a single query would. Treat every CDC topic as if it were the database itself.
Protect the door into the database
Least privilege user
A dedicated user with only the rights the connector needs, and read access only to the captured tables.
Encrypted connections
Use TLS between the connector and the database, and between the connector and Kafka.
Secrets kept outside
Passwords should come from a secrets manager through a configuration provider, not from plain text in a shared connector file.
Restricted admin access
Whoever can change connector settings can point it at any table. Limit that power.
Protect the data in the stream
Capture fewer columns
Exclude columns nobody needs. Data never captured cannot leak.
Mask or shorten sensitive values
The connector can replace values with fixed characters, or cut long text down, before the event is even created.
Encrypt fields where needed
Sensitive fields can be encrypted before they reach Kafka, with keys held by the owners. Community transforms and client libraries can help.
Lock down topics
Use Kafka access rules so only approved consumers can read each topic. Raw table topics deserve stricter access than curated business event topics.
Plan for deletion requests
Privacy laws may require erasing a person’s data. Compacted topics with tombstones and sinks that honor deletes help, but backups and history stores need their own plan.
A CDC stream is a live photocopier attached to your filing cabinet. It is wonderfully handy, and it means every secret in the cabinet can appear in a second place. You would not leave the copies on a public table.
The schema history topic and the offsets topic are also sensitive. The history topic can contain the text of structural statements, and the offsets topic reveals what is being captured. Restrict both.
27Monitoring, Logging, and Metrics
The worst CDC failure is a silent one. The connector looks healthy while data quietly stops flowing. Good monitoring catches that first.
The metrics that matter
| Signal | What it tells you | Alert when |
|---|---|---|
| Milliseconds behind source | How far the connector lags the database | It grows steadily or passes your freshness target |
| Time since last event | Whether events are still flowing | Long silence on a busy database |
| Connector and task status | Running, paused, or failed | Anything other than running |
| Queue remaining capacity | Whether the internal queue is filling | It stays near empty capacity |
| Snapshot progress | Rows copied and time taken | Snapshot much slower than expected |
| Events seen and filtered | Volume and how much is dropped | Sudden jumps or drops |
| Database log retained | How much log the database keeps for the reader | Growth toward disk limits |
| Consumer lag downstream | How far consumers trail the topics | Lag grows without recovering |
The end-to-end freshness check
Connector metrics can look fine while something else is broken. Add a simple test: a heartbeat table that a small job updates every minute, and a check that the update reaches the far end within an agreed time. This measures the whole path, not just one piece, and catches problems nobody thought to monitor.
A factory conveyor belt has sensors at each stage. But the best test of the whole line is to drop a marked item at the start and time how long it takes to reach the packing table. If it never arrives, something is wrong, even if every sensor says green.
Logs
Connector logs explain reconnects, snapshot phases, and structural changes. Keep them, and search them for repeated warnings. Set up dashboards that show per-table event rates, so a table that suddenly goes quiet stands out.
Troubleshooting guide
| Symptom | Likely cause | First check |
|---|---|---|
| Connector failed at start | Missing privileges, wrong log settings, unreachable database | Connector error message and database setup |
| Events stopped, connector shows running | Stuck on a huge transaction or blocked producer | Lag metric, queue capacity, Kafka health |
| Database disk growing (PostgreSQL) | Slot not advancing | Slot lag and connector status |
| Duplicates downstream | Restart after failure | Consumer idempotency |
| Missing columns after a change | Structural change not followed | Schema history and connector version notes |
| Connector cannot find position | Log purged | Log retention against outage length |
| Two connectors fighting (MySQL) | Duplicate server ID | Server ID uniqueness |
28Advantages, Disadvantages, and Trade-offs
Log-based CDC with Debezium is powerful, and it is not free. An honest view of both sides helps you decide well.
Advantages
- Captures every insert, update, and delete, in commit order
- Very low impact on the source database
- No changes to application code required
- Removes dual write bugs when combined with the outbox
- Consumers are decoupled and can be added freely
- Replayable history when topics are retained
- Consistent event shape across many databases
- Seconds of latency instead of hours
Disadvantages
- Needs database configuration and privileges, sometimes a restart
- Events mirror table structure unless you use the outbox
- At-least-once delivery means consumers must handle duplicates
- Ordering holds only per row, not across tables
- Snapshots of big tables take time and care
- Operational hazards such as log growth on PostgreSQL
- Schema changes need coordination
- More infrastructure to run: Connect, Kafka, registry
Comparing the capture approaches
| Feature | Query-based | Trigger-based | Log-based (Debezium) |
|---|---|---|---|
| Captures deletes | No | Yes | Yes |
| Catches every intermediate change | No | Yes | Yes |
| Load on source | Grows with poll frequency | Slows every write | Small |
| Application changes | Often needs extra columns | None, but adds triggers | None |
| Setup effort | Low | Medium | Medium to high |
| Latency | Poll interval | Low | Low |
Core trade-offs
Raw tables versus outbox
Raw capture is quick to start but couples consumers to table design. Outbox costs a little more design and gives a stable contract.
Full envelope versus unwrapped
The envelope keeps full detail. Unwrapped is simpler for sinks but loses the before state.
One connector versus many
One is simple. Several split the load and the blast radius, with more to manage.
Short versus long retention
Long retention allows replay and audit at storage cost.
29Design Patterns and Anti-Patterns
Shapes that work, and shapes that look fine until they fail.
Patterns worth copying
Outbox for shared events
Services publish curated business events, not table internals.
Idempotent consumers
Every consumer can safely apply the same event twice.
Compacted topics for state
New consumers rebuild current state from a compact topic.
Heartbeat monitoring
A canary table proves the full path works.
Registry with compatibility rules
Breaking changes are caught before they reach consumers.
Connector per table group
Independent readers for independent domains.
Anti-patterns to avoid
What it looks like
Publishing raw table changes from a service’s private database as the public interface to all other teams.
Why it hurts
Every column rename or table split breaks other teams. The database schema becomes a frozen public contract.
Better approach
Use the outbox pattern for shared events, and keep raw capture for private or internal use.
What it looks like
Assuming consumers will never see the same event twice.
Why it hurts
After a crash, duplicates arrive, and counters, balances, and totals become wrong.
Better approach
Make every write idempotent, or track positions and event IDs.
What it looks like
Leaving a PostgreSQL replication slot unmonitored, or forgetting to drop it when a connector is removed.
Why it hurts
The database quietly hoards log files until the disk fills and the production database stops.
Better approach
Alert on slot lag, cap retained log, use heartbeats, and clean up slots as part of connector removal.
What it looks like
Relying on order across different tables or topics.
Why it hurts
Children arrive before parents, and joins fail intermittently in ways that are very hard to reproduce.
Better approach
Tolerate any arrival order, use transaction metadata, or publish whole aggregates.
What it looks like
Making the CDC stream do heavy business logic through chains of scripted transforms.
Why it hurts
Throughput drops, debugging gets hard, and logic hides in connector settings.
Better approach
Keep transforms light. Put real logic in stream processors or services with tests.
30Best Practices and Common Mistakes
A checklist to walk through before going live, followed by the mistakes that catch most teams on their first attempt.
Before you go live
Prepare the database
Turn on the right log level and row detail, create a dedicated user, and confirm log retention is longer than your worst outage.
Capture only what you need
List tables and columns explicitly. Avoid “everything” patterns that pull in temporary and internal tables.
Set up Kafka properly
Choose replication, retention, and compaction per topic. Protect the schema history and offsets topics.
Choose format and registry rules
Pick a compact format with a registry and set a compatibility level.
Plan the snapshot
Decide the mode, estimate the duration, and pick a quiet time for large tables.
Make consumers idempotent
Test them by replaying the same events twice.
Wire up monitoring and runbooks
Lag, silence, slot growth, connector status, and an end-to-end heartbeat check.
Common mistakes
Tables without primary keys
Without a key, ordering, compaction, and incremental snapshots all suffer. Add keys before capturing.
Log retention too short
A weekend outage becomes a full re-snapshot because the log was already purged.
Forgetting deletes downstream
Sinks that ignore delete events leave ghost records in indexes and reports.
Treating money as floating point
Precision quietly drifts. Use exact or text representations for currency.
Sharing the history topic
Two connectors writing one history topic corrupt each other. One per connector.
Skipping a rehearsal
The first failover, the first big schema change, and the first re-snapshot should not happen in production without practice.
31Deployment and Rollout
A careful order of steps turns a risky launch into a boring one.
What the connector needs to know
At its core, a connector needs to know how to reach the database, which user to use, which tables to include, what name prefix to give topics, where to store structure history if required, which format to use for keys and values, and which transforms to apply. Keep these settings in version control and review changes like code.
Running Connect well
At least two workers
So a crash moves the connector to a survivor within seconds.
Replicated internal topics
Offsets, configs, and status topics need a high replication factor.
Separate clusters by criticality
A heavy analytics connector should not share workers with a critical outbox connector.
Containers and operators
Many teams run Connect on Kubernetes, using an operator to describe connectors as configuration and roll out upgrades safely.
A safe rollout order
Try it on a copy
Restore a recent backup to a test database and run the full pipeline there.
Start with one small table
Prove the whole path, including consumers and deletes.
Watch through a peak
Observe lag and database impact during the busiest hour.
Add tables in waves
Use incremental snapshots to add more without disruption.
Upgrade carefully
Read release notes, test connector upgrades in staging first, and upgrade one connector at a time.
Know how to stop a connector, what to do with its slot or position, and how to restart from a snapshot. Removing a connector without a plan can leave behind an active slot that quietly fills the database disk.
32Testing Your CDC Pipeline
The interesting bugs appear during failures and structural changes. Test those on purpose.
| Test | What you do | What you expect |
|---|---|---|
| Basic flow | Insert, update, and delete a row | Three matching events; a tombstone after the delete |
| Snapshot | Start a connector on a populated table | Read events for each row, then live changes without gaps |
| Connector crash | Kill the connector mid-stream | Resumes; possible duplicates; consumers stay correct |
| Kafka broker loss | Stop a broker | Brief delay; no loss |
| Database restart | Restart the source database | Connector reconnects on its own |
| Long outage | Pause the connector longer than log retention in a test | Behaves as planned: fails clearly or re-snapshots |
| Column added | Add a column, insert data | New field appears; consumers keep working |
| Column renamed or dropped | Rename or drop in test | You see exactly which consumers break |
| Huge transaction | Update a million rows | Memory and lag stay within limits |
| Duplicate replay | Send every event twice to a consumer | Final state identical |
| Failover | Switch the database to a standby | Recovery follows your written runbook |
Test with real containers
It is easy to spin up a database, Kafka, and Connect in containers on a laptop or in a build pipeline. Write automated tests that change rows and check the events that come out. Then wire in your consumers and confirm the end result. Tests like these catch surprises in connector upgrades before your users do.
Example: the surprise rename
A developer renames a column from “cust_name” to “customer_name” in a routine cleanup. The pipeline test fails: the search sink no longer receives a name and indexes blanks. The team adds the new column, copies data, updates consumers, and drops the old column a month later. Without the test, search results would have silently lost names.
33A Worked Example: An Online Bookstore Grows Up
A made-up story to tie everything together. The company, Northwind Books, and its teams are fictional.
Northwind Books started with one application and one PostgreSQL database. As it grows, the search page is slow, the recommendations are stale, and the finance team’s reports arrive a day late. The engineers decide to adopt CDC with Debezium on Kafka.
Prepare PostgreSQL
They set the log level to logical, create a dedicated capture user, and set the replica identity to full on the tables where they need the previous row. Log retention alerts are added from day one.
Capture books and customers first
One Debezium connector watches only the book and customer tables, with an explicit column list that leaves out password hashes. Topics are compacted so new consumers can rebuild quickly.
Fresh search
A search sink unwraps the events and upserts into the search engine. Deletes remove books that were withdrawn. Search results now match the catalogue within seconds.
Outbox for orders
The order service adds an outbox table. Placing an order writes the order and an “order placed” event in one transaction. A second connector routes outbox rows to an order events topic. Shipping and email services subscribe.
Idempotent consumers
After a test where a connector was killed, the email service sent two confirmations for one order. The team adds a check on the event ID, and duplicates disappear.
Warehouse feed
A warehouse sink appends all order changes, and a view resolves the latest state. Finance sees figures that are minutes old, not a day old.
The scare
During a holiday, the search connector is accidentally paused. Two days later, an alert on database log growth fires. The team resumes the connector, which catches up in minutes, and adds a stronger alert on connector status and a heartbeat check.
Schema change routine
A registry now blocks breaking changes. Renames follow the add, copy, and drop routine with notice to all consumers.
What the story teaches
Start small and useful
Two tables and one visible win built trust for the rest.
Outbox for contracts
Business events kept other services away from table internals.
Test the failure, not just the flow
The duplicate email showed up in a test, not in front of customers.
Watch the database side too
The log growth alert caught what connector metrics missed.
34Real-World Usage Patterns
How organizations of different shapes typically use Debezium. These are common industry patterns, described in general terms, not claims about any single company’s private setup.
The microservices company
Teams that split a large application into many services almost always meet the dual write problem, and the outbox pattern with Debezium is a widely used answer. Each service keeps its own database and publishes business events through its own outbox. Other services react to those events rather than reading each other’s data.
The modernizing enterprise
Large organizations with long-lived databases use CDC to escape batch integration. They stream changes from core systems into Kafka, then feed new digital products, analytics platforms, and cloud data stores from the same stream. The old system stays untouched, which is exactly what risk-averse teams want.
The analytics-first team
Data teams use Debezium to replace nightly extracts with continuous feeds into warehouses and lakes. They value that the source database is no longer hammered by big queries, and that fresh numbers arrive within minutes.
The cloud migration project
Teams moving databases between data centers or cloud providers use CDC to keep old and new in sync while they shift applications gradually. When every application has moved, they switch off the old database with confidence.
The platform team
Some companies offer CDC as an internal service. Application teams request a capture, and the platform provides connectors, topics, schemas, and monitoring in a standard way. Standard naming, ownership records, and default security rules turn CDC from a specialist project into an everyday tool.
35Quick Glossary
Every important word from this guide in one place.
| Term | Plain meaning |
|---|---|
| At-least-once delivery | Nothing is lost, but some events may repeat after a failure |
| Binlog | MySQL’s binary log of changes |
| CDC | Change Data Capture: turning database changes into events |
| Compaction | Keeping only the newest event per key in a topic |
| Connector | A configured job that moves data in or out of Kafka |
| CQRS | Separating the model for writing from the models for reading |
| Debezium | An open-source log-based CDC platform |
| Envelope | The wrapper holding before, after, operation, and source details |
| Idempotent | Safe to apply more than once with the same result |
| Incremental snapshot | Copying a table in chunks while streaming continues |
| Kafka Connect | A framework for running connectors |
| Logical decoding | PostgreSQL’s way to turn its log into row changes |
| Offset | The saved position that lets a reader resume |
| Outbox pattern | Writing events to a table in the same transaction as the business data |
| Publication | A PostgreSQL list of tables shared with readers |
| Replication slot | A PostgreSQL marker that keeps log data for a reader |
| Schema history topic | A topic recording table structure changes for some connectors |
| Schema registry | A service storing schema versions and checking compatibility |
| SMT | Single message transform: a small per-record change step |
| Snapshot | A one-time read of existing rows |
| Strangler fig | Replacing an old system gradually, piece by piece |
| Tombstone | A message with a key and no value, marking a deletion |
| Transaction log | The database’s internal record of every committed change |
| WAL | Write-ahead log, PostgreSQL’s transaction log |
36Frequently Asked Questions
Short, direct answers to the questions people ask most often.
37Summary and Key Takeaways
You now have the full picture, from the first row written to the last consumer reading it.
Change Data Capture turns a database from a silent island into a source of live events. Debezium reads the database’s own transaction log, turns each change into a structured event, and sends it through Kafka Connect into Kafka, where any number of consumers can use it at their own pace. The patterns built on top, such as fresh search indexes, the outbox, local copies for services, gradual migrations, audit trails, and analytics feeds, all share the same foundation.
Key Takeaways
- Log-based CDC is complete and gentle. It captures every insert, update, and delete in commit order without extra queries or application changes.
- Expect at-least-once delivery. Make every consumer idempotent, and use event IDs or log positions to skip repeats.
- Ordering holds per row, not across tables. Tolerate any arrival order, use transaction metadata, or publish whole aggregates.
- Use the outbox for shared events. It removes the dual write trap and keeps table internals private.
- The log has a shelf life. Keep database log retention longer than any outage, and watch PostgreSQL slots so the disk never fills.
- Plan for change. Use a schema registry with compatibility rules, and treat renames and type changes as migrations.
- Secure and monitor the stream. Mask sensitive columns, restrict topic access, and add an end-to-end heartbeat so silent failures are caught first.