A short refresher on the pieces that connectors rely on every day.

Apache Kafka is a platform for storing and moving streams of events. An event is a small record that describes something that happened, such as a new order, a changed customer address, or a temperature reading. Kafka keeps these events in an ordered, durable log so that many different applications can read them at their own pace.

The problem of the outside world

Most of a company’s data does not start life inside Kafka. It lives in relational databases, log files, cloud storage, software-as-a-service tools, mainframes, and message queues. Likewise, the results that Kafka carries must eventually land in places such as data warehouses, search indexes, and dashboards. Something has to carry data across that border, in both directions, again and again, without losing anything.

Two ways to build the docks

You could write your own producer application for every system that feeds Kafka, and your own consumer application for every system that Kafka feeds. Many teams start there, and many regret it. Each custom program needs its own retry logic, error handling, monitoring, scaling, and recovery after crashes. Kafka Connect exists so that this repeated work is done once, properly, and shared by everyone.

A ready-made framework for moving data between Kafka and everything else.

Kafka Connect is a component that ships with Apache Kafka. It is a runtime that hosts small, reusable plugins called connectors. You do not write code to move data. Instead, you pick a suitable connector, describe your setup in a configuration, and give it to Kafka Connect, which then runs the job continuously, scales it, and recovers it after failures.

The two directions

Every connector falls into one of two families, depending on which way the data flows relative to Kafka.

flowchart LR
  DB[Relational Database] --> SRC[Source Connector]
  FILES[Log Files] --> SRC
  SRC --> K[Kafka Topics]
  K --> SNK[Sink Connector]
  SNK --> DW[Data Warehouse]
  SNK --> SE[Search Index]
  SNK --> OBJ[Object Storage]
        

Both connector families share the same set of moving parts. Learn these words once and everything else becomes easier.

Connector versus task

Beginners often mix these two up, so let us make the difference crisp. The connector is the manager. It reads the configuration, looks at the job, and decides how to split it into pieces. The tasks are the employees who do the pieces. If you ask a connector to copy twenty database tables and allow four tasks, the connector will assign about five tables to each task, and the four tasks will work in parallel.

Where the settings for parallelism live

The connector setting named tasks.max sets the upper limit on how many tasks the connector may create. It is a ceiling, not a promise. The connector decides how many tasks it can actually use, and it may create fewer than the ceiling if the job cannot be split further.

The arrival crew: it goes out into the world, finds data, and brings it into Kafka.

A source connector continuously reads data from an external system and publishes it as records into Kafka topics. The external system might be a relational database, a set of files, a message queue, a software-as-a-service application, or even another Kafka cluster. The connector turns each row, line, message, or change into a Kafka record and appends it to the right topic.

A simple story

A retail company keeps its customer orders in a relational database. The analytics team, the shipping team, and the fraud team all want to react to new orders in real time, but nobody wants to hammer the production database with dozens of separate queries. So the company deploys a source connector for the database. The connector reads new and changed orders once and writes them to an “orders” topic in Kafka. Now every team simply reads the topic, and the database is touched only by the single connector.

sequenceDiagram
  participant EXT as External System
  participant TASK as Source Task
  participant CONN as Connect Framework
  participant KAFKA as Kafka Topic
  TASK->>EXT: Ask for new data since last position
  EXT-->>TASK: Rows or changes
  TASK->>CONN: Hand over records with position info
  CONN->>KAFKA: Write records to the topic
  KAFKA-->>CONN: Acknowledge the write
  CONN->>CONN: Save the source position
        

What a source connector must remember

Because a source connector reads from a system that Kafka knows nothing about, it needs its own way to remember how far it has read. That bookmark is called a source offset. For a file it might be a byte position. For a database table it might be the highest row number or timestamp seen so far. For a database transaction log it might be a log position. Kafka Connect saves these bookmarks safely, so that after a crash the connector resumes from the last saved position instead of starting over or skipping data.

Examples of source connectors

The departure crew: it takes records off Kafka and delivers them to their final destination.

A sink connector continuously reads records from one or more Kafka topics and writes them into an external system. That destination might be a relational database, a search engine, an object store such as a data lake, a cloud data warehouse, a caching system, or a web service that accepts data over HTTP.

A simple story

The same retail company wants its order data to be searchable on a customer support dashboard, and also stored cheaply for long-term analysis. It deploys two sink connectors that both read the “orders” topic. The first writes each order into a search index so support agents can find orders by name or product. The second writes batches of orders as files into cloud object storage, ready for reporting tools. The two connectors work independently, at their own speeds, and neither knows the other exists.

sequenceDiagram
  participant KAFKA as Kafka Topic
  participant CONN as Connect Framework
  participant TASK as Sink Task
  participant EXT as External System
  CONN->>KAFKA: Fetch a batch of records
  KAFKA-->>CONN: Records
  CONN->>TASK: Deliver records after conversion and transforms
  TASK->>EXT: Write the batch to the destination
  EXT-->>TASK: Confirm the write
  CONN->>KAFKA: Commit the offsets of the delivered records
        

What a sink connector must remember

Here is a key difference from source connectors. A sink connector does not need a private bookmark system, because Kafka already has one. Under the hood, each sink connector behaves like a consumer group, and it uses the normal Kafka consumer offsets to remember how far it has read. By default that group is named after the connector with a “connect-” prefix. This means all the familiar tools for checking consumer lag work on sink connectors too.

Examples of sink connectors

Now that each family is clear, here is a direct comparison of the differences that matter in practice.

flowchart TD
  subgraph SOURCE PATH
    A1[External data] --> A2[Source task]
    A2 --> A3[Transforms]
    A3 --> A4[Converter to bytes]
    A4 --> A5[Kafka topic]
  end
  subgraph SINK PATH
    B1[Kafka topic] --> B2[Converter from bytes]
    B2 --> B3[Transforms]
    B3 --> B4[Sink task]
    B4 --> B5[External destination]
  end
        

A closer look at how a source task finds data, labels it, and sends it to the right place.

The polling loop

Each source task runs a simple loop. It asks the external system for whatever is new, wraps the results into records, and gives them to the framework. The framework then sends them to Kafka through an internal producer, and only after Kafka acknowledges the records does it save the matching source offset. If the task waits too long between polls, the data feels stale, and if it polls too eagerly, it can overload the source system.

Source partitions and source offsets

Every record a source task produces carries two extra pieces of information for the framework’s own bookkeeping: a source partition, which identifies which slice of the source the record came from, and a source offset, which identifies the position within that slice. For a database connector, the source partition might be a table name, and the offset might be the last row identifier. Note that these are different from Kafka’s own partitions and offsets, even though the names sound alike.

Where the bookmarks are kept

In distributed mode, Kafka Connect stores source offsets in a dedicated internal Kafka topic. In simple single-process setups it uses a local file. Because the bookmarks are in Kafka, they survive worker failures, and any worker in the cluster can take over a task and continue.

Choosing topics and keys

A source connector decides which topic each record goes to, often based on the name of the source table or file, sometimes with a configurable prefix. It also chooses a record key, commonly the primary key of a database row. Choosing a good key matters, because Kafka uses the key to decide the partition, and keeping all changes for the same row in the same partition preserves their order. Connect can also create topics automatically when a connector needs them, using default settings for partition count and replication that an administrator configures.

A closer look at how a sink task receives records, writes them, and confirms progress.

The delivery loop

Behind every sink task is an ordinary Kafka consumer. It fetches a batch of records from the subscribed topics, and the framework converts and transforms them. The framework then hands the batch to the sink task, which writes them to the destination. Only after the task confirms that the data has been safely written does the framework commit the offsets. This order of “write first, commit after” is what prevents data from being lost.

Batching for speed

Most destinations are far more efficient when they receive many records at once than when they receive them one by one. A database can insert a thousand rows in one round trip, and an object store prefers large files over thousands of tiny ones. Sink connectors therefore collect records into batches, and settings such as the maximum number of records fetched per poll influence batch size. The best batch size is a balance between speed and how much data would need to be re-sent after a failure.

Backpressure and slow destinations

If the destination becomes slow, for example a database under heavy load, the sink task cannot keep up. The connector can ask the framework to pause fetching from Kafka until it has caught up. This is a healthy behaviour: Kafka stores the backlog safely, and the consumer lag grows visibly rather than data being dropped or memory exploding. Watching that lag is how operators notice a struggling destination.

Writing safely: inserts, upserts, and deletes

Between the outside world and Kafka’s bytes, two small stages shape every record.

Converters

Kafka itself stores only bytes and has no idea what they mean. Converters give the bytes meaning. A key converter and a value converter, both chosen in the configuration, decide the format used to serialise and deserialise the record’s key and value. Common choices are plain strings, JSON, and schema-based formats such as Avro and Protobuf, which are usually paired with a Schema Registry.

Single message transforms

A single message transform, often shortened to SMT, is a small piece of logic that changes one record at a time as it flows through the pipeline. Transforms can be chained, and each one runs in order. They are intended for light, simple adjustments, not heavy processing.

For databases especially, there are two very different ways to bring data into Kafka.

Polling

A polling source runs queries at regular intervals to look for new or changed rows. It typically relies on an always-increasing identifier column, a last-modified timestamp column, or a combination of both, and it remembers the highest value seen. It is easy to set up because it needs nothing more than read access to tables.

Change data capture

A change data capture source, often called CDC, reads the database’s own transaction log, the internal journal that the database keeps to recover from crashes. Every insert, update, and delete appears there in exact order. The connector turns these log entries into events, usually with the before and after values of each row, and typically begins with an initial snapshot of the existing data.

flowchart LR
  subgraph POLLING
    T1[Database table] -->|repeated queries| P1[Polling source]
    P1 --> K1[Kafka topic]
  end
  subgraph CDC
    L1[Transaction log] -->|reads every change| C1[CDC source]
    C1 --> K2[Kafka topic]
  end
        

Every kind of sink has its own quirks, and knowing them prevents surprises.

Across all of these, the golden question is the same: how does the destination behave if it receives the same record twice? Your answer to that decides how careful the rest of your design must be.

What happens to your data when machines crash halfway through? This is where connectors earn their reputation.

Why duplicates can happen

Consider a sink task that has written a batch to a database but crashes just before committing the offsets to Kafka. When another worker takes over, it reads from the last committed offset and delivers the same batch again. The data was written twice. On the source side, a source task that has published records to Kafka but crashes before saving its source offset will re-read and re-publish the same data after restart.

sequenceDiagram
  participant KAFKA as Kafka
  participant TASK as Sink Task
  participant DB as Destination
  KAFKA->>TASK: Batch of records 100 to 199
  TASK->>DB: Write records 100 to 199
  DB-->>TASK: Write succeeded
  TASK--xTASK: Crash before offset commit
  KAFKA->>TASK: Same batch delivered again after restart
  TASK->>DB: Write records 100 to 199 again
        

Living with at-least-once

For most sink connectors, the practical answer is to make the destination tolerate repeats. Upserts by key, unique identifiers for documents, and file names derived from offsets all ensure that a repeated write replaces the earlier one instead of adding to it. When the write is idempotent, at-least-once delivery behaves like exactly-once from the outside.

Exactly-once for sources

Recent versions of Kafka Connect added framework support for exactly-once delivery from source connectors, in which the records and the source offsets are committed together in a Kafka transaction. It must be enabled at the worker level, and the specific connector must declare that it supports it. Not every connector does, so check its documentation before relying on it.

Exactly-once for sinks

Connect cannot make an external system transactional on its own. Some sink connectors achieve exactly-once effects by storing progress information together with the data inside the destination, or by writing files whose names are derived deterministically from the offsets, so that repeats overwrite identical files. This depends entirely on the individual connector and destination.

How Kafka Connect spreads work across machines, and what really limits speed.

How work is distributed

In a Connect cluster, several workers share the tasks of all connectors. When you create a connector, the cluster spreads its tasks across the available workers. When a worker joins or leaves, the cluster rebalances and moves tasks to keep the load even. Modern Connect uses an incremental approach to rebalancing, so unaffected tasks keep running while only the necessary ones move.

flowchart TD
  C1[Connector A with 3 tasks] --> W1[Worker 1]
  C1 --> W2[Worker 2]
  C1 --> W3[Worker 3]
  C2[Connector B with 2 tasks] --> W1
  C2 --> W3
  W1 --> KC[Kafka Cluster]
  W2 --> KC
  W3 --> KC
        

What limits parallelism for sinks

A sink connector’s tasks are members of a consumer group, and in a consumer group each partition is read by only one member at a time. So the total number of partitions across the subscribed topics is the maximum number of tasks that can do useful work. If a topic has six partitions and you configure twelve tasks, six of them will sit idle.

What limits parallelism for sources

For sources, the limit depends on how the connector can slice the external system. A database connector might create one task per group of tables, so with a single big table only one task can work on it. A file connector might assign different files to different tasks. Change data capture connectors often use a single task per database, because the transaction log is a single ordered sequence that cannot safely be split.

Real data is messy. A good pipeline decides in advance what to do with a record that cannot be processed.

What can go wrong

The choices the framework offers

By default, a connector is strict: the first bad record causes the task to fail, and it stops until someone investigates. This protects data quality but can halt an entire pipeline because of a single faulty message. Alternatively, you can configure the connector to tolerate errors, logging them and continuing. For sink connectors, you can additionally route failed records to a dead letter queue, which is an ordinary Kafka topic where problem records are stored together with headers that describe what went wrong.

Sources have fewer safety nets

The dead letter queue feature belongs to sink connectors. For sources, problems such as unreadable rows or unsupported types must be handled by the connector’s own settings, by transforms that clean the data, or by fixing the source. Good monitoring of the source task’s status is therefore even more important.

Sources and sinks usually work together as two halves of a bigger design.

flowchart LR
  APP[Application] --> DB[Operational Database]
  DB --> CDC[CDC Source Connector]
  CDC --> T[Kafka Topic]
  T --> S1[Sink to Search Index]
  T --> S2[Sink to Data Lake]
  T --> S3[Sink to Cache]
  T --> KS[Stream Processing Application]
        

How you deploy the workers changes how safe and scalable your connectors are.

The three internal topics

In distributed mode, Connect keeps its own state in three special Kafka topics. One stores connector configurations, so every worker knows which jobs exist. Another stores source offsets, the bookmarks of source connectors. The third stores status information such as which tasks are running or failed. These topics must be replicated properly, because losing them means losing the memory of the cluster.

Rolling out changes

Configuration changes go through the REST interface. You can pause a connector to stop it without losing its position, resume it later, restart individual tasks that have failed, and update settings. Because plugins are loaded from disk at startup, upgrading a connector plugin usually means a rolling restart of the workers, one by one, to avoid downtime.

Connectors hold the keys to your databases and your Kafka cluster, and they run unattended. Both facts demand care.

Security essentials

What to monitor

A handful of habits separate smooth Connect deployments from fragile ones.

Helpful patterns

Anti-patterns to avoid

Kafka Connect is powerful, but it is not always the right tool.

A useful rule of thumb: if the job is “move data between Kafka and a well-known system with little change”, use Connect. If the job is “compute something new from the data”, use a stream processing application, and let Connect handle the movement on either side. If your system is unusual and no connector exists, you can either write a custom connector using the Connect framework, which keeps the operational benefits, or write a standalone application when the logic is too specific.

Let us follow a realistic incident from first complaint to lasting fix.

A logistics company uses a CDC source connector to stream shipment changes into Kafka, and a sink connector that writes them into a reporting database. One morning, the operations team complains that the dashboard shows shipments that were cancelled yesterday as still active.

The lesson is that a “green” connector status proves very little. The real evidence comes from comparing data at each stage of the pipeline, and from watching the places where problem records are parked.

Look at the direction of data relative to Kafka. A source brings data into Kafka from an external system. A sink sends data out of Kafka into an external system.

It is a component that ships with Apache Kafka, but it runs as its own set of processes called workers, separate from the Kafka brokers. It talks to Kafka through normal client connections.

It stores source offsets, which are bookmarks about its position in the external system. Connect saves them in a Kafka topic in distributed mode, and on restart the connector resumes from the last saved position.

It uses ordinary Kafka consumer offsets. Each sink connector acts as a consumer group, so standard tools for viewing consumer lag apply to it.

Yes. Each sink connector has its own consumer group and its own progress, so many sinks can read the same topic independently, each at its own speed.

The task setting is only a maximum. The number of useful sink tasks cannot exceed the total partitions of the subscribed topics, and connectors may choose to create fewer tasks than the limit.

Possibly, because the default delivery guarantee is at-least-once. A crash between writing data and saving progress leads to a repeated delivery. Idempotent writes such as upserts by key make duplicates harmless, and some connectors support exactly-once behaviour, which you must confirm individually.

The framework’s dead letter queue feature is provided for sink connectors. For sources, you handle problems through connector settings, transforms, and monitoring of task status and the source data itself.

Polling repeatedly queries tables for new or changed rows and cannot see deletes or intermediate changes. CDC reads the database’s transaction log, capturing every insert, update, and delete in order with low load and low delay.

In a source pipeline, transforms run before the converter serialises records into Kafka. In a sink pipeline, the converter first deserialises the records, and then transforms run before the sink task receives them.

No. Transforms operate on one record at a time. Joins, aggregations, and windowing belong in a stream processing application, with Connect moving data in and out.

In distributed mode, the remaining workers rebalance and take over the failed worker’s tasks, resuming from the saved offsets. In standalone mode, there is no failover, and the job stops until the process is restarted.

Use the REST interface to pause it. Its position is preserved, and when you resume it, work continues from where it stopped. For sinks, remember that lag grows during the pause and that Kafka’s retention must be long enough to keep the backlog.

It is convenient, but defaults may not suit your needs. In production, create important topics deliberately with the right partition counts and replication, or configure sensible defaults for automatic creation.

When no suitable connector exists for your system and you still want the framework’s scaling, recovery, and management benefits. If the logic is complex or highly specific, a standalone producer or consumer application may be simpler.

Kafka Connect gives you a standard, scalable, and recoverable way to connect Kafka with the rest of your systems. Source connectors read from external systems and publish into Kafka, tracking their own bookmarks called source offsets. Sink connectors read from Kafka and write to external systems, using normal consumer offsets. Both run as tasks on workers, both use converters and transforms in mirrored order, and both must be designed with failure in mind.

The best results come from choosing the right style of source, such as change data capture for accurate database replication, designing sinks so that repeated writes are harmless, sizing parallelism according to real limits, and watching for silent failures with end-to-end freshness checks and dead letter queues. Keep transforms simple, keep secrets safe, and let stream processing applications do the heavy thinking between the two ends.