Almost every company needs to move data between Kafka and other systems. Doing it by hand, again and again, is wasteful.

Data lives in many places

Apache Kafka is a system that stores streams of messages, called records, in named channels called topics. But a company’s data does not start in Kafka. It starts in databases, log files, cloud storage, search engines, messaging systems and business applications. To make use of Kafka, you must bring data in from those places, and often push data back out to them again.

The hand-written way

You could write a custom program for each system: one that reads rows from a database and produces Kafka records, another that consumes records and writes them to cloud storage. Each program would need code for connecting, retrying after failures, remembering where it stopped, scaling to more machines, handling bad records and reporting its health. Every team would solve those same problems again, in slightly different and slightly buggy ways.

What Kafka Connect is

Kafka Connect is a framework that ships as part of Apache Kafka. It runs as its own service, separate from the Kafka brokers, and its job is to move data between Kafka and other systems using reusable plug-ins called connectors. You do not write code for the common cases. You describe what you want in a configuration, submit it, and Kafka Connect does the rest. It also looks after scaling, fault tolerance, offset tracking and monitoring for you.

Three words appear everywhere in Kafka Connect, and they are easy to confuse. Let us separate them clearly.

flowchart TD
    subgraph Cluster[Kafka Connect Cluster]
        W1[Worker 1]
        W2[Worker 2]
        W3[Worker 3]
    end
    CFG[Connector configuration] --> CON[Connector instance]
    CON --> T1[Task 0]
    CON --> T2[Task 1]
    CON --> T3[Task 2]
    T1 --> W1
    T2 --> W2
    T3 --> W3
    T1 --> K[(Kafka topics)]
    T2 --> K
    T3 --> K
        

Source and sink connectors

Connectors come in two directions. A source connector pulls data from an outside system and writes it into Kafka topics. A sink connector reads records from Kafka topics and writes them into an outside system. The word “task” applies equally to both: a source task reads from the outside world and produces records, while a sink task consumes records and delivers them outward.

Workers can run alone for simple jobs, or together for real production systems.

Standalone mode

In standalone mode, a single worker process runs all connectors and tasks. Its settings and the connector settings are supplied as files when the process starts. Offsets, which record how far the connector has progressed, are kept in a local file. This mode is easy to start and good for learning, testing and small, single-machine jobs. Its weakness is obvious: if the machine dies, everything stops, and nothing takes over automatically.

Distributed mode

In distributed mode, many workers join into one cluster. They find each other through Kafka itself, using a shared group identifier. Connector configurations are submitted through a web interface and stored safely inside Kafka topics. If a worker disappears, its tasks are handed to the remaining workers, and if you add a worker, the cluster spreads the load onto it. Production systems almost always use this mode.

Where distributed mode keeps its memory

A distributed cluster stores three kinds of information in Kafka topics, so that no single worker holds the only copy. One topic holds the connector and task configurations. One holds the offsets of source connectors. One holds the current status of connectors and tasks. Because these topics live in Kafka, they are replicated and durable, and a newly started worker can rebuild its knowledge simply by reading them.

The configuration is the written order. It tells Kafka Connect what to run, where to read, where to write and how to behave.

A configuration is a set of named settings

A connector configuration is simply a collection of key and value pairs, where each key is a setting name and each value is your choice. In distributed mode you send it to the cluster as a small JSON document through the REST interface. In standalone mode you place it in a properties file. Either way, it contains the same kinds of settings, and Kafka Connect reads them to create the connector.

Two layers of settings

Beginners often mix up two different sets of settings, so it helps to name them clearly.

flowchart LR
    WC[Worker configuration] --> D[Defaults for all connectors]
    CC[Connector configuration] --> O[Settings for one job]
    D --> R[Effective behaviour of a task]
    O --> R
        

No matter which system you connect to, a handful of settings appear in almost every connector configuration.

Connector-specific settings

Beyond the shared settings, each connector defines its own. A database connector will ask for the database address, the user name, which tables to include and how often to check for new data. A cloud storage connector will ask for the bucket name, the file format and how large each file should be. These settings are documented by the connector’s authors, and they are the part of the configuration where you will spend most of your time.

Validation before you start

Kafka Connect can check a proposed configuration before running it. Each connector describes its own settings, including which are required, what type each has, and what values are allowed. A validation request returns a list of problems, so you can fix mistakes before creating the connector. This saves the frustration of a job that fails moments after you submit it.

A task is the smallest piece of running work in Kafka Connect. Understanding it explains scaling, failure and speed.

The manager and the workers, again

When you create a connector, Kafka Connect starts a connector instance. This instance is a coordinator. It looks at your configuration and at the external system, and decides how many pieces the work should be divided into. Each piece is described by a small set of settings called a task configuration. Kafka Connect then creates one task for each task configuration and hands them out to the workers in the cluster.

What a task actually does

Tasks are mostly stateless

A task does not keep important information only in its own memory. Its configuration lives in Kafka, and its progress lives in offsets stored in Kafka. That is why a task can be stopped on one worker and started on another without losing its place. This design is the reason Kafka Connect can recover so smoothly from failures.

The name suggests “run this many tasks.” The truth is more subtle, and it matters.

An upper limit, not a target

The setting tasks.max tells the connector the largest number of tasks it may create. The connector itself then decides how many tasks are actually useful and returns that many task configurations. If the connector can only split its work into three pieces, setting the limit to ten will still give you three tasks. The other seven would have nothing to do, so they are never created.

How the real number is decided

Why sink connectors have a natural ceiling

A sink connector’s tasks read from Kafka the same way ordinary consumers do, and they share a consumer group. In a consumer group, each partition is read by at most one member at a time. So if a topic has six partitions, more than six sink tasks would leave the extra tasks idle. For sink connectors, the number of useful tasks is capped by the number of partitions in the topics being read.

Why some source connectors can use only one task

Some source systems cannot be read in parallel without breaking correctness. A connector that follows a single ordered change log, for example, may need one task so that changes stay in order. Such a connector will create one task even if you set a higher limit. Always read the connector’s own documentation to learn what limits its parallelism.

From submitting a configuration to seeing data flow, several steps happen quickly. Let us walk through them.

sequenceDiagram
    participant U as You
    participant R as REST Interface
    participant L as Cluster Leader Worker
    participant C as Connector Instance
    participant W as Workers
    U->>R: Submit connector configuration
    R->>L: Store configuration in Kafka
    L->>C: Start connector
    C->>C: Decide how to split the work
    C-->>L: List of task configurations
    L->>W: Assign tasks to workers
    W->>W: Start tasks
    W-->>R: Report status running
        

The connector watches for change

The connector instance stays alive after planning. It can watch the external system for changes that require a new plan. For example, if a new table appears in a database, the connector can ask the runtime to re-plan its tasks so that the new table is covered. This is called a task reconfiguration, and it is one way Kafka Connect adapts to a changing world without you touching anything.

Restarting without losing or repeating work depends on remembering progress. Source and sink tasks do this differently.

Source offsets

A source task describes its position in the outside system in its own words, for instance “file name plus line number” or “database change log position.” The Kafka Connect runtime stores these source offsets in the offset topic, and hands them back to the task when it restarts, so the task can continue right after the last confirmed position.

Sink offsets

A sink task reads from Kafka, so its progress is simply a normal consumer position: how far it has read in each partition. These positions are stored in Kafka as consumer group offsets, under a group named after the connector. Kafka Connect only commits them once the sink task has confirmed that the records were written, which is what allows safe recovery.

Delivery guarantees in plain words

By default, Kafka Connect gives at-least-once delivery. If a task fails after writing some data but before recording its progress, the work between the last saved position and the failure point may be done again after recovery. Duplicates are therefore possible, so the receiving system should be able to tolerate them, for example by using unique keys so that a repeated write simply overwrites itself. Newer versions add stronger options for certain connectors, but availability depends on the version and on the specific connector.

Connectors read and write in the outside system’s language. Converters and transforms translate to and from Kafka’s world.

Converters

A converter changes data between Kafka Connect’s internal representation and the bytes stored in Kafka. There is one for keys and one for values, chosen with the settings key.converter and value.converter. Typical choices include plain text, JSON, and formats that work with a schema registry such as Avro or Protobuf. The converter is not part of the connector plug-in, so you can use the same connector with different data formats just by changing this setting.

Single Message Transforms

A Single Message Transform, often shortened to SMT, is a small step that changes one record at a time as it passes through the connector. You list the ones you want, in order, in the configuration. Common uses include renaming fields, removing sensitive fields, adding a timestamp, changing the topic a record is sent to, and picking out only one field.

Order matters

Transforms run in the order they are listed, and each one receives the result of the previous one. A transform that renames a field must therefore come before a transform that refers to the new name. Also remember that transforms work one record at a time. For anything that needs joining, counting or looking across many records, use a stream processing tool instead.

Every connector and every task has a status. Knowing what each status means makes troubleshooting far easier.

Newer versions of Kafka Connect also add more descriptive states, such as one for a connector that has been stopped with its position preserved, and one for a restart in progress. The exact list depends on your version, so it is worth confirming in the documentation of the version you run.

flowchart LR
    A[UNASSIGNED] --> B[RUNNING]
    B --> C[PAUSED]
    C --> B
    B --> D[FAILED]
    D --> B
        

Connector state versus task state

A connector and its tasks have separate states. A connector can be running while one of its tasks has failed. This surprises many beginners: the top-level status looks healthy, yet data is no longer flowing for part of the work. Always look at both the connector status and the status of each task.

In a distributed cluster, the assignment of connectors and tasks to workers is not fixed. It adjusts as the cluster changes.

Why rebalancing happens

Workers form a group, much like consumers do. Whenever a worker joins, leaves or crashes, or whenever a connector is added, removed or reconfigured, the group performs a rebalance. During a rebalance, the cluster decides again which worker should run which connectors and tasks, so that the load stays even and nothing is left without a home.

flowchart TD
    A[Worker 2 crashes] --> B[Group notices missing member]
    B --> C[Rebalance starts]
    C --> D[Leader recomputes assignment]
    D --> E[Tasks from Worker 2 given to Worker 1 and Worker 3]
    E --> F[Tasks resume from stored offsets]
        

Eager and cooperative rebalancing

In older versions, a rebalance was “stop the world”: every task on every worker was stopped, assignments were recomputed, and everything restarted. This caused a noticeable pause for all connectors even when only one thing had changed. Modern versions use incremental cooperative rebalancing. Only the tasks that actually need to move are stopped and restarted, while the rest keep running. This greatly reduces the disruption of routine cluster changes.

The grace period for returning workers

If a worker disappears briefly, for example during a quick restart, moving all its tasks elsewhere and then moving them back would be wasteful. The cluster therefore waits for a short, configurable delay before reassigning the work of a missing worker. If the worker comes back within that time, it simply resumes its tasks. This delay is a trade-off: a longer wait avoids needless movement, while a shorter wait recovers faster from permanent failures.

When data volume grows, you need more throughput. There are three levers, and they work together.

The right order of thinking

Real data is messy. A good configuration decides in advance what happens when a record cannot be processed.

Where errors can occur

Kafka Connect can meet trouble at several points: while converting a record, while running a transform, or inside the connector’s own logic when talking to the outside system. Some of these errors are temporary, such as a brief network glitch. Others are permanent, such as a record that simply cannot be understood. The two need different treatment.

Key error settings

flowchart TD
    A[Record arrives at task] --> B{Processing succeeds?}
    B -- Yes --> C[Write to destination]
    B -- No --> D{Tolerance allows skipping?}
    D -- No --> E[Task fails and stops]
    D -- Yes --> F[Log the error]
    F --> G{Dead letter queue configured?}
    G -- Yes --> H[Copy bad record to dead letter topic]
    G -- No --> I[Skip record]
    H --> J[Continue with next record]
    I --> J
        

Strict versus tolerant

Error handling in these settings covers problems in conversion, transforms and record delivery. It does not cover every failure. For example, if the outside system is completely unreachable, the connector’s own retry behaviour and the task failure mechanism apply instead. Read the connector’s documentation to see how it treats connection errors.

Connector configurations often need passwords and keys. Storing them in plain text is a common and dangerous habit.

Why plain text is a problem

In distributed mode, connector configurations are stored in a Kafka topic and can be read through the REST interface. If a password sits inside the configuration in plain text, anyone who can read the configuration can read the password. It may also appear in logs, tickets and copied examples. This risk grows with every person and tool that touches the configuration.

Config providers

Kafka Connect supports config providers, small components that fetch a value from a safer place when the connector starts. Instead of writing the password itself, you write a placeholder that says “get this value from that source.” The actual secret is resolved when the task starts and is not stored in the configuration topic. Providers exist for reading from files, from folders, and from environment variables, and external secret managers can be plugged in too.

Under the hood, tasks use ordinary Kafka producers and consumers. You can tune them, within limits.

Tasks use normal Kafka clients

A source task’s records are sent to Kafka by a producer that the runtime manages. A sink task’s records are read by a consumer that the runtime manages. Both clients have many settings, such as batch sizes, waiting times and buffer sizes, which decide how fast and how efficiently data moves. By default, they inherit values from the worker configuration.

Per-connector overrides

Sometimes one connector needs different client settings from the rest. Kafka Connect allows this through override settings placed inside the connector’s own configuration, using special prefixes for producer, consumer and admin clients. Whether overrides are permitted is controlled by a worker-level policy, so an administrator can allow all, some, or none of them. This prevents any single connector from changing settings that might harm the others.

The offset commit interval

Tasks do not save their progress after every record; that would be far too slow. Instead, progress is committed periodically. A shorter interval means less repeated work after a failure but more overhead. A longer interval reduces overhead but increases the amount of data that may be processed twice if a task fails. Choose based on how costly duplicates are for your destination.

In distributed mode, everything you do to connectors and tasks goes through a simple web interface.

Every worker exposes a REST interface, usually on port 8083 unless changed. You can talk to any worker in the cluster, and it will pass the request to the right place. You can use command-line tools, scripts, deployment pipelines or graphical consoles to call it. Here are the everyday actions.

Configuration as code

Mature teams keep connector configurations in a version-controlled repository, review changes like any other code, and let an automated pipeline apply them to the cluster. This gives history, reviews, easy rollbacks and consistent settings between test and production environments. Manual edits through the interface are convenient for experiments but easy to forget and hard to reproduce.

Kafka Connect touches Kafka, external systems and secrets all at once, so it deserves careful protection.

A pipeline you cannot see is a pipeline you cannot trust. Kafka Connect exposes status and metrics for both.

Status first

The simplest check is the status view from the REST interface. It shows each connector’s state, each task’s state and any error description for failed tasks. Many teams poll this regularly and raise an alert when any task is not running. Because a failed task does not restart by itself, this alert is the single most valuable one you can create.

Metrics that matter

flowchart LR
    T[Tasks and workers] --> J[Metrics from the workers]
    T --> S[Status from the REST interface]
    J --> D[Dashboard]
    S --> D
    D --> A[Alerts on failed tasks, high lag and errors]
        

Logs still matter

Worker logs contain the detailed stories behind failures, such as connection problems and conversion errors. When a task fails, its status includes a summary, and the logs contain the full explanation. Collect logs from all workers in one searchable place so you can follow a task even after it moves between workers.

Most Kafka Connect problems fall into a few familiar patterns. Here is a calm way to diagnose them.

A calm checklist

Habits that keep connectors healthy, and traps that catch teams again and again.

Good habits

Anti-patterns to avoid

Let us see how configuration and tasks play out in typical situations.

The connector is the planner defined by your configuration. It decides how to split the work. Tasks are the units that actually move data, and they run on the workers.

No. It is only an upper limit. The connector decides the real number based on the natural units of work, such as tables, files or topic partitions.

Sink tasks read as a consumer group, and each partition is read by at most one member. Extra tasks beyond the partition count would stay idle.

After a short grace period, the cluster rebalances and assigns the crashed worker’s tasks to other workers. Tasks resume from their stored offsets, so little or no work is lost.

Not by default. A failed task stays failed until you restart it, so monitoring and alerting on task states is very important.

Yes, in distributed mode you can send an updated configuration through the REST interface. The tasks are reconfigured, which causes a brief pause for that connector.

In distributed mode, in an internal Kafka topic used for configuration storage. In standalone mode, they come from files given at startup.

Yes. The default guarantee is at-least-once, so some work may be repeated after a failure. Design destinations to handle repeats, and check your connector’s documentation for stronger options.

It is a topic where a sink connector copies records that it could not process, optionally with error details, so that the pipeline can keep moving and the bad records can be reviewed later.

Use config providers. You register a provider in the worker configuration and refer to secrets by placeholder in the connector configuration. The real values are read from a protected location when the connector starts.

No. Transforms are meant for small, per-record changes. Use a stream processing application for joins, aggregations and heavy logic.

Standalone runs everything in a single process with local files and no failover. Distributed runs a cluster of workers that share work, store state in Kafka topics and recover from failures, and it is the standard choice for production.

Kafka Connect lets you move data between Kafka and other systems by writing configuration instead of code. A connector configuration names the plug-in, sets limits such as the maximum number of tasks, describes the systems and topics involved, and chooses converters, transforms and error behaviour. The connector instance acts as the planner and divides the work into tasks, and the tasks, running across the workers of a cluster, do the actual moving of data. Progress is stored as offsets in Kafka, which lets tasks move between workers and recover from failures. With careful choices about limits, error handling, secrets, security and monitoring, connectors become a reliable and low-effort backbone for data integration.