The name is a full description. Read it slowly, one word at a time, and the whole idea falls into place.

A message in Kafka is one record: a small package holding a key, a value, a timestamp and optional labels called headers. A transform is a change applied to that package. The word single tells you the most important rule: the change looks at exactly one record at a time. It cannot peek at the record before it, wait for the record after it, or combine two records into one.

The core promise

An SMT takes one record in and hands back either one record or nothing. If it hands back one record, that record may be changed a lot or a little. If it hands back nothing, the record is dropped and disappears from the pipeline. That is the entire contract.

Where SMTs live

SMTs are part of Kafka Connect, the framework that moves data between Kafka and other systems such as databases, search engines and cloud storage. They are not part of plain producers or consumers, and they are not part of Kafka Streams. You attach them to a connector, and they run inside the Connect workers alongside it. They were added to Kafka in version 0.10.2, released in 2017, after users kept asking for a simple way to do small edits without writing whole programs.

What people use them for

Why SMTs were created

Before SMTs existed, small edits were a real annoyance. Suppose a database connector produced a field called “cust_nm” but every other team expected “customerName”. The only fix was to write a separate consumer that read every record, renamed the field and wrote everything back to a second topic. That doubled the traffic, added a service to run and monitor, and delayed the data.

The Kafka community noticed that most such edits were tiny and identical in spirit: rename, drop, mask, re-route. SMTs were added in Kafka 0.10.2 so that these chores could be done inside the connector itself, in the same thread, on the same record, with no second topic. The change turned an entire category of small services into a few lines of settings.

SMTs only make sense inside Kafka Connect, so here is a short tour of the place where they live.

Kafka Connect is a ready-made framework for copying data into Kafka and out of Kafka. Instead of writing a custom program for every database or storage service, you install a connector, give it settings, and let the framework run it. Connect handles the boring parts: splitting work across machines, remembering how far it has read, restarting after failures and reporting its health.

Source and sink

The moving parts

flowchart LR
    DB[("Source system: database")] --> SC["Source connector task"]
    SC --> K["Kafka topic"]
    K --> KC["Sink connector task"]
    KC --> ES[("Target system: search index")]
    

Standalone and distributed modes

Connect can run in two ways. In standalone mode, one process runs everything, connector settings are read from files given at startup, and its progress markers are kept in a local file. It suits a laptop, a test or a single-machine job. In distributed mode, several workers form a cluster, connector settings and progress are stored in special Kafka topics, and you manage connectors through the REST interface. If one worker fails, the others take over its tasks.

SMT settings work identically in both modes. The difference is only where the settings are stored and how you update them. Production systems almost always use distributed mode.

Progress markers and repeated records

Connect keeps track of how far each task has got. A source task periodically saves a marker, called an offset, describing the position in the outside system. A sink task commits how far it has read in the Kafka topic. If a task crashes, it restarts from its last saved marker and may re-handle a few records. By default this gives at-least-once delivery: nothing is lost, but a few records can appear twice. Newer Kafka versions also offer exactly-once support for source connectors, starting with version 3.3.

This matters for SMTs because a transform may see the same record twice. That is fine as long as the transform is a pure function of the record: run it twice and you get the same output. It is one more reason to avoid transforms that keep counters or send messages elsewhere.

The exact position of an SMT decides what data it sees. Getting this right removes a lot of confusion.

On the way in: source connectors

A source connector task first reads data from the outside system and builds a record. Before that record is turned into bytes and written to Kafka, the chain of SMTs runs on it. The converter comes after the transforms. So an SMT working on a source connector sees a structured record, not bytes.

flowchart LR
    SYS[("External system")] --> ST["Source task builds a record"]
    ST --> T1["SMT 1"]
    T1 --> T2["SMT 2"]
    T2 --> CV["Converter: record to bytes"]
    CV --> TP["Kafka topic"]
    

On the way out: sink connectors

A sink connector task reads bytes from Kafka. The converter first turns those bytes into a structured record. Only then do the SMTs run, and finally the cleaned-up record is handed to the sink connector to write into the target system. On the sink side the converter comes before the transforms.

flowchart LR
    TP["Kafka topic"] --> CV["Converter: bytes to record"]
    CV --> T1["SMT 1"]
    T1 --> T2["SMT 2"]
    T2 --> SK["Sink task writes to target"]
    SK --> TS[("Target system")]
    

Why this order matters in practice

• On the source side, the transform edits the record as the connector understands it, before any Kafka format is applied. If the connector produces structured data, the transform can name fields directly.
• On the sink side, the transform sees data as Kafka stored it, after decoding. If Kafka holds plain JSON without a schema, the transform sees loose maps instead of a strict structure. That difference affects which transforms work well, as chapter 10 explains.
• The same SMT can be reused on both sides, but its results can differ because the incoming shapes differ.

What a transform can actually see

A record inside Connect carries more than a key and a value. Knowing every part helps you decide which transform to use.

One useful detail: on a source connector, the record has not yet been written to Kafka, so it has no offset. On a sink connector, the record was already read from Kafka, so its topic, partition and offset are all known. This is why an “insert the offset” option makes sense only on the sink side.

Every strength and every limit of SMTs comes from one rule: they look at one record at a time.

What that allows

Because each record is handled alone, SMTs need no memory. They have no counters, no lookups of earlier records and no waiting rooms. That makes them extremely cheap: a transform is little more than a small function that runs in a fraction of a millisecond. It also makes them safe to run in parallel and safe to repeat. If a task restarts and replays a few records, running the same transform again gives the same result.

What that forbids

The same rule rules out a whole family of jobs. An SMT cannot count records, add up amounts, work out an average or detect that three events happened close together. It cannot join two topics. It cannot sort records, hold one back until another arrives, or split one record into several. It cannot call out to a slow web service for every record without harming the entire pipeline.

The connection to stateless processing

In stream processing, the word stateless describes work that needs no memory between events. SMTs are stateless by design. If you have read about stateless and stateful operations in Kafka Streams, you can think of SMTs as the stateless corner of the picture: filtering, mapping and re-labelling. Anything stateful, such as counting, windowing and joining, belongs in Kafka Streams, ksqlDB or a stream processor, not in an SMT.

Order is preserved

Because an SMT handles records one by one and returns one record for each, the relative order of surviving records inside a partition stays the same. Record A still comes before record B. This is a valuable property. It is exactly what you lose when you use tools that reshuffle data, and it is why SMTs are safe to place on pipelines where ordering matters, such as change events from a database.

The only ways an SMT can disturb ordering are by changing the key (so later records may go to a different partition), by changing the topic name (so records flow into a different topic), or by dropping some records. Everything else is order-safe.

Using a built-in SMT means adding a few settings to a connector. No programming is required.

The four ideas you need

Key versions and value versions

Every Kafka record has a key and a value, and most built-in transforms come in two flavours: one that works on the key and one that works on the value. The names make this plain, with the word Key or Value at the end. Choosing the wrong flavour is a very common beginner mistake, and the symptom is usually “nothing changed” or “the wrong part changed”.

Where the settings live

SMT settings belong to a single connector. They are not global. Two connectors on the same cluster can use completely different transforms, or none at all. Changing a connector’s settings and updating it through the REST interface restarts its tasks with the new chain, while other connectors keep running.

Checking your settings before going live

Connect offers a validation call that checks a connector’s settings without starting it. It reports missing options, misspelled names and wrong types. Using it in your deployment pipeline catches simple mistakes early, long before a bad chain touches production data.

A small glossary to keep nearby

Apache Kafka ships with a set of ready-made transforms. This chapter covers the ones that change what is inside a record’s value or key.

The field-shaping toolbox

A closer look at masking

Masking is among the most common uses of SMTs, because it lets a pipeline hide private data before it leaves Kafka. The transform swaps the chosen field with a harmless placeholder. Be clear about what it is: masking hides a value; it does not protect it with a secret key. Once masked, the original is gone from that copy of the record, which is exactly what you want, but it is not encryption and it cannot be reversed.

A closer look at flatten and hoist

Nested data is common in modern applications. An order might hold an address block, which holds a city and a postcode. Many targets, such as relational tables, cannot store nested blocks directly. Flatten turns the block into separate top-level fields with names like the address block, a separator and the city. Hoisting is used on the other side: a bare number or text value cannot become a column without a name, so hoisting gives it one.

Worked examples in plain words

Numbers and names make the ideas concrete. Imagine a customer record that arrives with a nested address, a phone number and a spelling nobody likes. Here is what each transform does to it.

Some transforms do not touch the data fields much. They change where a record goes, how it is labelled, or how its time is written.

Keys and partitioning

The key of a record decides which partition it lands in, and the partition decides ordering. Sometimes a source produces records with no useful key. ValueToKey copies chosen fields from the value into the key, and ExtractField can then reduce that new key to a single plain value. Together they let you give every record a key, such as a customer identifier, so that related records stay together and in order.

Routing to other topics

On a source connector, routing changes which Kafka topic the record is written to. On a sink connector, it changes which topic name the sink believes the record came from, which many sinks use to choose a table, an index or a folder name. That second effect is the reason routing SMTs are so popular for sinks.

Time formats

TimestampConverter converts a time value between several forms: a number counting milliseconds since 1970, a text string in a chosen pattern, and proper timestamp, date or time types. Different systems disagree on how to write time, and this SMT is the translator that makes them agree.

Headers

Headers are small labels attached to a record without changing its value. Built-in transforms can copy or move a field into a header, insert a fixed header, or drop headers you no longer want. Headers are useful for routing hints, tracing identifiers and source tags, because downstream tools can read them without opening the value.

Real pipelines are rarely uniform. Some records need a transform, and others must be left alone. Predicates make that possible.

The Filter transform

The built-in Filter transform drops records. By itself, it is a blunt tool: it removes every record that reaches it. That sounds useless until you learn that it is designed to be paired with a predicate, a yes-or-no test that decides whether a transform should run for a given record.

Predicates

Predicates were added to Kafka Connect in version 2.6. You declare a predicate with an alias, choose its type, and then tell a transform to run only when the predicate is true, or only when it is false by turning on a “negate” switch.

Common uses

• Drop test traffic: filter out records that carry a “test” header, so demo data never reaches the warehouse.
• Handle deletions: remove tombstones for a sink that cannot delete rows, or keep them only for a sink that can.
• Apply masking selectively: mask personal fields only on topics that contain personal data.
• Route only some records: apply a topic rename to matching topics and leave the rest alone.

A worked scenario with predicates

Consider a sink connector that reads three topics: customers, orders and heartbeats. The team wants personal fields masked on customers, tombstones dropped everywhere because the target cannot delete rows, and heartbeat records ignored completely. A tidy design uses three small rules instead of one confusing one.

A single SMT does one small thing. Real work comes from lining several of them up, and the order you choose changes the result.

How a chain flows

The first transform receives the record built by the connector. Its output goes to the second transform, and so on down the list. If any transform returns nothing, the record is dropped and the later transforms never see it. Whatever comes out of the last transform is what the converter or the sink receives.

flowchart TB
    IN["Record from connector"] --> A["Step 1: unwrap the change envelope"]
    A --> B{"Step 2: is it a test record?"}
    B -- "yes" --> D["Dropped"]
    B -- "no" --> C["Step 3: mask personal fields"]
    C --> E["Step 4: rename the topic by date"]
    E --> OUT["Record continues to converter or sink"]
    

A worked example in words

Suppose a database connector produces change events. Each event wraps the useful row inside an envelope that also holds the previous version of the row and some bookkeeping. A sensible chain looks like this.

Keeping chains short

Every extra step adds a little work to every record and one more thing that can break. Long chains are also hard to read. If you find yourself writing more than four or five steps, stop and ask whether the source data could be cleaner, whether a different connector setting does the job, or whether the logic belongs in a stream processor.

Three chain designs to practise with

Designing a chain gets easier with practice. Here are three small scenarios, each with the order you would choose and the reason behind it.

Reading a chain backwards

A useful debugging trick is to work backwards. Start from the record you expect at the end and ask, “what did the last step need to see?” Then ask the same about the step before it. When you find the step where the real record differs from what the next step expects, you have found the fault. This is much faster than staring at the whole chain at once.

Most SMT trouble comes from a mismatch between the shape of the data and the shape the transform expects.

Two kinds of data

Inside Connect, a record’s value can arrive in two forms. With a schema, the data is a structured object with named, typed fields, and Connect knows that “age” is a whole number and “name” is text. Without a schema, the data is a loose map of names and values, and Connect can only guess at the types. Avro and Protobuf converters usually give schemas. Plain JSON often does not.

Common surprises

• Missing fields: a transform that names a field which some records do not have may fail or skip. Records are not always uniform.
• Null values: a field can be present and empty. Some transforms handle this gracefully, and others raise an error.
• Deletion markers: a tombstone has a key but no value. Transforms that assume a value exist can fail on it. Guard with a predicate.
• Nested versus flat: a field name that works on a flat record does not exist on a nested one until it has been flattened.
• Schema changes: when the source adds a field, the schema changes and some transforms react. Test what happens after an upgrade.

Nulls and tombstones in detail

Two special cases trip up more pipelines than any others. A null field is a field that exists but holds no value, such as a customer with no middle name. A tombstone is a whole record whose value is empty, used in Kafka to say “the thing with this key has been deleted”. Log-compacted topics rely on tombstones to clean up old data, and database connectors emit them after a row is deleted.

Many transforms assume the value is present. When a tombstone arrives, such a transform may fail, pass it through unchanged or produce an odd result. Meanwhile, a sink that cannot delete rows may choke on the marker. There are three sensible strategies.

These patterns appear in many production Kafka Connect setups. Each one is small, yet each removes a whole extra service.

When no built-in transform fits, Kafka lets you build your own. It is easier than most people expect, and easier to get wrong than most people expect.

When a custom SMT makes sense

Build your own only after checking three things. First, ask whether a chain of built-in transforms can do the job. Second, ask whether a change to the source or the sink could avoid the need. Third, ask whether the logic really is single-record and stateless. If all three answers point to “a small custom rule”, then a custom SMT is a good choice. Typical examples are converting a company-specific code format, applying a house naming standard, or changing a deeply nested payload in a way no built-in handles.

What a custom SMT is

A custom SMT is a small Java class that follows a public contract called Transformation. The contract has a handful of parts:

Built-in transforms usually have a Key version and a Value version of the same logic, sharing one core. Following that convention in your own code makes your transform feel familiar to every operator.

Getting it onto the workers

• Package the class, with only the libraries it needs, into a JAR file.
• Copy it into a folder listed in each worker’s plugin path. Connect loads each plugin in its own isolated space, so its libraries cannot clash with others.
• Restart the workers so they discover it, then reference it by its full class name in the connector settings.
• Keep versions in step across all workers. A cluster where half the machines have the old plugin behaves unpredictably.

Good habits for custom code

A design walk-through for a small custom SMT

To make the idea concrete, imagine a company where phone numbers arrive in twelve different styles, some with spaces, some with dashes, some with country codes and some without. The sink needs one clean style. No built-in transform does this, so the team decides to build one. They approach it in a calm, repeatable way.

Sooner or later, one strange record will arrive that a transform cannot handle. What happens next is a choice you make in advance.

Two behaviours

The setting errors.tolerance decides how Connect reacts when a record fails during conversion or transformation.

The safety nets

• Retries: for failures that may pass with time, Connect can retry for a set period before giving up.
• Logging: failures can be written to the worker log. Turning on message content in logs helps debugging but may reveal private data, so use it with care.
• Dead letter queue: for sink connectors, failed records can be sent to a separate Kafka topic so nothing is lost. Extra headers can explain why each record failed, including the stage and the error message.

flowchart TB
    R["Record enters converter and transforms"] --> OK{"Did it succeed?"}
    OK -- "yes" --> NEXT["Record continues to the sink"]
    OK -- "no" --> TOL{"errors.tolerance"}
    TOL -- "none" --> STOP["Task stops with an error"]
    TOL -- "all" --> DLQ{"Dead letter queue set?"}
    DLQ -- "yes" --> SEND["Record and error details go to the dead letter topic"]
    DLQ -- "no" --> SKIP["Record is skipped and logged"]
    

What is not covered

These settings cover failures in converting and transforming records. They do not cover every kind of problem. If the sink cannot reach its target system, or the source cannot connect to its database, that is a different class of failure, and it follows the connector’s own retry rules. Also, dead letter queues exist only for sink connectors. For source connectors, failures are logged or skipped but not saved to a topic.

Reading a dead letter record

A record in the dead letter topic keeps its original key and value, so you can see exactly what arrived. When context headers are switched on, Connect also attaches labels that describe the failure: the connector name, the task, the stage where it failed (for example a transformation), the class name of the step that raised the error, and the error message. This makes the dead letter topic a self-explaining inbox rather than a mystery pile.

A healthy routine has four parts. Someone is alerted when the topic grows. Someone reads a few records and finds the pattern. The cause is fixed in the source, the settings or the transform. Finally, the saved records are replayed into the pipeline once the fix is live, so no data is lost for good.

SMTs are one of four ways to change data around Kafka. Choosing the right one saves a lot of effort.

flowchart TB
    Q1{"Is the change small and per record?"} -- "no" --> Q2{"Does it need counting, joining or windows?"}
    Q1 -- "yes" --> Q3{"Is data moving through Kafka Connect?"}
    Q3 -- "yes" --> SMT["Use an SMT"]
    Q3 -- "no" --> Q4["Use a stateless step in Streams or a simple consumer"]
    Q2 -- "yes" --> Q5["Use Kafka Streams or ksqlDB"]
    Q2 -- "no" --> Q6["Use consumer code or a custom service"]
    

The dividing questions

Signs that you have outgrown SMTs

Most pipelines start with one or two harmless transforms. Over months, requests pile up. Recognizing the signs early saves painful rewrites.

How to migrate calmly

Moving logic out of an SMT chain does not have to be dramatic. Keep the light steps (unwrap, mask, rename) in the connector. Move the heavy steps into a Kafka Streams or ksqlDB job that reads the connector’s output topic and writes a new topic. Point the sink at that new topic. Run old and new side by side, compare counts, and switch over when they agree.

A well-chosen SMT is close to free. A badly chosen one can quietly become the slowest part of your pipeline.

Why they are usually cheap

An SMT runs inside the same thread as the task that owns the record, so there is no network hop and no extra queue. Most built-in transforms do a few small operations on data already in memory. The extra time per record is typically tiny compared with the time spent reading from the source, writing to the target or talking to Kafka.

How they scale

Because SMTs keep no state, they scale exactly as the connector scales. If you raise tasks.max and the source or sink can be divided further, each new task brings its own copy of the transform chain. There is nothing to move, restore or synchronize between tasks. This is the same reason stateless processing is easy to spread across many machines.

What can make them slow

Practical rules

• Put filters near the start of the chain so later steps handle fewer records.
• Prefer value-only or key-only versions rather than transforming both when only one matters.
• Measure the connector’s throughput with and without the chain on a test topic.
• If a transform needs outside data, bring that data into Kafka and use a stream processor instead.

A back-of-the-envelope example

Numbers help build feel. The figures below are illustrative, not measurements from a real system, so measure your own chain before relying on them.

Five steps at ten microseconds each cost fifty microseconds per record. At ten thousand records per second, that is about half a second of processor time for every second of clock time, so roughly half of one processor core. That is real, though still modest. Now imagine one custom step that calls a web service taking five milliseconds. That single step costs five thousand microseconds per record, one hundred times the whole rest of the chain, and at ten thousand records per second it would need fifty seconds of work every second. The task could never keep up, and adding more tasks would only hammer the service harder.

SMTs are small, but they sit on the path of every record. That makes them both a useful place to protect data and a place where mistakes can hide.

What to watch

Logging with care

Good logs name the connector, the task and the transform alias that failed. Avoid printing whole records into logs unless you must, because records may carry the very personal data you set out to mask. If you do turn on message logging to debug a problem, turn it off again afterwards and treat the log files as sensitive.

Security points

Change management and audits

Because transforms can hide or alter data, they deserve the same discipline as application code. Treat every change as a small release. Write down why it was made, who approved it and how it was tested. Keep the settings in a repository so an audit can show exactly which transforms were active on any given date.

For regulated data, add two extra habits. First, keep a written map of which connectors touch personal data, and which of them mask it. Second, test the masking regularly by reading a sample of output records and confirming that private fields are truly gone. A rule that was correct last year may have been broken by a new field added at the source.

A few repeatable designs make SMT-based pipelines calm and predictable. A few shortcuts make them painful.

Patterns that work

Anti-patterns to avoid

These lessons show up again and again in teams that run Kafka Connect for years.

The mistakes beginners meet first

A short launch checklist

Before any new transform chain reaches production, walk through these steps in order.

The projects and products below are publicly documented, and they show the same few SMT ideas at work in different settings.

Short answers to the questions beginners ask most often.

If you keep only one page of this guide, keep this one.

Single Message Transforms are small, stateless edits that Kafka Connect applies to each record as it enters or leaves Kafka. They are perfect for plumbing work such as renaming, masking, routing and filtering, and they are the wrong tool for anything that needs memory or more than one record at a time.