What Is Apache Kafka?
A beginner-friendly guide to the distributed event streaming platform that lets companies move, store, and react to huge streams of data in real time, explained step by step with simple analogies.
Imagine a huge city post office that never closes. Thousands of people drop letters into it every second, and thousands of other people pick letters up whenever they are ready. The post office keeps every letter in neat, numbered order for days, so a person who was away can come back and read everything they missed. Nobody has to wait for anybody else, and nothing gets lost even if one clerk goes home sick. That, in a nutshell, is the idea behind Apache Kafka. In this guide we will build up the whole picture slowly, defining every new word the first time it appears.
1The Problem Kafka Was Built to Solve
Before learning a tool, it helps to feel the pain that made the tool necessary.
Too many systems, too many conversations
Think about a modern online shop. When a customer places an order, many different parts of the company need to know about it. The payment system must charge the card. The warehouse must pack the box. The email service must send a confirmation. The analytics team wants to count sales. The fraud team wants to look for suspicious behaviour. Each of these is a separate software system, often built by a different team.
The simplest way to connect them is to let every system talk directly to every other system. With four systems this is manageable. With forty systems it becomes a tangled ball of wires, because every new system needs a custom connection to every existing one. Engineers call this a point-to-point mess. When one system is slow or broken, the systems that depend on it slow down or break too.
Picture a school where every student passes notes directly to every other student. The classroom becomes chaos. Now imagine a single notice board in the hallway. Anyone can pin a note, and anyone can read the notes when they want. The notice board removes the need for everyone to know everyone else. Kafka plays the role of that notice board, only far bigger, faster, and much more careful about keeping notes safe.
The world moves in real time
There is a second problem. Businesses used to collect data all day and process it at night in one big batch. Today people expect instant results. A bank wants to block a stolen card while the thief is still at the till. A ride-sharing app wants to show the driver’s position moving smoothly on a map. Waiting until tomorrow is no longer acceptable.
What these companies need is a way to treat data as a never-ending flow of things that just happened, and to let many programs react to that flow the moment it arrives. This is called event streaming, and Kafka is one of the most widely used tools for it.
2So, What Is Apache Kafka?
Kafka in one sentence, and then in plain words.
A short definition
Apache Kafka is an open-source distributed event streaming platform. Let us unpack each of those words. “Open-source” means the code is free to use and is developed in public. “Distributed” means it runs on many computers working together as one team, rather than on a single machine. “Event” means a record that something happened. “Streaming” means the events keep flowing continuously rather than arriving in one lump. “Platform” means it is not just one small feature but a complete foundation you can build on.
In everyday terms, Kafka is a very fast, very reliable middleman. Programs that create data hand it to Kafka. Kafka stores that data in order and keeps it safe. Programs that want the data ask Kafka for it, whenever they like, at whatever speed they can handle.
Where it came from
Kafka was created at LinkedIn, the professional networking company, by a small team that included Jay Kreps, Neha Narkhede, and Jun Rao. LinkedIn had a flood of activity data, such as page views, searches, and messages, and its existing tools could not keep up. The team built Kafka to solve their own problem. It was released as open source around 2011 and became a top-level project of the Apache Software Foundation soon after. The name comes from the author Franz Kafka, chosen because the system is a tool built for writing.
Later, several of the original creators founded a company called Confluent to offer commercial products and services around Kafka. Many cloud providers today also sell managed versions, so you do not always have to run it yourself.
Kafka is often called a message queue, but that label is only partly right. A classic queue usually deletes a message once it is read. Kafka keeps its records for a chosen period, so many readers can read the same record, and can even read it again later. This one difference changes almost everything about how Kafka is used.
3The Core Building Blocks
Six words explain most of Kafka. Learn these and the rest becomes easy.
Event (Record)
A single fact that something happened, such as “order 4521 was placed at 10:03”. It usually has a key, a value, a timestamp, and optional labels called headers.
Topic
A named category of events, like a folder or a TV channel. “orders”, “payments”, and “page-views” would each be a separate topic.
Partition
A topic is split into smaller slices called partitions. Each slice is an ordered list that only grows at the end, and slices can live on different machines.
Offset
A number that marks the position of an event inside its partition, starting at zero and counting up. It works like a page number in a book.
Producer
Any program that writes events into Kafka topics. The order website is a producer of order events.
Consumer
Any program that reads events from Kafka topics. The email service is a consumer of order events.
Events in more detail
An event is the smallest unit in Kafka. The key is an optional label, for example a customer number, that helps decide where the event goes. The value is the actual content, such as the details of the order. The timestamp records when it happened. Kafka does not care what the value looks like. It simply stores bytes, so it can carry text, structured data, or anything else that your programs agree on.
Topics and partitions
A topic on its own would be limited by the size and speed of one machine. Partitions solve this. By cutting a topic into several partitions and spreading them across machines, Kafka lets many computers share the work of storing and serving the data at the same time. More partitions generally means more capacity to read and write in parallel.
A topic is like a supermarket with several checkout lanes. Each lane is a partition. Customers in one lane are served in the order they joined, but the lanes work side by side, so the store serves many more people than it could with a single lane. Order is guaranteed inside a lane, not across the whole store.
flowchart LR
P[Producer] --> T
T --> P0[Partition 0]
T --> P1[Partition 1]
T --> P2[Partition 2]
P0 --> C[Consumer]
P1 --> C
P2 --> C
Why offsets matter
Because every event in a partition has an offset, a consumer can remember exactly where it stopped. If it goes offline for an hour, it simply resumes from the last offset it processed. It can also jump back to an earlier offset to read old events again, which is very useful for fixing mistakes or building new features from history.
Many beginners assume Kafka guarantees order across an entire topic. It does not. Order is guaranteed only inside a single partition. If two events must stay in order, they need to share the same key so that they land in the same partition.
4Producers: The Writers
Producers are the programs that create the flow of events.
What a producer does
A producer takes a piece of information, wraps it into an event, chooses a topic, and sends it to Kafka. It does not need to know who will read the event or how many readers there are. It only needs to know where to send it. This is the source of Kafka’s flexibility: writers and readers are completely separated from each other.
How a producer picks a partition
When a topic has several partitions, the producer must decide which one receives each event. There are three common approaches. If the event has a key, Kafka runs the key through a formula so that the same key always goes to the same partition. If there is no key, the producer spreads events across partitions in a balanced way. A producer can also use its own custom rule if the business needs something special.
Think of sorting mail by postcode. All letters for the same postcode always go into the same bag, so they stay together and in order. The postcode is the key and the bag is the partition.
Acknowledgements: how sure do you want to be?
After sending an event, a producer can ask Kafka to confirm that it was received. There are three levels of care. In the fastest setting the producer does not wait for any confirmation, which is quick but risky. In a middle setting it waits until the main copy of the partition has saved the event. In the safest setting it waits until all the required copies have saved it. Safer settings take slightly longer, so teams pick the level that matches how precious the data is. Payment events deserve the safest setting, while a casual click counter might not.
Batching and compression
Sending events one at a time would waste effort, like driving a truck across town to deliver a single envelope. Producers therefore gather events into small batches and send them together. They can also compress each batch to save network bandwidth and disk space. Batching and compression are two of the main reasons Kafka can handle such large volumes.
5Consumers and Consumer Groups: The Readers
Reading in Kafka is designed so that many programs can share the work or each read everything.
Pull, not push
Kafka consumers ask for data when they are ready, instead of Kafka pushing data at them. This is called a pull model. It is friendly to slow readers, because a slow consumer is never overwhelmed. It simply takes what it can handle and comes back for more later.
Consumer groups
A consumer group is a team of consumer programs that share one job. Kafka divides the partitions of a topic among the members of the group, so each partition is read by exactly one member of that group at a time. If a topic has six partitions and the group has three members, each member handles two partitions. If the workload grows, you add more members and Kafka rebalances the partitions automatically.
Different groups are independent. The email service and the analytics system can each be their own group, and each one receives every event in the topic, at its own pace, with its own memory of where it stopped.
Imagine a large pile of homework to grade. A group of teachers splits the pile so each teacher grades a separate stack. That is one consumer group. A different group, the school inspectors, may want to look at every paper too. They work from their own copy of the list and do not disturb the teachers. Two groups, two independent readers of the same material.
flowchart TD
T[Topic with 4 partitions]
T --> G1A[Group A: Consumer 1]
T --> G1B[Group A: Consumer 2]
T --> G2A[Group B: Consumer 1]
G1A --> R1[Email service work]
G1B --> R1
G2A --> R2[Analytics work]
Committing offsets
As a consumer processes events, it periodically tells Kafka how far it got. This bookmark is called a committed offset. If the consumer crashes and restarts, or if another member takes over its partitions, the new reader starts from the bookmark. Getting the timing of these bookmarks right is important. Commit too early and you may skip events that were never really processed. Commit too late and you may process some events twice.
Within one group, you cannot usefully have more active consumers than partitions, because a partition is read by only one member at a time. Extra members would sit idle. This is why choosing the number of partitions is an important early design decision.
6Brokers and Clusters: The Machines Behind It
Kafka runs on a team of servers, and each has a name.
What is a broker?
A broker is a single Kafka server. It receives events from producers, stores them on its disks, and serves them to consumers. A broker is the working muscle of the system. One broker can handle a very large amount of traffic, but running just one would be dangerous, because if it failed everything would stop.
What is a cluster?
A cluster is a group of brokers working together. Partitions of each topic are spread across the brokers in the cluster, so the load is shared and no single machine holds all the data. Clusters can start small, perhaps three brokers, and grow to hundreds as the business grows. Adding brokers is how Kafka scales out.
Who keeps the cluster organised?
Someone has to keep track of which brokers are alive, which partition lives where, and which broker is in charge of each partition. In older versions of Kafka this job was done by a separate tool called ZooKeeper. Newer versions use a built-in mechanism called KRaft, short for Kafka Raft, in which a small group of Kafka nodes called controllers keeps this information and agrees on it among themselves. Recent Kafka releases have removed ZooKeeper entirely, which makes the system simpler to run because there is one less product to learn and maintain.
A cluster is like a fleet of delivery vans. Each van is a broker. The dispatch office, the controller, knows which van carries which parcels and reassigns them if a van breaks down. Customers never need to know which van holds their parcel, because the dispatch office directs them.
flowchart LR
PR[Producers] --> B1[Broker 1]
PR --> B2[Broker 2]
PR --> B3[Broker 3]
CT[Controller quorum] -.manages.-> B1
CT -.manages.-> B2
CT -.manages.-> B3
B1 --> CO[Consumers]
B2 --> CO
B3 --> CO
7Replication: Never Lose a Message
Computers fail. Kafka is designed with that fact in mind.
Copies of every partition
Each partition is copied onto several brokers. The number of copies is called the replication factor. A common choice is three, meaning every event is stored on three different machines. If one machine dies, two copies remain and the system carries on without losing anything.
Leaders and followers
For each partition, one copy is chosen as the leader and the others are followers. All reading and writing goes through the leader. The followers quietly copy everything the leader receives so they stay up to date. If the leader’s broker fails, the controller promotes one of the up-to-date followers to become the new leader, usually within moments. Producers and consumers automatically discover the change and continue.
In-sync replicas
Kafka keeps a list, called the in-sync replicas or ISR, of followers that are fully caught up with the leader. Only members of this list are eligible to take over. A setting called minimum in-sync replicas lets a team demand that a certain number of copies must have confirmed each write before it counts as successful. Combined with the safest acknowledgement level on the producer, this gives strong protection against data loss.
Consider an important contract that is photocopied and stored in three different offices. One office is the “official” one that people visit, and the other two quietly update their copies. If the official office floods, one of the other offices immediately becomes the official one, and nobody loses the contract.
sequenceDiagram
participant P as Producer
participant L as Leader (Broker 1)
participant F1 as Follower (Broker 2)
participant F2 as Follower (Broker 3)
P->>L: Send event
L->>F1: Replicate
L->>F2: Replicate
F1-->>L: Copy saved
F2-->>L: Copy saved
L-->>P: Acknowledge success
A replication factor of one means a single copy and no safety net. That is fine on a laptop for learning, but in production it turns any broker failure into permanent data loss for the affected partitions.
8How Kafka Works Inside
The design choices that make Kafka both fast and dependable.
The commit log
At the heart of Kafka is a very simple idea: an append-only log. A log here means a file where new records are only ever added at the end, never squeezed into the middle and never edited. Each partition is exactly one such log. Adding to the end of a file is one of the fastest things a disk can do, which is a big part of why Kafka is quick even though it writes everything to disk.
Segments
A single giant file would be awkward, so Kafka breaks each partition’s log into pieces called segments. When a segment reaches a certain size or age, Kafka closes it and starts a new one. Old segments can be removed as a whole, which is much cheaper than deleting individual events one by one.
Retention: how long data stays
Kafka does not delete an event when it is read. Instead, it keeps events for a period you choose, with seven days being a common default, or until a topic reaches a chosen size. After that, old segments are removed. Teams can also keep data forever if they wish. There is also a special mode called log compaction, in which Kafka keeps only the latest event for each key. This is handy for topics that represent the current state of things, such as the latest address of each customer.
Leaning on the operating system
Kafka deliberately uses the operating system’s page cache, which is memory that the computer already uses to speed up file access. Recently written data is often still in memory, so reading it is very fast. Kafka also uses a technique called zero-copy, which lets the operating system move data from disk to the network without shuffling it through the application’s own memory. Together these tricks cut a lot of wasted work.
Think of a ship’s logbook. The captain writes each entry on the next empty line and never erases anything. Entries are numbered, so anyone can say “show me from entry 500”. When a notebook fills up, a fresh one is started, and very old notebooks are archived or thrown away as complete books rather than page by page.
9Life of an Event: From Creation to Deletion
Follow a single order event through the whole system.
The event is born
A customer places an order. The shop’s website turns this into an event with a key, such as the customer number, and a value describing the order.
The producer chooses a partition
The producer uses the key to pick a partition of the “orders” topic, batches the event with others, compresses it, and sends it to the broker that leads that partition.
The leader stores it
The leader broker appends the event to the end of the partition’s log and assigns it the next offset number.
Followers copy it
The follower brokers fetch the new event and save their own copies. Once enough copies exist, the leader confirms success to the producer.
Consumers read it
The payment service, the warehouse, the email service, and the analytics system each belong to their own consumer group. Each pulls the event when ready and processes it independently.
Bookmarks are saved
Each consumer group records how far it has read by committing an offset, so it can resume correctly after any restart.
The event ages out
After the retention period, the segment holding the event is deleted, or the event is superseded if the topic is compacted.
sequenceDiagram
participant S as Shop website
participant K as Kafka orders topic
participant PAY as Payment service
participant WH as Warehouse
participant AN as Analytics
S->>K: Publish order event
K-->>PAY: Deliver when PAY asks
K-->>WH: Deliver when WH asks
K-->>AN: Deliver when AN asks
10Delivery Guarantees: How Sure Can You Be?
Networks and machines are imperfect, so Kafka lets you choose how strict to be.
Three promises
When a message travels between two systems, three outcomes are possible. It may arrive at most once, meaning it could be lost but is never repeated. It may arrive at least once, meaning it is never lost but might be repeated. Or it may arrive exactly once, meaning it is neither lost nor repeated. Each level costs a little more effort than the one before it.
| Guarantee | What can happen | Typical use |
|---|---|---|
| At most once | Some events may be lost, none are duplicated | Non-critical metrics where a tiny loss is acceptable |
| At least once | No event is lost, but some may be processed twice | Most everyday pipelines, especially when consumers can safely handle repeats |
| Exactly once | Every event is processed once and only once | Financial totals, billing, and other counting that must be precise |
Idempotence and transactions
Kafka supports exactly-once behaviour through two features. An idempotent producer gives each event a sequence number so that if a network hiccup causes a retry, the broker recognises the duplicate and ignores it. Transactions let a program write to several partitions, and record its own progress, as one all-or-nothing unit. Either every part of the transaction becomes visible, or none of it does.
Imagine sending a cheque by courier. With “at most once” you hand it over and hope. With “at least once” you keep sending copies until you hear back, so the receiver might get two. With “exactly once” the receiver stamps each cheque with a unique number and refuses to cash the same number twice.
Exactly-once is powerful but comes with extra overhead. Many teams choose at-least-once delivery and make their consumers safe against repeats, which is often simpler and just as correct for their needs.
11The Wider Kafka Ecosystem
Kafka’s core is small, but a family of tools grows around it.
Kafka Connect
A ready-made framework for moving data between Kafka and other systems, such as databases, search engines, and cloud storage, using reusable connectors instead of custom programs.
Kafka Streams
A library that lets an ordinary application process events as they flow, doing things like counting, joining, and filtering, without needing a separate processing cluster.
Schema Registry
A shared library of “shapes” that describe what events should look like, so producers and consumers agree on the format and can change it safely over time.
ksqlDB
A tool that lets people work with streams using familiar database-style questions rather than writing full programs.
MirrorMaker
A tool for copying topics from one Kafka cluster to another, useful for backups, moving between data centres, or sharing data across regions.
Why these tools matter
Without these helpers, every team would have to write and maintain its own code for common jobs, such as loading a database table into Kafka. Connect turns that into configuration. Streams lets developers add real-time logic as part of a normal application. Schema Registry prevents the classic problem where one team quietly changes the format of its events and breaks everyone downstream.
Example: Database to Search Engine
A company wants product changes stored in its database to appear instantly in its search engine. A Connect source reads changes from the database and writes them to a topic, and a Connect sink reads that topic and updates the search index. No custom glue programs are needed.
12Kafka Compared with Traditional Queues
A side-by-side view helps clear up the most common confusion.
| Aspect | Traditional message queue | Apache Kafka |
|---|---|---|
| After a message is read | Usually removed | Kept until the retention period ends |
| Number of readers | Each message goes to one worker | Any number of groups can read the same event |
| Replaying old data | Generally not possible | Easy, by moving back to an earlier offset |
| Ordering | Often per queue | Guaranteed within each partition |
| Scaling | Can be limited by a central node | Scales by adding partitions and brokers |
| Main strength | Simple task distribution and flexible routing | High-volume streaming and durable event history |
Neither style is “better” in every case. A traditional queue is a great fit when you simply want to hand tasks to workers and forget them. Kafka shines when many systems need the same stream of facts, when history matters, or when the volume is very large.
If your main need is “do this job once, then discard it”, a classic queue is often simpler. If your main need is “record what happened so many teams can use it now and later”, Kafka is usually the better choice.
13Real-World Uses of Kafka
Kafka appears in almost every industry that deals with fast-moving data.
Activity Tracking
This was Kafka’s original job at LinkedIn. Every click, view, and search is published as an event, so recommendation and analytics teams can learn from what users do almost instantly.
Ride-Sharing and Delivery
Companies such as Uber use Kafka to move location updates, trip requests, and pricing signals between systems, so that the driver and rider see a consistent, live picture.
Streaming Entertainment
Netflix has publicly described using Kafka as part of the pipeline that carries viewing and operational events, helping teams monitor the service and personalise what viewers see.
Banking and Payments
Banks use Kafka to stream transactions to fraud detection systems, to update balances, and to feed reports, with replication providing the durability that money data demands.
Internet of Things
Factories, vehicles, and smart devices produce constant readings. Kafka collects these readings, and downstream programs watch for danger signs such as overheating machines.
Log and Metrics Collection
Instead of every server shipping its logs to many tools, servers publish them to Kafka once, and monitoring, search, and archive systems each read what they need.
Event-Driven Microservices
In a system made of many small services, Kafka acts as the shared backbone. Services announce facts, and other services react, without needing to call one another directly.
14Advantages and Trade-offs
No tool is perfect. Knowing the costs is as important as knowing the benefits.
Advantages
- Very high throughput, handling large volumes of events
- Data is replicated and stored durably on disk
- Scales out by adding brokers and partitions
- Many independent readers can use the same data
- Events can be replayed for fixes and new features
- A large community, many connectors, and managed cloud options
Disadvantages / Trade-offs
- Has a learning curve, with many settings to understand
- Running a cluster well takes operational skill, or a managed service
- Ordering is only within a partition, which needs careful key design
- Changing the number of partitions later can disturb key-to-partition mapping
- Overkill for small, simple messaging needs
15Security in Kafka
Because Kafka often carries sensitive business data, protecting it matters.
Encryption in Transit
Data travelling between clients and brokers, and between brokers themselves, can be encrypted with TLS so eavesdroppers see only scrambled bytes.
Authentication
Clients prove who they are before connecting, using methods such as certificates or username-and-password style mechanisms, so strangers cannot join the cluster.
Authorization
Access control lists decide who may read from, write to, or manage each topic, following the principle of giving each program only the access it needs.
Encryption at Rest
Disks holding the data can be encrypted, usually at the operating system or cloud storage level, to protect against stolen hardware.
Auditing
Logging who did what helps teams investigate incidents and satisfy compliance rules.
Security in Kafka is like protecting an office building. Encryption in transit is an armoured van for deliveries, authentication is the ID check at the door, authorization is the key card that opens only certain rooms, encryption at rest is the locked safe, and auditing is the security camera log.
16Monitoring and Health Checks
A cluster you cannot see is a cluster you cannot trust.
Kafka exposes many measurements that teams collect and display on dashboards. A handful of them tell you most of what you need to know about health.
Consumer Lag
How far behind a consumer group is compared with the newest event. Rising lag means readers cannot keep up.
Under-replicated Partitions
The count of partitions whose followers have fallen behind. Anything above zero deserves attention.
Throughput
How many bytes and events per second enter and leave each broker and topic.
Request Latency
How long producers and consumers wait for the broker to respond. Sudden jumps often signal overloaded disks or networks.
Disk Usage
How full the brokers’ storage is. Retention settings must match the space available.
Broker and Controller Status
Whether every broker is online and whether the controller group has a healthy leader.
Teams commonly export these measurements to popular monitoring dashboards and set alerts so that a human is told before customers notice a problem. Application logs from producers and consumers, kept alongside the cluster metrics, give the full story when something goes wrong.
If you set up only one alert, make it consumer lag. It is the clearest early sign that some part of your pipeline is falling behind the real world.
17Design Patterns and Anti-patterns
Good habits to copy and bad habits to avoid.
Helpful patterns
Event Sourcing
Store every change as an event and rebuild the current state by replaying them. Kafka’s durable log fits this naturally.
Publish and Subscribe
Producers announce facts to a topic and any interested service subscribes, keeping systems loosely connected.
Change Data Capture
Turn each database change into an event so other systems can stay in sync without repeatedly querying the database.
Dead Letter Topic
Send events that repeatedly fail processing to a separate topic for later inspection, so one bad event does not block the whole stream.
Stream Processing Pipeline
Chain several small steps, each reading one topic and writing another, to transform raw data into useful results.
Habits to avoid
Problem
Using Kafka as a general-purpose database and expecting to look up individual records quickly by arbitrary fields.
Why It’s Harmful
Kafka is designed for ordered streams, not for flexible searching. Lookups become slow, awkward, and expensive.
Correct Approach
Let Kafka carry the events and let a proper database or search engine, fed by Kafka, answer the questions.
Problem
Sending very large files, such as videos, straight through Kafka events.
Why It’s Harmful
Huge events strain brokers, slow replication, and waste memory for every reader.
Correct Approach
Store big files in object storage and send only a small event that points to them.
Problem
Putting every kind of event into one giant topic with no clear naming or structure.
Why It’s Harmful
Consumers must sift through irrelevant data, permissions become hard to manage, and retention cannot be tuned per kind of data.
Correct Approach
Create focused topics with clear names and agreed event formats.
18Best Practices and Common Mistakes
Practical lessons that save teams from painful surprises.
Best practices
Plan partitions early
Estimate future traffic and choose enough partitions to allow your consumer groups to grow, without going wildly beyond what you need.
Use replication factor three in production
Combine it with a sensible minimum in-sync replicas setting and the safest producer acknowledgement level for important data.
Choose keys thoughtfully
Pick a key that keeps related events together while spreading load evenly. A key that is too popular creates a “hot” partition that gets overloaded.
Define event formats and manage change
Use a schema registry and agree on compatibility rules so producers and consumers can evolve without breaking each other.
Make consumers safe to repeat
Design processing so that seeing the same event twice does no harm. This makes recovery from failures much easier.
Monitor and rehearse failures
Watch lag and replication health, and practise what happens when a broker disappears, before it happens for real.
Mistakes beginners often make
Too few limits how far you can scale readers. Too many adds overhead on brokers and slows recovery. Aim for a thoughtful middle, based on real expected load.
If retention is longer than your disks can hold, brokers run out of space. If it is too short, slow consumers may miss data that has already been deleted.
When consumers join or leave a group, partitions are reassigned, and processing may pause briefly. Frequent, unexpected rebalances usually point to consumers that are too slow or unstable.
Adopting Kafka for a tiny project can add more complexity than it removes. Match the tool to the size of the problem.
19Frequently Asked Questions
Not in the usual sense. Kafka stores events durably and in order, and you can keep them for a long time, but it is not built for flexible searching or updating individual records. It is best seen as the flowing backbone that feeds databases and other systems.
No. Large companies made it famous, but small teams also use it, especially through managed cloud services. Still, if your needs are tiny, a simpler queue may be easier.
If the data is replicated, a follower on another broker is promoted to leader and work continues. Producers and consumers reconnect to the new leader automatically, often within seconds.
Yes, as long as the events are still within the retention period. A consumer can move its bookmark back to an earlier offset or timestamp and read forward again.
Only within a single partition. To keep related events in order, give them the same key so they go to the same partition.
Newer Kafka versions use the built-in KRaft approach and no longer need ZooKeeper. Older clusters may still use it, so check the version you run and the documentation for it.
Traditional brokers focus on delivering a message to a worker and then removing it, with rich routing rules. Kafka focuses on keeping an ordered, replayable log that many groups can read, and is generally chosen for high-volume streaming and event history.
It can if it is configured carelessly, for example with a single copy of each partition or with the weakest acknowledgement level. With proper replication and safe producer settings, it is designed to protect data against ordinary failures.
No. Several cloud providers and vendors offer managed Kafka, where they handle servers, upgrades, and much of the monitoring, and you focus on your topics and applications.
Begin by understanding topics, partitions, producers, consumers, and consumer groups with a small local setup. Then explore replication and delivery guarantees, and finally the ecosystem tools such as Connect and Streams.
20Summary and Key Takeaways
Apache Kafka is a distributed event streaming platform that acts as a fast, durable, and shared notice board for the data of a whole organisation. Producers write events into named topics, topics are split into partitions that spread across a cluster of brokers, and consumer groups read those events at their own pace, keeping bookmarks called offsets. Replication protects the data when machines fail, and an append-only log design makes the whole system quick. Around this core sits a rich ecosystem of tools that move, shape, and analyse streams. Used with care, Kafka lets many teams build real-time systems that are loosely connected, resilient, and ready to grow.
Key Takeaways
- Kafka is an event streaming platform — it stores and delivers streams of events in order, for many readers, in real time.
- Topics, partitions, and offsets — topics organise events, partitions provide parallelism and order, and offsets mark each event’s position.
- Producers and consumers are decoupled — writers and readers never talk directly, which keeps systems flexible and independent.
- Consumer groups share the work — partitions are divided among group members, and different groups each get the full stream.
- Replication provides safety — leaders and followers, in-sync replicas, and careful acknowledgement settings protect against data loss.
- The log design makes it fast — append-only writes, segments, the page cache, and zero-copy give high throughput.
- Choose the right tool for the job — Kafka excels at high-volume streaming and replayable history, but may be more than you need for simple task queues.