Why Use Kafka?
A beginner-friendly guide to the problems Apache Kafka solves, the ideas behind it, and how to decide whether your system really needs it.
Imagine a busy city with hundreds of small shops, a few big warehouses, and a handful of delivery companies. If every shop had to phone every warehouse and every delivery company personally, whenever something happened, the phone lines would jam and nobody would get the right message on time. Now imagine the city builds one giant, well-organised post office. Every shop drops its news there, and anyone who cares about that news comes to pick it up whenever they are ready. Apache Kafka is that post office for the software world. This guide explains, in simple words, why so many companies decided they could not live without it.
1The Problem Before Kafka
To understand why Kafka exists, we first need to feel the pain it was built to remove.
Small systems start simple
Think of a small online store. It has a website, a payment system, and a warehouse system. When a customer buys something, the website tells the payment system and the warehouse system directly. With only three parts, this works fine. Each part knows the address of the others, and a developer can draw the whole picture on a single sheet of paper.
The trouble starts when the business grows. Soon the company adds an email system, an analytics dashboard, a fraud checker, a recommendation engine, a mobile app, and a reporting tool for managers. Every new system wants to know about things that happen in the other systems. A customer places an order, and suddenly ten different systems all need to hear about it.
The spaghetti problem
When every system talks directly to every other system, the number of connections grows very fast. With 3 systems you have around 3 connections. With 10 systems you may have up to 45. With 50 systems you may have more than a thousand. Engineers call this a “spaghetti architecture” because the wires are tangled and nobody can follow them anymore.
Picture a school where every student must personally tell every other student the daily news. Nobody would finish class. A school notice board fixes this: one person writes the news once, and everyone reads it when they have time. Kafka plays the role of that notice board, but a very fast, very organised one.
What goes wrong in the spaghetti
Direct connections create many painful problems. If the email system is down for ten minutes, the website may fail while trying to talk to it, and a customer cannot buy anything. If the analytics team wants a copy of the same data, a developer must change the website code to send it one more place. If one system is slow, everything waiting on it becomes slow too. Each new connection is a new place for bugs to hide.
Many beginners think the main reason for Kafka is speed. Speed is a bonus. The deeper reason is that Kafka untangles how systems talk to each other, so teams can grow without breaking one another.
2What Is Kafka in Plain English?
Kafka is a system for moving, storing, and sharing streams of events between applications.
A short history
Kafka was created at LinkedIn, the professional networking company, around 2010. LinkedIn had exactly the spaghetti problem described above, with enormous amounts of activity data such as page views, searches, and messages that many internal systems needed. The engineers built Kafka to solve it, and in 2011 they released it as open source. Today it is a top-level project of the Apache Software Foundation and is used by thousands of organisations around the world.
What is an event?
An event is a small record that says “something happened.” A customer placed an order. A driver’s location changed. A temperature sensor reported a new reading. A user clicked a button. Each event usually contains a few facts, such as what happened, when it happened, and who or what was involved.
What Kafka actually does
Kafka lets one application publish events and lets many other applications read them. The publisher does not need to know who is reading. The readers do not need to know who published. Kafka sits in the middle, keeps the events safely on disk, and hands them out to whoever asks. Because it is designed as a distributed system, it runs on several computers working together, which gives it both power and safety.
Think of a newspaper. The journalists write articles once. The printing press stores copies. Thousands of readers pick up the paper at their own time and their own pace. A reader who is on holiday can read the old copies later. Kafka works in the same way for data: write once, store safely, read by many, read any time.
3The Building Blocks You Must Know
Before we discuss the reasons to use Kafka, let us learn its small vocabulary. Each word is simple once you see the picture.
Event (Message)
A single record of something that happened, like “order 4521 was placed at 10:02 a.m.”
Producer
Any application that writes events into Kafka. The website that reports new orders is a producer.
Consumer
Any application that reads events from Kafka. The email system that sends receipts is a consumer.
Broker
One Kafka server. A group of brokers working together is called a cluster.
Topic
A named category of events, like a labelled folder. For example, “orders” or “payments”.
Partition
A topic is split into smaller slices called partitions so many machines can share the work.
Offset
The position number of an event inside a partition, like a page number in a book.
Consumer Group
A team of consumers that share the reading work of a topic so that each event is handled once per team.
A picture of the pieces
The diagram below shows how these words fit together. Producers write into topics, the brokers store the partitions, and different consumer groups read the same events independently.
flowchart LR
P1["Website Producer"] --> T["Topic: orders"]
P2["Mobile App Producer"] --> T
T --> B["Kafka Cluster - Brokers and Partitions"]
B --> C1["Consumer Group A: Email Service"]
B --> C2["Consumer Group B: Analytics"]
B --> C3["Consumer Group C: Fraud Check"]
A topic is like a TV channel. Producers are the studios that broadcast. Consumers are viewers with their own television. Partitions are like several parallel lanes that let the channel carry a lot of shows at once. The offset is like a bookmark that remembers where each viewer stopped watching.
4Reason One: Decoupling Your Systems
The first and most important reason to use Kafka is that it lets systems stop depending on each other directly.
What does decoupling mean?
Two systems are “coupled” when one cannot work properly without the other being available, fast, and speaking the same language. Decoupling means putting a middle layer between them so each can live its own life. The producer only needs to talk to Kafka. The consumer only needs to talk to Kafka. Neither needs to know the other exists.
Why this matters in daily engineering
Suppose a new team wants to build a loyalty-points system that rewards customers for each purchase. Without Kafka, they must ask the order team to change their code and send events to the new system. That means meetings, planning, testing, and a release. With Kafka, the order events are already available in a topic. The loyalty team simply subscribes and starts reading. The order team does not have to change a single line.
A radio station does not know who is listening. It broadcasts, and any radio can tune in. A new listener buying a radio does not require the station to rebuild its tower. Kafka gives your data the same freedom: anyone who is allowed can tune in.
Time decoupling
Kafka also separates systems in time. A producer can write events at 10 a.m., and a consumer can read them at 10:05 a.m. or even the next morning. If the consumer is being upgraded or has crashed, the events wait safely in Kafka. When the consumer returns, it continues where it stopped. Without this, the producer would need to wait, retry, or lose the data.
Decoupling is the reason Kafka is so common in microservice designs. Small services can be built, changed, and released by different teams without a giant coordination effort.
5Reason Two: Handling Huge Amounts of Data
Some systems generate more data in a minute than a normal tool can handle in an hour. Kafka was built for that scale.
Why ordinary tools struggle
A traditional database is wonderful for storing and searching records, but it is usually not designed to accept hundreds of thousands of tiny writes every second from many sources while also serving many readers. A single machine has limits on how fast its disk and network can work. When the load grows beyond that, the system slows down or falls over.
How Kafka scales
Kafka splits every topic into partitions and spreads those partitions across many brokers. Each broker handles only its share of the traffic. When you need more capacity, you add more brokers and more partitions, and the work spreads out further. This is called horizontal scaling: instead of buying one enormous computer, you add many ordinary ones.
Kafka is also very efficient in how it writes to disk. It always adds new events to the end of a file, which is one of the fastest things a disk can do. It also groups small events into batches and can compress them, so the network carries less. These choices let a modest cluster move enormous volumes of events with very low delay.
A single-lane road jams quickly. A highway with many lanes lets far more cars travel at the same time. Partitions are the lanes of Kafka’s highway, and brokers are the toll booths that share the traffic.
Consumers scale too
The same idea works on the reading side. If one consumer cannot keep up, you start more consumers in the same group. Kafka automatically divides the partitions among them, so the reading work is shared. If one consumer dies, the others take over its share.
flowchart TD
T["Topic: page-views"] --> P0["Partition 0"]
T --> P1["Partition 1"]
T --> P2["Partition 2"]
P0 --> C1["Consumer 1"]
P1 --> C2["Consumer 2"]
P2 --> C3["Consumer 3"]
Example: Activity tracking at a social network
Every click, like, search, and scroll on a large social platform becomes an event. Millions of users produce billions of events daily. Kafka absorbs this flood and lets many teams read it for feeds, safety checks, and product analysis at the same time.
6Reason Three: Data That Lasts and Can Be Replayed
Kafka is not a pipe where data disappears after being read. It is a log that remembers.
Messages are kept, not thrown away
In many older messaging tools, once a consumer reads a message, the message is deleted. Kafka behaves differently. It stores events on disk for a period you choose, such as seven days by default, or for weeks, months, or even forever if you decide so. Reading an event does not remove it. Ten different consumers can read the same event, and it stays there for others.
The power of replay
Because events stay, a consumer can go back in time. Each consumer remembers its own offset, the bookmark of what it has read. If you fix a bug in your analytics code and want to recalculate last week’s numbers, you can simply move the bookmark back and read the events again. If a brand-new system joins next month, it can start from the beginning and build its own view of history.
Think of a video streaming service versus live television. Live television is gone once it is broadcast. A streaming library keeps every episode so you can rewatch, skip, or start from season one. Kafka is the streaming library of your business events.
Why replay is such a big deal
Recovering from mistakes
If a consumer has a bug and processes events wrongly, you fix the bug and replay the events. No data is lost and no manual repair is needed.
Onboarding new systems
A new service can read the full history and build up its own state without asking any other team to export data.
Auditing and investigation
Because a trustworthy record of everything that happened is kept in order, teams can investigate what occurred and when.
Testing with real history
Teams can replay real past traffic into a test environment to see how a new version would have behaved.
Retention is a setting you control. Keeping data longer costs more disk space, so choose a period that matches how far back you truly need to look.
7Reason Four: Real-Time Processing
Businesses no longer want yesterday’s report. They want to react in seconds.
Batch versus streaming
In the older style, called batch processing, data is collected all day and processed once at night. Managers see results the next morning. This is fine for some tasks, but useless when a stolen credit card is being used right now or when a delivery driver is stuck in traffic right now.
Streaming means handling each event as soon as it arrives. Kafka delivers events with very small delay, often just a few milliseconds, so applications can respond almost instantly. A fraud checker can block a suspicious payment before it completes. A ride-hailing app can update the driver’s position on your map continuously.
Batch processing is like reading a monthly magazine to learn what happened in the world. Streaming is like watching a live news channel. Both have their place, but only one helps you cross the road safely right now.
Processing while data flows
The Kafka ecosystem includes tools that let you filter, combine, count, and transform events while they are moving. For example, you can count how many orders arrived in each five-minute window, or join a stream of clicks with a table of user profiles. This lets you build live dashboards, alerts, and smart features without waiting for a nightly job.
Example: Live fraud detection
A bank sends every card transaction as an event. A fraud service reads the stream and compares each payment with the customer’s usual behaviour. If a purchase looks strange, it raises an alert within moments, before the money leaves.
Example: Live delivery tracking
A delivery company receives location updates from thousands of vehicles every few seconds. Kafka carries them to the map service, the arrival-time estimator, and the customer notification system at once.
8Reason Five: Reliability and Fault Tolerance
Computers fail. Networks break. A good system keeps working anyway.
Copies of your data
Kafka does not keep only one copy of your events. Each partition is copied to several brokers. This is called replication. One copy is the leader, which handles the reading and writing, and the others are followers, which quietly keep themselves up to date. If the broker holding the leader crashes, one of the followers is promoted and takes over, usually within seconds.
Imagine an important school notebook. Instead of keeping just one, the teacher makes three identical copies and stores them in three different cupboards. If one cupboard is locked or burned, the notebook still lives on. Replication does the same for your data.
flowchart LR
P["Producer"] --> L["Broker 1 - Leader copy"]
L --> F1["Broker 2 - Follower copy"]
L --> F2["Broker 3 - Follower copy"]
L --> C["Consumer"]
Confirmations you can trust
Producers can ask Kafka to confirm a write only after enough copies have safely stored it. If you want the strongest safety, you wait for all in-sync copies. If you want more speed and can accept a little risk, you wait for fewer. This gives you a dial between safety and speed that you set based on how precious the data is.
Consumers survive too
When a consumer crashes, Kafka remembers its last bookmark. A replacement consumer picks up from that point. In a consumer group, the remaining members automatically take over the work of the fallen one. The system heals itself instead of waiting for a human to wake up at night.
A common production setup keeps three copies of each partition across three different machines, ideally in different racks or availability zones, so that the loss of a whole zone does not stop the business.
9Reason Six: Order and Many Readers at Once
Two quiet superpowers of Kafka are keeping events in order and serving many independent readers.
Ordering guarantees
Order matters in real life. If a customer creates an account, changes the address, and then deletes the account, processing these steps in a different order would create a mess. Kafka guarantees that inside one partition, events are stored and delivered in the exact order they were written.
To keep related events together, producers attach a key to each event, such as a customer number. Kafka sends all events with the same key to the same partition. This means all steps for one customer stay in order, while different customers are handled in parallel on different partitions. You get both order and speed.
At a bank with many counters, each customer’s file always goes to the same clerk. That clerk handles the file’s papers in the order they arrive, while other clerks handle other customers at the same time. Nobody’s file gets shuffled, and the queue still moves quickly.
Many independent readers
In a classic queue, a message goes to one worker and then disappears. In Kafka, every consumer group gets its own full copy of the stream. The billing team, the analytics team, and the search team can all read the same “orders” topic without disturbing each other. One slow reader never slows the others, because each keeps its own bookmark.
Kafka does not promise a global order across all partitions of a topic. It only promises order within each partition. If your design needs strict order for everything, you must plan the keys carefully or reconsider the design.
10How an Event Travels Through Kafka
Let us follow one order from the moment a customer clicks “Buy” until many systems have reacted to it.
The producer creates an event
The order service builds a small record with the order number, items, and time, and attaches the customer number as the key.
Kafka picks a partition
Using the key, Kafka decides which partition of the “orders” topic should hold the event, so all events for that customer stay together.
The leader stores it
The broker leading that partition appends the event to the end of its log and gives it the next offset number.
Followers copy it
The follower brokers fetch the event and store their own copies. Once enough copies exist, the producer receives a confirmation.
Consumers read it
The email service, the analytics job, and the fraud checker each read the event from their own bookmark, at their own speed.
Bookmarks move forward
Each consumer group records how far it has read. If it stops and restarts, it continues from that spot. The event itself stays on disk until the retention period ends.
sequenceDiagram
participant O as Order Service
participant K as Kafka Cluster
participant E as Email Service
participant A as Analytics
O->>K: Publish order event
K-->>O: Confirmation after copies are stored
E->>K: Ask for new events
K-->>E: Deliver order event
A->>K: Ask for new events
K-->>A: Deliver same order event
11Kafka Compared with Other Tools
A fair question is: why not just use a database or a traditional message queue? The answer depends on the job.
Different tools, different strengths
Kafka does not replace databases, and it is not the best answer to every messaging need. The table below shows where each kind of tool shines, using simple language.
| Question | Kafka | Traditional Message Queue | Database |
|---|---|---|---|
| Main purpose | Store and stream a continuous log of events | Hand a task to one worker | Store and query current facts |
| After reading | Event stays until retention ends | Message usually removed | Data stays until changed |
| Replay history | Yes, by moving the bookmark | Usually not | Only if you designed history tables |
| Many independent readers | Natural and cheap | Possible but less natural | Possible but adds load |
| Very high volume | Designed for it | Depends on the product | Often becomes a bottleneck |
| Complex searching | Not its job | Not its job | Excellent |
| Simple task distribution | Works, but may be more than needed | Excellent | Awkward |
A database is like a filing cabinet that holds the latest version of every document. A message queue is like a to-do tray where each note is picked up and thrown away. Kafka is like a security-camera recording: everything is kept in order, many people can watch, and you can rewind whenever you wish.
Kafka and databases often work side by side. Kafka carries the changes between systems, and each system keeps its own database for searching and reporting.
12Real-World Ways Companies Use Kafka
These patterns appear again and again across industries.
Streaming platforms and recommendations
Video and music services record what viewers play, pause, and skip. This stream feeds recommendation systems, so the next suggestion reflects what you did a moment ago.
Ride-hailing and logistics
Trip requests, driver locations, and price changes move as events. Matching, pricing, and maps all read the same streams and react in seconds.
Banking and payments
Every transaction becomes an event that flows to fraud checks, account balances, notifications, and compliance records. The permanent log also helps with audits.
Online retail
Orders, stock levels, and shipping updates travel through Kafka so the website, the warehouse, and the customer messages stay in step without direct wiring.
Internet of Things
Factories, vehicles, and smart devices send thousands of small readings every second. Kafka collects them and lets teams watch for faults and predict maintenance needs.
Log and metrics collection
Applications send their logs and health numbers into Kafka, which then delivers them to search tools and monitoring dashboards. This keeps monitoring separate from the applications being watched.
Moving data between databases
Change-data-capture tools watch a database for changes and publish each change as an event, so other systems and data warehouses can stay up to date without heavy copying jobs.
13Advantages and Trade-offs
Every technology has a price. A wise engineer looks at both sides.
Advantages
- Untangles connections between systems and teams
- Scales out by adding more brokers and partitions
- Keeps events for replay, recovery, and new consumers
- Delivers events in real time with low delay
- Survives machine failures through replication
- Supports many independent readers of the same data
Disadvantages / Trade-offs
- Has a learning curve, with many new ideas to master
- Running a cluster needs care, monitoring, and planning
- Order is only guaranteed within a partition
- Not built for complex searching or one-off queries
- Can be too heavy for very small or simple projects
- Data is spread across systems, so debugging needs good tooling
Managed services reduce the burden
Many cloud providers and companies offer Kafka as a managed service. They handle the servers, upgrades, and repairs, and you focus on your events. This lowers the operational burden, though it adds a monthly cost and some limits on fine tuning. Modern Kafka versions also run without a separate coordination tool, which makes the cluster simpler to operate than in earlier years.
14When You Should Not Use Kafka
Knowing when to say no is as valuable as knowing when to say yes.
Problem
A small team with one application and a few hundred messages a day installs a full Kafka cluster because it sounds modern.
Why It’s Harmful
The team spends more time running Kafka than building the product. The cost and complexity are far bigger than the problem.
Correct Approach
Start with a simple queue or direct calls. Move to Kafka when the number of systems, the volume, or the need for replay clearly grows.
Problem
Using Kafka as the main database, expecting to search and filter old events with flexible questions.
Why It’s Harmful
Kafka is built to read events in order, not to answer random questions quickly. Searching becomes slow and painful.
Correct Approach
Let Kafka carry the events and let a database or search engine store the results in a form that is easy to query.
Problem
Needing an instant answer to a request, such as “is this password correct?”, and sending it through Kafka.
Why It’s Harmful
Kafka is designed for events flowing one way. Forcing a request and answer through it adds delay and complexity.
Correct Approach
Use a normal direct call for questions that need an immediate reply, and use Kafka for events that others may react to.
Choosing Kafka because it is popular. Popularity is not a requirement. Write down your real needs first, such as volume, number of readers, replay, and delay, then decide.
15Best Practices and Common Mistakes
A few habits keep a Kafka setup healthy from day one.
Good habits
Name topics clearly
Use readable, consistent names so that anyone can guess what a topic holds.
Choose keys wisely
Pick a key that groups events that must stay in order, such as a customer or an order number.
Plan partitions early
Think about how much parallel reading you will need, because changing the count later can affect ordering.
Keep events small and clear
Send only what readers need, in an agreed format that all teams understand and can evolve safely.
Make consumers repeat-safe
Design readers so that seeing the same event twice does not cause harm, because retries and replays can happen.
Set retention on purpose
Decide how long data must be kept and configure it, rather than accepting a default without thought.
Frequent mistakes
Beginners often create too few partitions and later discover they cannot add more readers. Others forget to watch consumer lag, which is the gap between the newest event and the last one read, and only notice a problem when customers complain. Some skip an agreed event format, so one team quietly changes a field and breaks three other teams. A shared, versioned description of each event type prevents this pain.
Because events may be delivered more than once in some situations, always ask, “what happens if this reader sees the same event again?” If the answer is “something bad,” redesign the reader.
16Security and Monitoring Basics
An event stream often carries sensitive business facts, so it must be protected and watched.
Protecting the stream
Kafka supports encrypting data while it travels between applications and brokers, so that nobody listening on the network can read it. It also supports proving who is connecting, called authentication, and deciding what each connection may do, called authorisation. For example, the order service may write to the “orders” topic, but the marketing tool may only read from it, and nobody outside the company can touch it at all.
Think of an office building. Encryption is the sealed envelope that hides a letter on its journey. Authentication is the guard checking your identity card at the door. Authorisation is the list that says which floors your card can open.
Watching the health
Good teams watch a few numbers closely. Consumer lag shows whether readers are falling behind. Broker disk usage warns before storage runs out. The number of under-replicated partitions tells you whether copies are missing. Message rates and delays show whether the system is busy or slowing. Alerts on these numbers let engineers repair a small problem before it becomes an outage.
17Frequently Asked Questions
Not in the usual sense. Kafka stores events durably and in order, but it is not made for flexible searching. Think of it as a shared log, and keep your query-friendly data in a database.
They are cousins, but different. A queue usually hands each message to one worker and removes it. Kafka keeps events and lets many groups of readers read them independently, including replaying old ones.
Not when configured with sensible safety settings. With replication and confirmations from enough copies, data survives the loss of a machine. Losses usually come from settings that favour speed over safety.
As long as you decide. The common default is about seven days, but you can keep data for months, for years, or by size limit. Some topics are even kept forever.
Yes, especially with a managed service. But if you have only a few systems and low volume, a simpler tool may serve you better and cost less effort.
A topic is the named category of events. A partition is one slice of that topic. Splitting a topic into partitions lets many machines and many readers share the work.
No. Kafka stores events, so a consumer can be offline and catch up later. This time separation is one of Kafka’s biggest benefits.
It was born in a big company, but its ideas help any team that has many systems sharing data. The right question is not the size of your company but the shape of your data flow.
18Summary and Key Takeaways
We began with a city full of shops phoning each other and ended with a single well-run post office. That is the heart of why people use Kafka. It replaces a tangle of direct connections with one shared, durable stream of events. Systems become independent, data becomes replayable, volume becomes manageable, reactions become real-time, and failures become survivable. At the same time, Kafka asks for care, learning, and a genuine need. Use it when your data flows between many systems, when volume or replay matters, or when you want to react quickly. Choose something simpler when your needs are small.
Key Takeaways
- Decoupling — producers and consumers never talk directly, so teams can change and grow independently.
- Scale — partitions and brokers spread the work, so capacity grows by adding machines.
- Durability and replay — events are kept on disk, so readers can recover, catch up, or start from the beginning.
- Real-time reaction — events reach readers within moments, enabling live alerts, dashboards, and smart features.
- Reliability — replicated copies and automatic takeover keep the system running when machines fail.
- Order and sharing — order is kept within each partition, and many groups read the same events independently.
- Right tool, right job — do not use Kafka for tiny projects, complex searching, or instant question-and-answer calls.