To understand a tool that inspects consumer groups, we first need to recall what a consumer group is and what it remembers.

Topics, partitions, and consumers

Apache Kafka stores streams of events, such as orders, payments, clicks, or sensor readings, in topics. Each topic is split into partitions, and every event in a partition has a position number called an offset. Applications that read events are called consumers.

Consumer groups share the work

Consumers that share the same group name form a consumer group. Kafka divides the partitions of a topic among the members of the group, so that each partition is read by exactly one member at a time. Different groups read the same topic independently, each keeping its own progress.

The bookmark

As a group reads, it periodically saves the offset it has reached for each partition. This saved number is the committed offset, and it means “everything before this has been handled”. The bookmarks are stored inside Kafka, in a special internal topic. If a consumer crashes, the replacement starts from the last saved bookmark.

Lag: the height of the unmarked pile

For each partition, the newest event has a position called the log end offset. The difference between the log end offset and the group’s committed offset is called lag. It says how many events are waiting to be processed. Small, steady lag is normal, because events arrive all the time. Growing lag means the group cannot keep up.

flowchart LR
  A["Events already handled - up to committed offset"] --> B["Lag - events waiting to be processed"]
  B --> C["Log end offset - newest event in the partition"]
        

It is a small program, shipped with Kafka, that lets you look at and manage consumer groups from a terminal.

The simple definition

kafka-consumer-groups.sh is a command-line script in the bin folder of every Kafka download. When you run it, it connects to a Kafka cluster, asks about consumer groups or asks to change their saved positions, and prints the result. It can list all groups, describe a group’s progress and members, show a group’s state, delete a group, and reset a group’s bookmarks.

Where it comes from

It belongs to a family of small administrative scripts shipped with Kafka, each starting with “kafka-” and doing one job. On Windows, the same tools appear as batch files. In some packaged distributions, such as Confluent’s, the script is provided without the “.sh” ending, but behaves the same.

What happens when you run it

The script starts a small Java program that uses Kafka’s administrative client library. It contacts the brokers you name, sends a request such as “describe this group”, and prints the reply. For a consumer group, the broker that acts as the group coordinator holds the group’s membership and state, and the internal offsets topic holds its bookmarks. The tool reads the bookmarks, and compares them with the newest events in each partition, to calculate lag.

flowchart LR
  U["You in a terminal"] --> S["kafka-consumer-groups.sh"]
  S --> A["Kafka admin client library"]
  A --> B["Bootstrap broker"]
  B --> GC["Group coordinator"]
  A --> OT["Internal offsets topic - bookmarks"]
  A --> LP["Partition logs - newest offsets"]
        

Before anything else, the tool must know where the cluster is and how to log in.

The bootstrap server option

Every use requires the address of at least one broker through the bootstrap server option. It is like the first phone number in a contact book: the tool contacts that broker, which tells it about the rest of the cluster. Giving two or three addresses is wise, in case one broker is down.

Talking to a secured cluster

Production clusters normally require encrypted connections and proof of identity. The tool supports this through the command config option, which points to a small properties file holding the connection and login settings. Without it, a secured cluster will refuse the tool.

Permissions matter

What you can see and do depends on the identity you present. Describing a group needs permission to read information about that group and its topics. Deleting a group or resetting its bookmarks needs stronger rights over the group. On a secured cluster, the same command may work for one person and be refused for another.

The tool has a small set of actions. Each run performs one of them.

The supporting options

Listing is the simplest action: it answers “which consumer groups exist?”

What you see

The list action prints the name of every group the cluster knows, one per line. On a busy cluster this can be a long list, including groups of many teams, groups created by tools, and groups left over from old experiments.

Why a group appears even when nobody is running

A group is remembered as long as its bookmarks are stored. A group with no active members can still appear in the list until its bookmarks expire. So the list shows groups that are working now and groups that stopped some time ago.

Practical uses

Describe is the most used action. It answers “how is this group doing?”

The default view: offsets and lag

The basic description prints one line for every partition the group has bookmarks for. Each line shows the group name, the topic, the partition number, the committed offset, the log end offset, the lag, and, if the partition is currently owned by a running consumer, the identity of that consumer, its host, and its client name.

Reading the lag

flowchart TD
  D["Describe the group"] --> Q1{"Is lag growing over time?"}
  Q1 -->|"No"| H["Healthy"]
  Q1 -->|"Yes"| Q2{"On all partitions?"}
  Q2 -->|"Yes"| S["Consumers too slow: tune or add members"]
  Q2 -->|"No"| Q3{"Does the lagging partition have an owner?"}
  Q3 -->|"Yes"| K["Look for a hot key or stuck processing"]
  Q3 -->|"No"| N["Nobody owns it: check group state and members"]
        

Besides progress, the tool can show who is in the group and what the group is doing right now.

The members view

The members option lists the consumers currently in the group. Each line shows the consumer’s identity, its host, its client name, and how many partitions it owns. With the verbose option, it also lists exactly which partitions each member owns. In groups that use stable, static member names, the fixed name of each member also appears.

What the members view teaches

The state view

The state option shows the group’s current state, its coordinator, the assignment strategy, and the number of members. The state tells you what the group is doing at this moment.

stateDiagram-v2
  [*] --> Empty
  Empty --> PreparingRebalance: First member joins
  PreparingRebalance --> CompletingRebalance: Members have rejoined
  CompletingRebalance --> Stable: Assignments delivered
  Stable --> PreparingRebalance: Member joins, leaves, or fails
  Stable --> Empty: All members leave
  Empty --> Dead: Group deleted or offsets expire
        

Reset is the most powerful action of the tool. It can rewind time for a group or push it forward.

Why anyone would reset a group

Where to move the bookmark

flowchart LR
  E["Earliest stored event"] --- M1["Rewind to a time or offset"] --- C["Current bookmark"] --- M2["Shift forward"] --- L["Latest event"]
        

Because a reset can cause large reprocessing or permanent skipping, the tool builds in safeguards. Understand them well.

Rule 1: The group must be inactive

The tool refuses to reset a group that has active members. Otherwise, the running consumers would keep their own positions in memory and later overwrite the new bookmarks. Stop all consumers of the group first, or wait until the group’s state is empty.

Rule 2: The default is a preview

By default, a reset only shows what it would do, without changing anything. This preview, called a dry run, prints the planned new positions for every partition. Only when you add the execute option does the tool actually save the new bookmarks. Always look at the preview first.

Rule 3: Record the old positions

Before executing a reset, save the current bookmarks, for example by writing them to a file with the export option. If the reset turns out to be a mistake, that file lets you put the bookmarks back where they were.

Changing a bookmark is easy. Living with the consequences takes thought.

Moving backwards: reprocessing

When a group is rewound, its consumers will read events they have already handled. If their processing is safe to repeat, no harm is done. If not, duplicates appear: emails sent twice, records inserted twice, totals counted twice. Before rewinding, check that the consumers, and the systems they write to, tolerate repeats, or clean up the affected downstream data first.

Moving forwards: skipping

When a group is moved forward, everything in the skipped range is never processed by that group. Those events remain in Kafka until retention removes them, but the group will not see them unless it is rewound again. Skipping is often chosen to clear a huge backlog, but it should be a conscious decision, ideally recorded and approved.

The load spike

Rewinding a group to the earliest position on a big topic may force consumers to process days or weeks of events as fast as they can. This can overwhelm databases and other downstream services. Consider adding consumers, throttling, or resetting in stages.

Groups and their bookmarks can be removed, either wholly or in part.

Deleting a group

The delete action removes a group and all its saved bookmarks. Like reset, it works only when the group has no active members. After deletion, if an application starts again with the same group name, it is treated as a brand-new group, with no history, and will start according to its reset rule, either from the earliest available event or from the newest.

When deletion makes sense

Deleting offsets for chosen topics

The delete offsets action removes only the bookmarks for chosen topics in a group, leaving the group and its other bookmarks intact. This is useful when a group stops reading a topic for good, and the old bookmarks would otherwise sit around and keep showing up as lag in monitoring.

The tool shines during incidents. Here are frequent situations and how it helps.

flowchart TD
  P["Problem reported"] --> L["List groups: is the group name correct?"]
  L --> S["Describe state: Stable, rebalancing, or Empty?"]
  S --> M["Describe members: who owns which partitions?"]
  M --> O["Describe offsets: where is the lag?"]
  O --> A["Decide: tune, scale, fix code, or reset"]
        

The tool is a snapshot. Serious operations need continuous watching as well.

Limits of a snapshot

Running the tool by hand tells you what the group looks like right now. It does not warn you at night, and it does not show the history of lag over hours. For that, teams add continuous monitoring: exporters that publish lag numbers to a metrics system, dashboards that draw lag over time, and alerts that fire when lag keeps growing.

How the two fit together

Because the tool can move bookmarks and delete groups, who may use it matters greatly.

The tool has no powers of its own

The script is only a messenger. What it can do depends entirely on the identity it presents and the permissions that identity holds. Reading a group’s information, reading topic details for lag calculation, deleting a group, and changing bookmarks are separate permissions that administrators can grant or withhold.

Sensible arrangements

Like every tool, it has strengths and limits.

Most trouble with this tool repeats across teams.

Best practices at a glance

Let us follow an on-call engineer at an online shop during an incident.

sequenceDiagram
  participant E as On-call Engineer
  participant T as kafka-consumer-groups.sh
  participant K as Kafka Cluster
  E->>T: Describe state of email group
  T->>K: Ask coordinator
  K-->>T: Stable, 3 members
  E->>T: Describe offsets
  T->>K: Read bookmarks and log ends
  K-->>T: Lag growing on all partitions
  E->>E: Scale consumers to six
  E->>T: Describe again after some minutes
  K-->>T: Lag falling
        

It is the command-line tool shipped with Kafka for listing consumer groups, seeing their progress, lag, members, and state, and for resetting or deleting their saved positions.

No. It reads only bookmarks and log positions to calculate lag, and it changes bookmarks when asked. Your consumers do the actual processing.

Lag is the number of events between the group’s committed bookmark and the newest event of a partition, meaning how many events are waiting to be processed.

Because no running consumer currently owns those partitions. The group may be stopped, in the middle of a rebalance, or the partition has no assigned member.

No. The tool refuses, because running consumers would overwrite the new bookmarks. Stop the group and confirm its state is empty first.

Not by default. The default is a preview that changes nothing. You must add the execute option to save the new positions.

The group and all its bookmarks are removed. If an application starts with the same name, it behaves as a new group and begins from the earliest or the newest event according to its reset rule.

Because its bookmarks are still stored. It remains visible in an empty state until its bookmarks expire or it is deleted.

It is great for investigation but gives only a snapshot. Use continuous monitoring and alerting on lag and group state for day-to-day safety.

You can, but for repeatable production changes many teams prefer the admin client library, which is easier to test and does not depend on parsing text. Whichever you choose, keep the safety steps, reviews, and records.

We began with a school where a supervisor walks the corridors with a clipboard, checking the height of each team’s pile of unmarked papers, who is on each team, and occasionally correcting a team’s notebook. kafka-consumer-groups.sh is that clipboard for Kafka. It ships with Kafka, connects through a bootstrap broker, and offers a few actions: list the groups, describe their progress, lag, members, and state, reset their bookmarks, and delete groups or their bookmarks. Describing is safe and constantly useful, especially when read in a sensible order: name, state, members, then lag. Resetting is powerful and dangerous: it works only when the group is inactive, previews by default, can move bookmarks by time, offset, duration, or file, and can cause heavy reprocessing or permanent skipping, so it demands stopped consumers, a saved copy of the old positions, a read preview, verification afterwards, and repeat-safe consumers. The tool is a snapshot, so it must be paired with continuous lag and state monitoring, and permissions should give read access widely and change rights sparingly. Learn to read a group calmly, and change one only with a plan.