removing a kafka consumer group in zookeeper

Question

I'm using kafka_2.9.2-0.8.1.1 with zookeeper 3.4.6.

Is there a utility that can automatically remove a consumer group from zookeeper? Or can I just remove everything under /consumers/[group_id] in zookeeper? If the latter, is there anything else I'm missing & can this be done with a live system?

Update: As of kafka version 2.3.0, there is a new utility:

> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group

Related doc: http://kafka.apache.org/documentation/#basic_ops_consumer_lag

See below for more discussion

Answer 1

Currently, as I know, the only way to remove a Kafka consumer group is manually deleting Zookeeper path /consumers/[group_id].

If you just want to delete a consumer group, there is nothing to worry about manually deleting the Zookeeper path, but if you do it for rewinding offsets, the below will be helpful.

First of all, you should stop all the consumers belongs to the consumer group before removing the Zookeeper path. If you don't, those consumers will not consume newly produced messages and will soon close connections to the Zookeeper cluster.

When you restart the consumers, if you want the consumers to start off from the beginning, give auto.offset.reset property to smallest (or earliest in new Kafka releases). The default value of the property is largest (or latest in new Kafka releases) which makes your restarting consumers read after the largest offset which in turn consuming only newly produced messages. For more information about the property, refer to Consumer Config in the Kafka documentation.

FYI, there is a question How can I rewind the offset in the consumer? in Kafka FAQ, but it gave me not much help.

Answer 2

As of v0.9.0, Kafka ships with a suite of tools in the /bin one of which is the kafka-consumer-groups.sh tool. This will delete a consumer group. ./kafka-consumer-groups.sh --zookeeper <zookeeper_url> --delete --group <group-name>

Answer 3

For new consumers (which use a kafka topic to manage offsets instead of zookeeper) you cannot delete the group information using kafka's built in tools.

Here is an example of trying to delete the group information for a new style consumer using the kafka-consumer-groups.sh script:

bin/kafka-consumer-groups.sh --bootstrap-server "kafka:9092" --delete --group "indexer" --topic "cleaned-logs"
Option '[delete]' is only valid with '[zookeeper]'. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

Here's the important part of that response:

Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

This is kind of annoying from a monitoring perspective (esp. when tracking offsets via something like burrow) because it means that if you change consumer group names in your code you'll keep seeing that old groups are behind on their offsets until those offsets expire.

Hypothetically you could write a tombstone to that topic manually (which is what happens during offset expiration) but I haven't found any tools that make this easy.

Answer 4

you can delete group from kafka by CLI

kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group group_name

Answer 5

I've had an issue with an consumer group and wanted to delete the group. The following approach worked for me:

• Set the offsets.retention.minutes=1 in the Kafka server.config file and restart the Kafka server(s)
• Shutdown/exit the processes which use the consumer group
• Set the offsets of this topic/partitions to 0 (something like kafka-consumer-groups.sh --bootstrap-server yourserver:9092 --group yourgroup --reset-offsets --to-latest --all-topics --execute

Don`t forget to set the parameters back and to restart the Kafka(s) again.