Is Zookeeper a must for Kafka?

Question

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of dat

Answer 1

Updated on Nov 2020

For the latest version (2.6.0) ZooKeeper is still required for running Kafka, but in the near future ZooKeeper will be replaced with a Self-Managed Metadata Quorum.

See details in the accepted KIP-500.

---

1. Current status

Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller.

Currently, removing this dependency on ZooKeeper is work in progress (through the KIP-500) .

2. Profit of removal

Removing the Apache ZooKeeper dependency provides three distinct benefits:

• First, it simplifies the architecture by consolidating metadata in Kafka itself, rather than splitting it between Kafka and ZooKeeper. This improves stability, simplifies the software, and makes it easier to monitor, administer, and support Kafka.
• Second, it improves control plane performance, enabling clusters to scale to millions of partitions.
• Finally, it allows Kafka to have a single security model for the whole system, rather than having one for Kafka and one for Zookeeper.

3. Roadmap

ZooKeeper removal is expected in 2021 and has some milestones which are represented in the following KIPs:

|   KIP   |                           Name                           |      Status      | Fix Version/s |
|:-------:|:--------------------------------------------------------:|:----------------:|---------------|
| KIP-455 | Create an Administrative API for Replica Reassignment    |     Accepted     | 2.6.0         |
| KIP-497 | Add inter-broker API to alter ISR                        |     Accepted     | 2.7.0         |
| KIP-543 | Expand ConfigCommand's non-ZK functionality              |     Accepted     | 2.6.0         |
| KIP-555 | Deprecate Direct ZK access in Kafka Administrative Tools |     Accepted     | None          |
| KIP-589 | Add API to update Replica state in Controller            |     Accepted     | None          |
| KIP-590 | Redirect Zookeeper Mutation Protocols to The Controller  |     Accepted     | None          |
| KIP-595 | A Raft Protocol for the Metadata Quorum                  |     Accepted     | None          |
| KIP-631 | The Quorum-based Kafka Controller                        | Under discussion | None          |

KIP-500 introduced the concept of a bridge release that can coexist with both pre- and post-KIP-500 versions of Kafka. Bridge releases are important because they enable zero-downtime upgrades to the post-ZooKeeper world.

References:

1. KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
2. Apache Kafka Needs No Keeper: Removing the Apache ZooKeeper Dependency
3. Preparing Your Clients and Tools for KIP-500: ZooKeeper Removal from Apache Kafka