Kafka Unchained: Running Kafka Without ZooKeeper

Apache Kafka has traditionally been synonymous with ZooKeeper. ZooKeeper is vital in managing Kafka clusters – storing metadata, handling broker coordination, leader elections, and more. However, with the introduction of KRaft mode, Kafka can officially shed its ZooKeeper dependency, bringing about simplified architecture and exciting advantages.

What is KRaft Mode?

Kraft, introduced in Kafka Improvement Proposal KIP-500, utilizes the Raft consensus algorithm to manage Kafka’s internal metadata. Let’s break down what this means:

• Goodbye ZooKeeper: ZooKeeper, a separate distributed system, is no longer required. Kafka brokers themselves handle metadata management.
• Raft Consensus: Raft is a well-established algorithm for reaching agreements in distributed systems. In Kafka, a quorum of controller brokers uses Raft to elect a leader and maintain consistency of cluster metadata.
• Metadata Management: All information about partitions, topics, configurations, and broker states is stored and replicated within the Kafka cluster.

Benefits of Kafka without ZooKeeper

• Simplified Architecture: Removing ZooKeeper streamlines your Kafka setup, reducing the operational complexity of managing multiple systems.
• Enhanced Stability: A single, integrated system can be inherently more reliable and easier to troubleshoot.
• Improved Scalability: KRaft mode has the potential to handle more partitions and topics, as ZooKeeper’s scaling constraints no longer limit Kafka.
• Easier Operations: Monitoring, configuration, and administration are more straightforward, with one less system to manage.
• Faster Restarts: KRaft’s metadata storage model can significantly speed up cluster recovery after outages.

Considerations and Getting Started

• KRaft Availability: KRaft mode became generally available starting from Apache Kafka version 2.8.0.
• Migration Process: Existing Kafka clusters require careful planning to migrate from a ZooKeeper-based architecture to KRaft mode.
• Maturity: While KRaft offers compelling advantages, it’s important to understand that it’s still a relatively new feature compared to the time-tested Zookeeper-based model.

To begin using KRaft mode, here are the basic steps:

1. Upgrade: Ensure your Kafka cluster is running version 2.8.0 or later.
2. Configuration: Modify your Kafka broker configuration to enable KRaft mode (process.roles=broker, controller).
3. Start (or Restart): Start your brokers with the updated configuration. The cluster will initiate the bootstrapping of a controller quorum.

The Future of Kafka

Kraft marks an exciting shift in Kafka’s evolution. While ZooKeeper has served Kafka well, removing this dependency is a significant stride toward a more streamlined and resilient framework. As KRaft mode matures, expect wider adoption and increasing benefits for those running Kafka in production.