Metadata: From Zookeeper to KRaft

As a distributed system, Kafka does not only require streaming data to function; it also needs to know information about the distributed system itself, known as metadata. Which node is the controller? What is the topic configuration? Who is assigned to access control lists (ACLs)? Metadata answers all of these questions. Metadata management with Kafka has evolved through its history, and recently underwent its most drastic change: a migration from ZooKeeper to KRaft.

In this session, we will discuss what metadata is and why it is important in an event streaming system. We will also explore the difference between ZooKeeper and KRaft, learn the reasons why a migration is necessary, and understand the general operations that may be followed for migrating with the Confluent Platform.

ZooKeeper has been paramount to the Kafka ecosystem since its inception. Being a general system for managing distributed metadata, ZooKeeper requires separate nodes within a cluster. For this reason, and more that we will discuss, a quorum-based approach was developed from the ground-up. KRaft, based on the Raft consensus algorithm, allows each Kafka broker to dually act as a metadata manager, reducing complexity and infrastructure management requirements.

Migrating from ZooKeeper to KRaft should be done carefully. Metadata is often expected to be persisted permanently; for example, ACLs for authorization are managed as metadata. A loss of metadata during migration may result in disruptive data loss; however, Confluent has released tools and processes to make this process safer. We will demonstrate such a process, known as dual_write mode, as we walk through a safety-first migration using Confluent Platform and Ansible.

Through an increased knowledge of metadata management with Kafka, this session seeks to grant you a greater peace-of-mind during your next migration from ZooKeeper to KRaft.