Apache Kafka - Real-time Stream Processing (Master Class) - Kafka Cluster Architecture

The video tutorial explains the scalability and clustering of Apache Kafka, focusing on how Kafka brokers form a cluster and manage workloads. It discusses the role of Zookeeper in maintaining active brokers and the concept of a masterless cluster. The tutorial also covers the responsibilities of the Kafka controller, which is elected among brokers to handle administrative tasks. The process of controller election and the role of ephemeral nodes in Zookeeper are detailed, providing a comprehensive understanding of Kafka's distributed system architecture.

What is the primary purpose of forming a Kafka cluster?

• To increase the number of topics

• To distribute workload and enhance scalability

• To reduce the number of brokers

• To simplify configuration

What is the role of a broker in a Kafka cluster?

• To store and serve messages

• To create topics

• To act as a client

• To manage Zookeeper

What are the two critical questions raised about Kafka clustering?

• How to configure brokers and how to monitor them?

• Who creates topics and who deletes them?

• How to increase the number of brokers and how to reduce latency?

• Who manages cluster membership and who performs administrative tasks?

How does Kafka maintain the list of active brokers?

• Through a configuration file

• By manual updates

• Using a master node

• With Apache Zookeeper

What happens when a broker loses connectivity to Zookeeper?

• The broker is manually restarted

• The broker continues to function normally

• The broker is automatically removed from the list of active brokers

• The broker becomes the controller

What is the significance of ephemeral nodes in Zookeeper for Kafka?

• They are not used in Kafka

• They store permanent data

• They represent active broker sessions

• They are used for logging

What is the main function of Zookeeper in a Kafka cluster?

• To handle client requests

• To maintain the list of active brokers

• To manage broker configurations

• To store messages

What is the role of the controller in a Kafka cluster?

• To delete old brokers

• To act as a master node

• To perform routine administrative tasks

• To create new topics

How is the controller elected in a Kafka cluster?

• By manual assignment

• The first broker that starts becomes the controller

• By a voting process among brokers

• Based on the broker's ID

What happens when the current controller in a Kafka cluster fails?

• The cluster stops functioning

• A new controller is elected automatically

• All brokers become controllers

• The failed controller is restarted