Apache Kafka - Real-time Stream Processing (Master Class) - Caution with States

The video tutorial addresses a problem in Kafka stream processing where invoices are partitioned by store ID, leading to issues with state management across tasks. It proposes solutions like changing the message key to customer ID or using a custom partitioner to ensure all invoices for a customer are processed in the same partition. The tutorial explains the implementation of a custom partitioner and highlights the importance of minimizing repartitioning to maintain performance.

What is the main issue with using store ID as a key in Kafka streams?

• It results in all invoices from the same store going to the same partition.

• It leads to inefficient data processing.

• It causes all invoices to go to different partitions.

• It increases the complexity of the Kafka setup.

Why does processing invoices in different partitions cause issues with loyalty points?

• Because the invoices are duplicated across partitions.

• Because the state stores are shared across tasks.

• Because the state stores are local to each task.

• Because the invoices are not processed in real-time.

What is the first approach to ensure all invoices for a customer are processed in the same partition?

• Use a different Kafka topic for each customer.

• Increase the number of partitions.

• Change the message key to customer ID.

• Use a custom partitioner.

What is the role of an intermediate topic in the first approach?

• To allow repartitioning based on customer ID.

• To reduce the number of partitions.

• To change the message value.

• To store all invoices temporarily.

What is the main advantage of using a custom partitioner?

• It eliminates the need for state stores.

• It reduces the number of Kafka topics needed.

• It allows partitioning based on any desired key.

• It simplifies the Kafka setup.

How does the 'through' method help in repartitioning?

• It reduces the number of partitions.

• It combines multiple streams into one.

• It changes the message key automatically.

• It writes the stream to a new topic and reads it back.

Why is it important to minimize repartitioning in Kafka streams?

• It increases the number of partitions.

• It complicates the message key design.

• It can be expensive and impact performance.

• It requires more Kafka topics.