Skip to main content

Consuming Kafka Messages with Multiple Consumer Groups

Hands-On Lab

 

Photo of Will Boyd

Will Boyd

DevOps Team Lead in Content

Length

00:30:00

Difficulty

Intermediate

In this hands-on lab, we will have the opportunity to consume data from a Kafka topic using multiple consumers groups. One of these groups will have a single consumer, and the other will have two consumers. Going through this will provide a firsthand look at how different consumers and consumer groups handle messages in Kafka. Additionally, Kafka's persistent data structure for storing messages offers a great deal of flexibility in terms of how we want to consume messages. We can use consumer groups to perform different operations against the same set of messages while also being able to scale each consumer application horizontally.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Consuming Kafka Messages with Multiple Consumer Groups

Introduction

In this hands-on lab, we will have the opportunity to consume data from a Kafka topic using multiple consumers groups.

Solution

Begin by logging in to the lab servers using the credentials provided on the hands-on lab page:

  ssh cloud_user@PUBLIC_IP_ADDRESS

Set Up the First Consumer Group with One Consumer

  1. Set up the first consumer as the sole consumer in its group. We will consume some messages and save the output to /home/cloud_user/output/group1_consumer1.txt.

    The format should look like this:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 1 > /home/cloud_user/output/group1_consumer1.txt

    Note: The --topic called inventory_purchases exists already and messages are being published to it.

  2. Let it run for a few seconds, and then use the Ctrl + C shortcut to verify the number of messages being consumed.

  3. We can check on the consumed messages by running:

    cat /home/cloud_user/output/group1_consumer1.txt

    Note: Sometimes the data may appear out of order, but there should not be any sequence gaps between messages.

Set Up the Second Consumer Group with Two Consumers

  1. Create a consumer in a separate group, and store its output in /home/cloud_user/output/group1_consumer1.txt.

    It should look like this:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 2 > /home/cloud_user/output/group2_consumer1.txt
  2. Create a second consumer in the same group, and store its output in /home/cloud_user/output/group2_consumer2.txt:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 2 > /home/cloud_user/output/group2_consumer2.txt
  3. Use Ctrl + C on both of the consumers to view the number of messages being processed.
  4. Check the consumed messages on the first consumer with:

    cat /home/cloud_user/ouput/group2_consumer1.txt
  5. Check the consumed messages on the second consumer with:

    cat /home/cloud_user/output/group2_consumer2.txt

    Note: The first consumer that we ran will exhibit more messages due to the run time being a tad longer than the second consumer. A message that is not consumed by the first gets consumed by the second.

Conclusion

Congratulations - you've completed this hands-on lab!