Skip to main content

Writing Tests for a Kafka Consumer

Hands-On Lab

 

Photo of Will Boyd

Will Boyd

DevOps Team Lead in Content

Length

00:45:00

Difficulty

Intermediate

You can accomplish a great deal by implementing your own Kafka consumers. Like almost any source code, it is a good idea to build unit tests to verify the functionality of your consumer code. Kafka's MockConsumer test fixture simplifies the process of building unit tests for producer code. In this lab, we will work with consumer test fixtures by writing a few unit tests for an existing consumer.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Writing Tests for a Kafka Consumer

Introduction

You can accomplish a great deal by implementing your own Kafka consumers. Like almost any source code, it is a good idea to build unit tests to verify the functionality of your consumer code. Kafka's MockConsumer test fixture simplifies the process of building unit tests for producer code. In this lab, we will work with consumer test fixtures by writing a few unit tests for an existing consumer.

Solution

Log in to the lab server using the credentials provided on the hands-on lab page:

ssh cloud_user@PUBLIC_IP_ADDRESS

Clone the Starter Project from GitHub and Perform a Test Run

  1. Clone the starter project from GitHub (first changing to the home directory if you aren't already in it):

    cd ~/
    git clone https://github.com/linuxacademy/content-ccdak-consumer-tests-lab.git
  2. Change to the new content-ccdak-consumer-tests-lab directory:

    cd content-ccdak-consumer-tests-lab
  3. Perform a test run to make sure the code is able to compile and run:

    ./gradlew test

    The code should compile, but the tests should fail since they are not implemented yet.

Implement the Unit Tests for the MemberSignupsConsumer

  1. Edit the test class for MemberSignupsConsumer:

    vi src/test/java/com/linuxacademy/ccdak/consumer/MemberSignupsConsumerTest.java
  2. Implement the testHandleRecords_output test:

    @Test
    public void testHandleRecords_output() {
        // Verify that the testHandleRecords writes the correct data to System.out
        // A text fixture called systemOutContent has already been set up in this class to capture System.out data.
        String topic = "member_signups";
        ConsumerRecord<Integer, String> record = new ConsumerRecord<>(topic, 0, 1, 2, "ROSENBERG, WILLOW");
        Map<TopicPartition, List<ConsumerRecord<Integer, String>>> records = new LinkedHashMap<>();
        records.put(new TopicPartition(topic, 0), Arrays.asList(record));
        ConsumerRecords<Integer, String> consumerRecords = new ConsumerRecords<>(records);
    
        memberSignupsConsumer.handleRecords(consumerRecords);
        Assert.assertEquals("key=2, value=ROSENBERG, WILLOW, topic=member_signups, partition=0, offset=1n", systemOutContent.toString());
    }
  3. Implement the testHandleRecords_none test:

    @Test
    public void testHandleRecords_none() {
        // Verify that testHandleRecords behaves correctly when processing no records.
        // A text fixture called systemOutContent has already been set up in this class to capture System.out data.
        String topic = "member_signups";
        Map<TopicPartition, List<ConsumerRecord<Integer, String>>> records = new LinkedHashMap<>();
        records.put(new TopicPartition(topic, 0), Arrays.asList());
        ConsumerRecords<Integer, String> consumerRecords = new ConsumerRecords<>(records);
    
        memberSignupsConsumer.handleRecords(consumerRecords);
        Assert.assertEquals("", systemOutContent.toString());
    }
  4. Implement the testHandleRecords_multiple test:

    @Test
    public void testHandleRecords_multiple() {
        // Verify that testHandleRecords behaves correctly when processing multiple records.
        // A text fixture called systemOutContent has already been set up in this class to capture System.out data.
        String topic = "member_signups";
        ConsumerRecord<Integer, String> record1 = new ConsumerRecord<>(topic, 0, 1, 2, "ROSENBERG, WILLOW");
        ConsumerRecord<Integer, String> record2 = new ConsumerRecord<>(topic, 3, 4, 5, "HARRIS, ALEXANDER");
        Map<TopicPartition, List<ConsumerRecord<Integer, String>>> records = new LinkedHashMap<>();
        records.put(new TopicPartition(topic, 0), Arrays.asList(record1, record2));
        ConsumerRecords<Integer, String> consumerRecords = new ConsumerRecords<>(records);
    
        memberSignupsConsumer.handleRecords(consumerRecords);
        Assert.assertEquals("key=2, value=ROSENBERG, WILLOW, topic=member_signups, partition=0, offset=1nkey=5, value=HARRIS, ALEXANDER, topic=member_signups, partition=3, offset=4n", systemOutContent.toString());
    }
  5. Save and exit the file.

  6. Run your tests and make sure they pass:

    ./gradlew test

Conclusion

Congratulations on successfully completing this hands-on lab!