Skip to main content

Connecting to Kafka Programmatically in Java

Hands-On Lab

 

Photo of Will Boyd

Will Boyd

DevOps Team Lead in Content

Length

01:15:00

Difficulty

Intermediate

The real power of Kafka comes from the ability to write applications that utilize it. Kafka's Java APIs make it easy to build applications that work with Kafka data. Through this lab, we will have the opportunity to build a simple Java program that consumes data from a Kafka topic. Tackling this challenge will provide hands-on experience with the Kafka Java APIs and consuming data from a real cluster. After completing this introductory lab, we will be ready to move on to more complex scenarios involving both Kafka and Java code.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Connecting to Kafka Programmatically in Java

Introduction

In this hands-on lab, we will build a simple Java program that consumes data from a Kafka topic.

Solution

Begin by logging in to the lab servers using the credentials provided on the hands-on lab page:

  ssh cloud_user@PUBLIC_IP_ADDRESS

Clone the Starter Project from GitHub and Perform a Test Run

  1. Clone the starter project from GitHub:

    cd ~/
    git clone https://github.com/linuxacademy/content-ccdak-kafka-simple-consumer.git

    Note: We can use ls to view the folder named after the repository.

  2. Perform a test to make sure the code is able to compile and run:

    cd content-ccdak-kafka-simple-consumer/
    ./gradlew run

    The output should contain the message printed by the main class: Hello, world!.

Implement a Consumer in the Main Class and Run It

  1. Add the Kafka Client Libraries as a project dependency in build.gradle with:

    vi build.gradle
  2. In the dependencies { ... } block, add the following line:

    implementation 'org.apache.kafka:kafka-clients:2.2.1'
  3. Edit the Main class:

    vi src/main/java/com/linuxacademy/ccdak/kafkaSimpleConsumer/Main.java
  4. Implement a basic consumer that consumes messages from the topic and prints them to the screen:

    package com.linuxacademy.ccdak.kafkaSimpleConsumer;
    
    import org.apache.kafka.clients.consumer.*;
    import java.util.Properties;
    import java.util.Arrays;
    import java.time.Duration;
    
    public class Main {
    
      public static void main(String[] args) {
          Properties props = new Properties();
          props.setProperty("bootstrap.servers", "localhost:9092");
          props.setProperty("group.id", "my-group");
          props.setProperty("enable.auto.commit", "true");
          props.setProperty("auto.commit.interval.ms", "1000");
          props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
          props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
          KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
          consumer.subscribe(Arrays.asList("inventory_purchases"));
          while (true) {
              ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
              for (ConsumerRecord<String, String> record : records) {
                  System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
              }
          }
      }
    
    }
  5. Save and exit.

  6. Run the code:

    ./gradlew run

    The program should print a series of messages from the Kafka topic containing information about item purchases. We should see an offset, key, value, id, product, and quantity listed for each record in the output.

    Note: Use the Ctrl + C keyboard shortcut to stop printing messages.

Conclusion

Congratulations - you've completed this hands-on lab!