Skip to main content

Tuning a Kafka Producer

Hands-On Lab

 

Photo of Will Boyd

Will Boyd

DevOps Team Lead in Content

Length

00:30:00

Difficulty

Intermediate

Configuring Kafka producers is a relatively simple process. The real challenge is knowing what configuration options to change in order to tune the producer for your use case. In this lab, you will have the opportunity to work hands-on with the process of tuning a Kafka producer. You will be presented with a few issues that can be addressed through configuration tuning. You will then need to determine what configuration changes to make in order to address these issues, and you will need to implement those changes in the producer.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Tuning a Kafka Producer

Introduction

Configuring Kafka producers is a relatively simple process. The real challenge is knowing what configuration options to change in order to tune the producer for your use case. In this lab, you will have the opportunity to work hands-on with the process of tuning a Kafka producer. You will be presented with a few issues that can be addressed through configuration tuning. You will then need to determine what configuration changes to make in order to address these issues, and you will need to implement those changes in the producer.

Solution

Log in to the lab server using the credentials provided on the hands-on lab page:

ssh cloud_user@PUBLIC_IP_ADDRESS

Clone the Starter Project from GitHub and Perform a Test Run

  1. Clone the starter project from GitHub (first changing to the home directory if you aren't already in it):

    cd ~/
    git clone https://github.com/linuxacademy/content-ccdak-producer-tuning-lab.git
  2. Change to the content-ccdak-producer-tuning-lab directory:

    cd content-ccdak-producer-tuning-lab
  3. Perform a test run to make sure the code is able to compile and run:

    ./gradlew run

    The code should compile and tests should succeed.

Implement the Configuration Changes in the Producer

  1. Open the file:

    vi src/main/java/com/linuxacademy/ccdak/producer/MemberSignupsProducer.java
  2. Fix the issue with data loss by setting acks=all:

    props.put("acks", "all");
  3. Fix the out-of-order record issue caused by retries by setting max.in.flight.requests.per.connection=1:

    props.put("max.in.flight.requests.per.connection", "1");
  4. Increase the batch size by setting batch.size=65536:

    props.put("batch.size", "65536");
  5. Save and exit the file.

  6. Run your code to verify that it still works:

    ./gradlew run

Conclusion

Congratulations on successfully completing this hands-on lab!