Kafka Fundamentals and a Hands-On Java Example

Apache Kafka has become indispensable for building scalable, distributed applications that handle real-time data streams. In this blog post, we’ll delve into the fundamentals of Kafka and demonstrate its power with a practical Java example.

What is Apache Kafka?

At its core, Kafka is a distributed publish-subscribe messaging system. Let’s break down what this means:

• Distributed: Kafka runs as a cluster of servers (brokers), providing fault tolerance and high availability.
• Publish-Subscribe: Applications called producers publish messages to Kafka topics. Consumers then subscribe to these topics to receive and process the messages.
• Messaging System: Kafka reliably stores messages in an ordered, immutable sequence. It supports real-time and historical message consumption.

Key Kafka Concepts

• Topics: Logical categories for organizing messages.
• Partitions: Topics are divided into partitions, improving scalability and allowing messages to be processed in parallel.
• Brokers: Servers that make up the Kafka cluster, handling message storage and requests from producers and consumers.
• Producers: Applications that send messages to Kafka topics.
• Consumers: Applications that subscribe to Kafka topics and process the messages.

Setting Up the Project

1. Include the Dependency: Add the Kafka-clients dependency to your project’s build system (Maven or Gradle). Here’s the Maven example:
2. XML
3. <dependency>
4.     <groupId>org.apache.kafka</groupId>
5.     <artifactId>kafka-clients</artifactId>
6.     <version>3.3.1</version> </dependency>
7. Use code 
8. content_copy
9. Start Kafka: If you don’t have a running Kafka cluster, you can download and set up a single-node cluster locally from the official website 

Java Producer Example

Java

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;

import org.apache.kafka.clients.producer.ProducerConfig;

import org.apache.kafka.clients.producer.ProducerRecord;

public class SimpleProducer {

    public static void main(String[] args) {

        Properties props = new Properties();

        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, “localhost:9092”);

        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, “org.apache.kafka.common.serialization.StringSerializer”);

        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, “org.apache.kafka.common.serialization.StringSerializer”);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {

            for (int i = 0; i < 10; i++) {

                String message = “Test message ” + i;

                ProducerRecord<String, String> record = new ProducerRecord<>(“my-topic”, “key-” + i, message);

                producer.send(record);

                System.out.println(“Message sent: ” + message);

            }

        } 

    }

}