Harnessing Apache Kafka’s Power on Windows

Apache Kafka has become essential for handling real-time data streams in modern software architectures. While its roots are in the Linux world, Kafka is perfectly capable of running on Windows. This blog post guides you through installing, configuring, and using Apache Kafka on your Windows machine.

What is Apache Kafka?

Apache Kafka is a highly scalable, fault-tolerant, distributed publish-subscribe messaging system. Let’s break down what that means:

• Publish-subscribe: Software components called ‘producers’ send messages (events) to Kafka ‘topics.’ Other components called ‘consumers’ can subscribe to these topics and receive the events.
• Distributed: Kafka operates as a cluster of nodes (servers), providing redundancy and high availability.
• Scalable: Kafka can manage massive data streams by distributing the load across multiple nodes.
• Fault-tolerant: Kafka replicates data across nodes, ensuring your data is preserved even if a node fails.

Why use Kafka on Windows?

• Real-time Event Streaming: Kafka powers applications that process events in real time, such as user activity tracking, financial transactions, and IoT data.
• Data Pipelines: Kafka builds reliable pipelines for moving data between systems (databases, microservices, etc.).
• Log Aggregation: Collect and centralize logs from different applications for easier monitoring and analysis.
• Decoupling Systems: Kafka decouples producers and consumers, allowing systems to evolve independently.

Installation Steps

1. Prerequisites

• • Java: Download and install Java Development Kit (JDK) version 8 or later from the official Oracle website.
  • Zookeeper: Kafka relies on Zookeeper for coordination. You’ll need to download Zookeeper separately.

1. Download Apache Kafka:  Get the latest binary release of Apache Kafka from https://kafka.apache.org/downloads
2. Extract Files: Extract the downloaded Kafka archive (e.g., kafka_2.13-3.6.1.tgz) to your desired installation directory.

Configuration

1. Zookeeper Configuration:

• • Navigate to the config directory within the extracted Zookeeper folder.
  • Edit the zookeeper.properties file. You’ll likely need to change the dataDir property path to a suitable Windows directory.

1. Kafka Configuration:

• • Navigate to the config directory inside your Kafka installation.
  • Edit server.properties and modify the log—dirs property to a Windows-appropriate path.

Starting the Services

1. Start Zookeeper:

• • Open a command prompt and navigate to the Zookeeper bin directory.
  • Run .\zkServer.cmd

1. Start Kafka:

• • Open another command prompt and navigate to the Kafka bin directory.
  • Run .\kafka-server-start.bat .\config\server.properties

Testing

1. Create a Topic:

• • Open a new command prompt and run: .\kafka-topics.bat –create –zookeeper localhost:2181 –topic test-topic –partitions 1 –replication-factor 1

1. Produce Messages:

• • Run: .\kafka-console-producer.bat –broker-list localhost:9092 –topic test-topic . Type some messages and press Enter to produce them.

1. Consume Messages:

• • Open another command prompt and run: .\kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic test-topic –from-beginning