Harnessing the Power of Apache Kafka on Your Mac

Apache Kafka has become invaluable for building scalable, fault-tolerant, real-time data streaming applications and platforms. If you’re a developer or data engineer using a Mac, you’re lucky; setting up and running Kafka on your machine is relatively straightforward.

Prerequisites

• Java: Kafka is a Java-based system, so you must install the Java Development Kit (JDK). Version 8 or later is recommended. You can check if Java is installed by running your terminal’s Javathe Java version. If not, download the JDK from Oracle’s website.
• Homebrew (Optional): While not strictly required, Homebrew is the most convenient way to manage software packages on macOS. If you don’t have Homebrew, follow the instructions on their website to install it.

Installation

There are two main ways to install Kafka on your Mac:

1. Manual Download

1. Visit the Apache Kafka downloads page: 
2. Download the latest binary release (choose a Scala version that aligns with your project).
3. Extract the downloaded tar file:
4. Bash
5. tar -xzf kafka_2.13-3.0.0.tgz
6. Use code 
7. content_copy
8. Move the extracted folder to a convenient location like /usr/local/Kafka.

2. Using Homebrew

1. Open your terminal and run:
2. Bash
3. brew install Kafka
4. Use code 
5. content_copy

Starting the Servers

The Kafka system relies on two core components:

• ZooKeeper: A coordination service that manages the Kafka cluster.
• Kafka Broker: The core Kafka server that handles message streams.

1. Start ZooKeeper:

• Manual: Navigate to the Kafka directory and run:
• Bash
• bin/zookeeper-server-start.sh config/zookeeper.properties
• Use code 
  • content_copy
• Homebrew:
• Bash
• zkServer start
• Use code 
  • content_copy

1. Start Kafka Broker:

• Manual: Open a new terminal, navigate to the Kafka directory, and run:
• Bash
• bin/kafka-server-start.sh config/server.properties
• Use code 
  • content_copy
• Homebrew:
• Bash
• Kafka-server-start
• Use code 
  • content_copy

Creating a Topic

A topic is a category or feed to which you publish and subscribe messages in Kafka.

1. In a new terminal, navigate to your Kafka directory.
2. Run the following command:
3. Bash
4. bin/Kafka-topics.sh –create –topic quickstart-topic –bootstrap-server localhost:9092
5. Use code 
6. content_copy
7. (Replace ‘quickstart-topic’ with your desired topic name)

Testing

1. Producer:
2. Bash
3. bin/kafka-console-producer.sh –topic quickstart-topic –broker-list localhost:9092 
4. Use code 
5. content_copy
6. Start typing messages and hit enter to send them.
7. Consumer:
8. Bash
9. bin/Kafka-console-consumer.sh –topic quickstart-topic –from-beginning –bootstrap-server localhost:9092
10. Use code 
11. content_copy
12. You should see the messages you typed in the producer.

Beyond the Basics

This blog has given you a quick start. Kafka offers rich features:

• Kafka Connect: Integrate with databases and external systems.
• Kafka Streams: Real-time stream processing capabilities.
• Configuration: Explore to optimize performance.

Stopping Kafka

Run these (using the correct method depending on how you installed Kafka):

Bash

bin/kafka-server-stop.sh 

bin/zookeeper-server-stop.sh