A Hands-On Tutorial to Setup Kafka Python Client

Imagine having the power to process billions of real-time data events effortlessly, right from your Python code! Whether you’re tracking live transactions, monitoring social media feeds, or building event-driven applications, Apache Kafka combined with Python can make it happen. 

This article will guide you through setting up a Kafka Python environment and show you how to set up your Kafka Python Client. If you’re looking to unlock the full potential of real-time data streaming, keep reading—you won’t want to miss this!

Prerequisites

• Understanding of Streaming Data.

What is Apache Kafka?

Originally developed by LinkedIn, Apache Kafka is an Open-source and Distributed Stream Processing platform that collects, stores, organizes, and manages real-time data. In other words, it is a Message Streaming Platform that allows you to develop event-driven applications using real-time streaming information.

 Key Features of Kafka

1. High Throughput: Kafka is capable of handling millions of messages per second, making it a go-to solution for large-scale, real-time data streaming.
2. Distributed Architecture: Kafka’s distributed nature allows it to spread data across multiple servers (brokers), improving fault tolerance, scalability, and availability.
3. Real-Time Processing: Kafka processes data streams in real time, making it suitable for use cases like real-time analytics, monitoring, and alerting.
4. Multi-Client Support: Kafka supports multiple programming languages, including Java, Python, Scala, and Go, making it flexible and accessible to developers across different platforms.

How Can We Setup Kafka Python Client?

Prerequisites:

1. Install Kafka on your machine or use a managed Kafka service like Confluent Cloud or AWS MSK.
2. Kafka-Python Library

Now let’s setup Kafka python client: 

Step 1: Starting Kafka

Start the Zookeeper service (Kafka depends on Zookeeper for managing cluster states):

bin/zookeeper-server-start.sh config/zookeeper.properties
 Now, Start the Kafka Server.
bin/kafka-server-start.sh config/server.properties

Step 2: Create your Kafka Producer

The Kafka producer is responsible for sending messages to Kafka topics. Create Kafka producer by running the code:

from kafka import KafkaProducer
import json

# Initialize Kafka Producer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')  # Serializing JSON data
)

# Produce a message to a topic
producer.send('my_topic', {'key': 'value'})
producer.flush()  # Ensure all messages are sent
producer.close()  # Close the producer when done

Step 3: Create your Kafka Consumer.

Next, let’s set up a Kafka consumer to read messages from the topic:

from kafka import KafkaConsumer
import json

# Initialize Kafka Consumer
consumer = KafkaConsumer(
    'my_topic',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',  # Start reading at the earliest offset
    enable_auto_commit=True,  # Automatically commit message offsets
    group_id='my_consumer_group',  # Define the consumer group
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))  # Deserialize JSON data
)
# Consume messages
for message in consumer:
    print(f"Received message: {message.value}")

Step 6: Create New Topics

To create a new Topic in Kafka, open a new command terminal and execute the following command. 

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic

From the above command, you create a new topic, “test_topic,“ with a single partition and replication factor. Producers and Consumers can use this topic name to send and receive messages to and fro the Kafka Servers while building a Kafka Python client.

Step 5: Test your Apache Environment

You can start the Producer Console for writing messages into Kafka Brokers. Open a new command terminal and execute the following command. 

kafka-console-producer.sh --broker-list localhost:9092 --topic test_topic

Now, start the Consumer Console for reading messages from the Kafka brokers. Open a new command prompt and execute the following command.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test_topic --from-beginning

Step 6: Error Handling in Python

Error handling is crucial when working with Kafka Python in production. Here’s an example of how to manage Kafka errors effectively:

from kafka.errors import KafkaError

try:
    producer.send('my_topic', {'key': 'value'})
    producer.flush()
except KafkaError as e:
    print(f"Error occurred: {e}")

KafkaTimeoutError and KafkaError are few common errors that can occur.

Conclusion

In this article, you have learned how to send a simple message from the Python Producer Console and instantly receive messages in the Python Consumer Console and How to use kafka in Python. However, using the Kafka Python Client, you can send large or high-end messages from producers to consumers by having Kafka Servers as a Mediator.

As this article only focuses on the Kafka-Python library for building Kafka Python client applications, you can use other libraries like “pykafka” and “confluent-Kafka” to compare each library’s performance and throughput rates.

Extracting complicated data from Apache Kafka, on the other hand, can be Difficult and Time-Consuming. If you’re having trouble with these and want to find a solution, Hevo Data is a good place to start!

Hevo Data is a No-Code Data Pipeline that offers a faster way to move data from 150+ Data Sources including Apache Kafka, Kafka Confluent Cloud, and other 40+ Free Sources, into your Data Warehouse to be visualized in a BI tool. You can use Hevo’s Data Pipelines to replicate the data from your Apache Kafka Source or Kafka Confluent Cloud to the Destination system. Hevo is fully automated and hence does not require you to code. 

Sign up for a 14-day Free Trial and experience the feature-rich Hevo suite first hand. You can also have a look at the unbeatable Hevo Pricing that will help you choose the right plan for your business needs.

Frequently Asked Questions