Kafka Event Driven Architecture for Seamless Streaming

An Event Driven Architecture (EDA) is a microservices-based architectural paradigm that is becoming more prominent with rising in Big Data and Cloud environments. This isn’t just a coincidence. From the standpoint of Developers, EDA provides an efficient technique of linking microservices, which can aid in the development of future-proof systems.

Furthermore, event-driven systems when integrated with robust streaming services such as Apache Kafka become more agile, durable, and efficient than prior messaging approaches.

This article will help you understand the reason behind the rising popularity of Event Driven Architecture. You will discover more about the basic components and frequently used patterns for building the Event Driven Architecture.

Introduction to Apache Kafka

• Apache Kafka is a Distributed Event Streaming solution that enables applications to efficiently manage large amounts of data. Its fault-tolerant, highly scalable architecture can easily manage billions of events.
• The Apache Kafka framework is a Java and Scala-based distributed Publish-Subscribe Messaging system that accepts Data Streams from several sources and allows real-time analysis of Big Data streams.
• It can quickly scale up and down with minimal downtime. Kafka’s global appeal has grown as a result of its minimal data redundancy and fault tolerance.
• Thousands of businesses, including more than 60% of the Fortune 100, use Kafka. Box, Goldman Sachs, Target, Cisco, Intuit, and others are among them.
• Kafka is a trusted platform for enabling and developing businesses. Kafka Event Driven Architecture allows firms to upgrade their data strategy and increase productivity.

Want to explore more about Apache Kafka? You can visit the Kafka website or refer to Kafka documentation. 

Why Use Kafka for an Event Driven System?

• It is critical to understand the true meaning of the term “Event” when we use it. A record of “something that happened” is called an Event. When you read or transmit data to Kafka, you do so using Events.
• A key, value, timestamp, and optional metadata headers can all be part of an Event. As a result, an Event may be seen as a single atomic data point. 
• For example, when you visit a website and register or sign up, that signup is an event with some data associated with it. This may contain all of the information required to sign up. The event can also be thought of as a message including data.
• Kafka is an Event-Streaming system that handles a continuous stream of events. Event Streaming is the process of capturing data in the form of streams of events in real-time from event sources such as databases, IoT devices, or others.
• Then you store them durably for later retrieval, analysis, or process them in real-time, and route them to various destinations as needed.
• Event Streaming allows for continuous data flow and interpretation, ensuring that the appropriate information is available at the right time and in the right place. In this article, you will learn more about Event Driven Architecture using Apache Kafka. 

The Need for Kafka Event Driven Architecture

• In an Event Driven Architecture, an event notification is generated, the system captures what happened and waits to provide the response back.  The application that got the notification might either reply right away or wait till the status changes.
• Event Driven Architecture allows for more flexible, scalable, contextual, and responsive digital business systems. This is why this architecture style has been gaining popularity.
• The key rationale for leveraging Apache Kafka for an Event Driven system is the decoupling of microservices and the development of a Kafka pipeline to connect producers and consumers. Instead of checking for new data, you may just listen to a certain event and take action. Kafka provides a scalable hybrid approach that incorporates both Processing and Messaging.
• Another advantage of using Kafka Event Driven Architecture is that, unlike messaging-oriented systems, events published in Kafka are not removed as soon as they are consumed. They are removed once a specific amount of time has passed.

Components of an Event Driven Architecture

Event Driven Architecture is mostly employed in real-time systems that require you to act and respond to data changes and events in real-time. This architecture is especially beneficial for IoT systems. Let’s understand the basic components included while building the Event Driven Architecture.

• Event: When users take action, a major change in the state of an object that happens is referred to as an Event.
• Event Handler: It is a software routine that handles the Event occurrence.
• Event Loop: The interaction between an Event and the Event Handler is handled by the Event Loop.
• Event Flow Layers: The logical layers that make up the event flow layer are as follows:
  • Event Producer: They are responsible for detecting and generating Events.
  • Event Consumer: They consume the events generated by the Event Producer.
  • Event Channel: It is also called Event Bus. It helps to transfer events from the Event Generator to the Event Consumer.

Patterns of Event Driven Architecture

Kafka Event Driven Architecture can be implemented in a variety of ways. Here’s a short rundown of some of the most common Event Driven Architectural patterns that you can deploy using Apache Kafka:

• Event Notification
• Event Carried State Transfer
• Event Sourcing

1) Event Notification

• This is a simple and direct model. A microservice merely broadcasts events to alert other systems of a change in its domain. For example, when a new user is created, a user account service could generate a notification event.
• Other services can use this information, or it can be disregarded. Notification events often contain little data, resulting in a loosely coupled system with less network traffic dedicated to messaging.

2) Event Carried State Transfer

• In this approach, the event’s recipient also obtains the data it needs to do additional actions on the required data. For example, the aforementioned user account service may send out an event with a data packet including the new user’s login ID, complete name, hashed password, and other relevant information.
• This architecture may appeal to developers who are experienced with RESTful interfaces. However, it might result in a lot of data traffic on the network and data duplication in storage, depending on the system’s complexity.

3) Event Sourcing

• This model’s purpose is to describe every change of state in a system as an event, with each event being recorded in chronological sequence. As a result, the event stream itself becomes the system’s primary source of truth.
• It should be able to “replay” a sequence of events in order to reproduce the state of a SQL database at a certain moment in time. This approach has a lot of interesting potentials, but it may be difficult to get it working right, especially when events involve external system engagement.
• As you can see above, Event Driven Architecture can be deployed in many patterns. Some are simpler to implement and others may be more adaptive to complicated requirements.
• So, which pattern is appropriate for a specific use case depends on a variety of parameters, including the number of microservices involved, how closely they should be connected, etc.

Steps to Build a Kafka Event Driven Architecture

In this section, you will understand how to create a Kafka Event Driven Architecture using Python. Before you proceed further make sure you have started the Kafka and Zookeeper services. 

In the given Kafka Event Driven Architecture example, the producer will send out an event to Kafka along with the timestamp. The consumer will receive this event and print the timestamp. So, follow the steps below to get get started:

• Step 1: Set Up the Environment
• Step 2: Configure the Event Producer
• Step 3: Configure the Event Consumer
• Step 4: Execute the Kafka Event Driven Architecture

Step 1: Set Up Python Environment

After you have set up your docker and started the Kafka and Zookeeper services, create 2 python projects namely “producer” and “consumer”. Or you can simply clone the GitHub repository from here. 

Download the requirements.txt file from the above link which contains the kafka-python==2.0.2 dependency info. Then type the following command in your terminal to set up the Python version in each project.

python3 -m pip install -r requirements.txt

Step 2: Configure the Event Producer

After setting up the Python dependency, let’s proceed further to set up the Event Producer. It will produce timestamps and send them to Event Consumer via Kafka. Create a main.py file in producer python project and enter the following code snippet:

from kafka import KafkaProducer
from datetime import datetime
from json import dumps
from time import sleep

producer = KafkaProducer(bootstrap_servers='localhost:9092',
                        value_serializer=lambda x: dumps(x).encode('utf-8'))
while True:
   timestampStr = datetime.now().strftime("%H:%M:%S")
   print("Sending: " + timestampStr)
   producer.send('timestamp', timestampStr)
   sleep(5)

In the above code:

• bootstrap_servers parameter provides the host location of Kafka. By default it is localhost:9092.
• value_serializer specifies how the messages will be encoded.

The code will produce a timestamp string in the format of H:M:S.

Step 3: Configure the Event Consumer

After the producer creates timestamps, you need a consumer to listen to them. So, create a main.py file in the consumer python project and enter the following code snippet:

from kafka import KafkaConsumer
from json import loads

consumer = KafkaConsumer('timestamp',
                        value_deserializer=lambda x: loads(x.decode('utf-8')))

for message in consumer:
   print(message.value)

In the above code:

• value_deserializer specifies how the received messages will be decoded.

The above code will print the timestamp received from the Event Producer.

Step 4: Execute the Kafka Event Driven Architecture

After you have completed the above steps, it’s time that you test your Kafka Event Driven Architecture. Your project structure will basically look like this:

code /
- docker-compose.yml
- producer
-- main.py
- consumer
-- main.py

In your directory code, the Kafka and Zookeeper can be started using the following command:

docker-compose up -d

Next, change to producer directory and activate the environment using the following command:

$ source venv/bin/activate
(venv) $ python3 main.py

Similarly, do it for the consumer directory.

Now, you will see the output of the above Kafka Event Driven Architecture build as shown below. The Event Producer is sending timestamps and the Event consumer is listening to them and printing them.

Hurray! You have learned the basic steps to get started with Kafka Event Driven Architecture. If you wish to generate and print timestamps in JSON format, you can refer to this article.

Benefits & Use Cases of Kafka Event Driven Architecture

Using a scalable Kafka Event Driven Architecture, you can generate and respond to a huge number of events in real-time seamlessly. Microservices, which are loosely connected software, benefit greatly from an Event Driven design. These architectures are very versatile since they operate well with unpredictable and non-linear events. 

Key Benefits of Kafka Event Driven Architecture

Let’s discover some of the benefits of Kafka Event Driven Architecture:

• Decoupling: Brokers decouple services to make it easier to add new ones or adjust existing ones. Hence, the long sequences are broken and synced workflows can be decomposed easily.
• Efficient State Transfer: Your system’s dataset includes Events. Streams are a convenient way to distribute datasets such that they can be reassembled and queried within a limited environment. When resources are available, messages are buffered and consumed.
• Faster Operations: It’s easy to merge, join, and enrich data from several services. Joins are simple and quick to be executed. For high-volume Event processing, Event Driven services scale easily.
• Easy Traceability: When there’s a central, immutable, retentive narrative documenting each action as it unfolds in time, it’s simpler to debug the errors and issues.

Use Cases of Kafka Event Driven Architecture

The below figure depicts some of the common use cases in various industries where the Kafka Event Driven Architecture can be implemented:

Conclusion

• In this article, you gain an in-depth understanding of Kafka Event Driven Architecture.
• You understood the need for Event Driven Architecture, its component, and commonly deployed patterns in the industry.
• In addition, you learned the key steps to building a Kafka Event Driven Architecture using Python.
• At the end of this article, you discovered some of the use cases and benefits of Kafka Event Driven Architecture.
• Hence, with the increase in popularity of Event Driven Architectures, it is critical to identify the right method to boost your business workflow and simplify complex tasks.

Feel free to share your experience of building Kafka Event Driven Architecture with us in the comments section below!