Kafka Partitions: 3 Easy Steps to Create and Use

Apache Kafka is an event-streaming platform that handles daily massive amounts of real-time data. To manage this data efficiently, Kafka uses Topics to organize messages divided into Kafka Partitions across multiple servers in a Kafka Cluster. This partitioning helps distribute data, ensuring fault tolerance and preventing data loss in case of server failure. Even if one server fails, the data remains available across other servers. This article will explore Apache Kafka, its partitions, and how to create topic partitions for better data distribution and reliability.

Prerequisites

• Fundamental knowledge of Streaming Data.

What is Apache Kafka?

Apache Kafka is an open-source data streaming platform designed for handling real-time, continuously streaming data. It enables data collection, storage, and organization within distributed servers or brokers. Kafka facilitates the development of event-driven applications and systems that can make decisions based on real-time data streams. Known as a publish-subscribe messaging service, it allows producers to publish data and consumers to subscribe and retrieve it as needed. Kafka’s distributed architecture and high throughput make it a popular choice for major companies, including over 80% of Fortune 100 firms like Netflix, Goldman Sachs, and Target.

Key Features of Kafka

• Real-time Data Streaming: Supports continuous data streams in real-time for event-driven applications.
• Distributed Architecture: Comprises multiple servers or brokers that store and manage data efficiently.
• Publish-Subscribe Messaging: Producers publish data, while consumers subscribe and read from Kafka servers.
• High Throughput: Kafka can handle large volumes of data with low latency, ensuring fast processing.
• Scalability: Easily scales horizontally by adding more brokers to handle growing data loads.
• Fault Tolerance: Replication ensures data availability even if some servers fail.
• Widely Adopted: Trusted by leading companies, including major players like Netflix and Uber, for data-driven operations.

How does Apache Kafka Topic Partitions Work?

In the Apache Kafka Ecosystem, messages or data received from Producers are stored and organized inside an entity called Topics. Inside Kafka Brokers, Topics are further subdivided into multiple parts called Kafka Partitions. The representation of Topic Partitions is similar to linear data structures, like arrays, which store and append data whenever new data arrives in Kafka Brokers.

• Producers store messages inside topics for Consumers to access and fetch from respective Kafka Servers.

Producer partition strategies

Strategy	Description
Default partitioner	The key hash maps messages to partitions. Null key messages are sent to a partition in a round-robin fashion.
Round-robin partitioner	Messages are sent to partitions in a round-robin fashion.
Uniform sticky partitioner	Messages are sent to a sticky partition (until the batch.size is met or linger.ms time is up) to reduce latency.
Custom partitioner	This approach implements the Partitioner interface to override the partition method with some custom logic that defines the key-to-partition routing strategy.

Consumer assignment strategies

Strategy	Description
Range assignor (default)	(Total number of partitions) / (Number of consumers) partitions are assigned to each consumer. The aim is to have co-localized partitions, i.e., assigning the same partition number of two different topics to the same consumer (P0 of Topic X and P0 of Topic Y to the same consumer).
Round-robin assignor	Partitions are picked individually and assigned to consumers (in any rational order, say from first to last). When all the consumers are used up but some partitions still remain unassigned, they are assigned again, starting from the first consumer. The aim is to maximize the number of consumers used.
Sticky assignor	This approach works similarly to a round-robin assignor but preserves as many existing assignments as possible when partitions are reassigned. The aim is to reduce or completely avoid partition movement during rebalancing.
Custom assignor	Extends the AbstractPartitionAssignor class and overrides the assign method with custom logic.

• Apache Kafka stores Producers’ Messages inside different partitions of a specific Topic across various Kafka Brokers in a Kafka Cluster.
• Topics are logical entities, while the actual data storage occurs in Topic Partitions.

Example of Kafka Topic Partitioning

Consider an example where you create:

• Topic-A with 3 partitions across 3 brokers: “Broker 101,” “Broker 102,” and “Broker 103.”
• Topic-B with 2 partitions across “Broker 101” and “Broker 102.”

This demonstrates how topics are internally partitioned across different Kafka Brokers or Servers in a Kafka Cluster.

You can also decide the number of partitions while creating topics by executing commands in the command prompt.

• A Topic Partition is identified by a Partition Number, uniquely representing it within a topic.
• Example: Partition 0, Partition 1, and Partition 2 uniquely identify the partitions of a single Kafka Topic.

Message Storage in Kafka Partitions

• Each partition stores messages in a log file.
• New messages are always appended to the rear end of the partition.
• Data within a partition is ordered based on the arrival period (older to newer).
• Each partition contains an ordered sequence of numbers called Offsets.
• Offsets are immutable and cannot be changed once messages are published.
• Example: Messages M1, M2, M3, etc., are stored in order in a topic partition.

Role of Offsets

• Offsets are used by Kafka consumers to fetch messages from a specific topic partition.
• When a producer writes a message, the log file gets appended with a new Offset.

High Availability

When creating topics, Kafka achieves high availability using a Replication Factor parameter.

• Replication Factor defines the number of replicas of a single topic partition.
• Replicas are distributed across different Kafka servers, ensuring data safety.

Example:

• Topic 1 with two partitions and a replication factor of 2.
  • Partition 0 is distributed between Broker 1 and Broker 2.
  • Partition 1 is distributed between Broker 2 and Broker 3.

Fault Tolerance

• If a Kafka server fails or shuts down, messages remain available in other servers, ensuring no data loss.
• This replication mechanism makes Kafka a fault-tolerant and scalable platform, ensuring the safety and security of data.

How to Create Partition in Kafka Topic?

In Kafka, you can create Topic Partitions and set configurations only while creating Topics. To create Topic Partitions, you have to create Topics in Kafka as a prerequisite. Below are the steps to create Kafka Partitions.

• Kafka Partitions Step 1: Check for Key Prerequisites
• Kafka Partitions Step 2: Start Apache Kafka & Zookeeper Severs
• Kafka Partitions Step 3: Creating Topics & Topic Partitions

Kafka Partitions Step 1: Check for Key Prerequisites

Before proceeding into the steps for creating a Kafka Topic Partition, ensure that Kafka and Zookeeper are pre-installed, configured, and running on your local machine. You have to also make sure that the Java 8+ Version is installed and running on your computer.

Further, set up the File Path and Java_Home environment variables for enabling your operating system to point or head towards the Java utilities.

Kafka Partitions Step 2: Start Apache Kafka & Zookeeper Servers

After all the above-mentioned prerequisites are satisfied, you are now all set to start and set up the Kafka and Zookeeper Servers. Initially, you have to start the Kafka Server. For that, open the Command Prompt or Windows PowerShell to execute the Kafka Commands.

You can use Batch scripts (.bat) to work with Kafka Configuration while working with the Windows Operating System. If you are using the Linux Operating System, you can use Shell Scripts (.sh) to proceed further with Kafka configurations.

This article concentrates on creating Kafka Topics and Partition configurations using a command-line tool in Windows OS. For that, you can use the (.bat) scripts in Kafka. 

For starting the Kafka server, execute the following command.

.binwindowskafka-server-start.bat .configserver.properties

Then, open a new command terminal for starting the Zookeeper Server and execute the following command:

zookeeper-server-start.bat .configzookeeper.properties

After executing the above commands, Kafka and Zookeeper Servers are started and running successfully. Ensure you do not close both the command windows that run Zookeeper and Kafka Instances. 

Kafka Partitions Step 3: Creating Topics & Topic Partitions

Now, you are ready to create Topics in Kafka. Open another command prompt and execute the following command. 

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

The above command will successfully create a new Kafka Topic in the name of “Topic Test.” with One Partition and One Replication Factor. The Replication Factor is the number of copies or replicas of a topic partition across the Kafka Cluster. 

After the execution of the command, you will get a success message saying “Created Topic Test.” in your command terminal. From this message, you can ensure that Kafka’s Topics are successfully created for sending and receiving events or data. 

Since you provide the Partition Parameter as 1, Kafka will create a single partition under the topic named “Topic Test.” Similarly, according to the command mentioned above, Apache Kafka will create a single replication factor for the Respective Topic Partition.

However, if the size of your message is large, Kafka will allow you to create many partitions or divisions under a Single Topic. 

For example, when you want to create a new topic with 2 Partitions and 3 Replication Factors, you can execute the command, as given below. 

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partitions 2 --topic test

You can use the same command, as shown above, for creating different topics with specific Topic Configuration parameters by just customizing the Topic Name, Number of Partitions, and Replication Factors.

With the above steps, you have successfully created Topic and Topic Partitions in Kafka.

Effective Strategies to Customize Kafka Partitions for Topics

You can perform some Configurations and Customizations while creating Kafka Partitions by choosing the Proper Number of Partitions for a specific Topic. For example, if you are setting up and running a basic Kafka Cluster with very few brokers, you can choose the approximate partitions for performing Stream Processing operations.

However, if you are about to run a high-end Kafka Cluster with a huge number of Brokers, you have to implement some effective strategies to properly Partition Messages to achieve maximum throughput. 

The first prerequisite to achieving a high degree of Throughput and Parallelism in Kafka is choosing the appropriate number for Kafka partitions across Kafka servers.

By splitting Producer Messages into Partitions over multiple Kafka Servers, end-consumers can effectively read a message with respect to the specific topic instead of searching between the messy and unorganized data.

Furthermore, in a Kafka Cluster, the larger the Partitions, the greater the message parallelization and consequently greater the throughput you can achieve by effectively splitting messages.

The simple formula to determine the number of partitions for each Kafka topic is given below.

Partitions = Desired Throughput / Partition Speed

The default Partition speed of a single Kafka Topic is 10MB/s. For example, consider that the desired message throughput is 5 TB per day, about 58 MB/s. By keeping the default partition speed as 10MB/s, calculate the number of Partitions.

When you apply the parameters in the formula, you will get 5.8, which can be approximated as 6. Now, you can confirm that your Apache Kafka Topics need six partitions to achieve the maximum throughput. 

Conclusion

In this article, we covered the basics of Apache Kafka, including Kafka Topics and Kafka Partitions, and explored how Kafka partitions work within its architecture. You learned how partitions allow Kafka to efficiently handle and distribute large amounts of real-time data across multiple servers, ensuring fault tolerance and preventing data loss. While there are various strategies for implementing Kafka Topic Partitions, such as the Round Robin and Range Assignor methods, these can be explored once you understand basic partitioning. With customized partition strategies, you can optimize message distribution across the Kafka Cluster for specific use cases. In case you need to migrate from Kafka, you can select tools like Hevo Data, which can connect to your Target Destination in 2 Steps.

FAQ on Apache Kafka Partitions