The Ultimate Guide on Using Pulsar Message Queue With Easy Steps 101

Apache Pulsar is emerging as the new go-to platform for enterprises who need to efficiently transport their data in these days of increasing data. It includes data pipelines, microservices, instant messaging, data integration, and more, all of which are integrated into a single platform to satisfy a wide range of real-time event-streaming demands. Pulsar has been adopted by hundreds of firms since it was open-sourced in 2016, including Verizon Media, Yahoo! Japan, Tencent, Comcast, and Overstock.

This blog article is mainly aimed at explaining the method or procedure to employ Apache Pulsar message queue. Moreover, to make it easy to understand, this blog article will shed some light on the key concepts that are involved during this procedure. 

Table of Contents

• What is Apache Pulsar?
• What is Message Queue?
• Apache Pulsar Message Queue Process
  • Pulsar Message Queue: Producers and Consumers
  • Pulsar Message Queue: Pub-Sub Messaging
  • Pulsar Message Queue: Using Pulsar as a Message Queue
  • Pulsar Message Queue: Controlling Message Dispatch
  • Pulsar Message Queue: Pulsar Java Client Configuration
• Conclusion

What is Apache Pulsar?

Thanks to virtualization, nowadays, software is found involved in almost everything. Most of what we do in our day to day routine, from sending an email to using a cellular phone to making a call or sending a text message, depends on the software architecture of the systems that you use. Concerning the software architecture, among other elements, Pulsar message queues are considered of great import and can be found in nearly every modern software and scalable architecture. With the help of message queues, asynchronous processing is introduced between different components and layers in an architecture. Asynchronous processing allows multiple workflow processes to run simultaneously. 

Message queues, containing messages that are sent by one application to the other, act as a medium of communication. In general, message queues are handled by different systems. Apache Pulsar is one among many other available systems or software which is very much popular nowadays because of its upgraded and modernized features. Moreover, owing to the features that it is equipped with, it can handle hundreds of billions of events every day.

Pulsar Message Queue is the right choice because of the reason it was built to have persistent message storage. Furthermore, it also provides automatic load balancing to all over the consumers for messages on a topic. To understand how to use pulsar as a message queue, some basic concepts need to be clear. Succeeding paragraphs will shed light upon some of the basic concepts and will describe the procedure to use pulsar message queue. 

What is Message Queue?

In general, a queue can be used to refer to objects that make themselves part of a line by entering into it from its beginning point and then waiting to be handled and processed sequentially. Similarly, a queue of messages that are used by different applications to establish communication between them is called a message queue. The data delivered by the sender and received by the receiver application is referred to as a message. 

Message can be anything, but to understand, we can take the example of a directive message which carries a direction or instruction for the system. Upon reaching, the given message instructs or directs the system to begin processing a task, or finish a task, or it can also be a plain message with some instructions. 

Apache Pulsar Message Queue Process

Pulsar is a pub-sub messaging and streaming platform. It was developed by Yahoo! in 2015, but later on, in the year 2017, It was contributed to the Apache Software Foundation as a top-level project. Originally, Pulsar was designed to aid internet-scale applications. Pulsar works on the scheme of a horizontally-scalable distribution system, which capacitates it to stream data without having it lost. Pulsar can handle all sorts of data communication use cases required these days.

Pulsar Message Queue: Producers and Consumers

Message producers create a message, whereas message consumers being receivers of the message take responsibility and charge to have the task completed as per the instructions issued in the message by the producers. In the whole process, the messages are carried by the Pulsar message queues, which perform the duty to take messages from producers to the consumers. To understand, we can take the example of a restaurant.

A person namely, A (message producer), places his order (message request) and communicates to the waiter (Pulsar message queue), who carries the order and delivers it to the kitchen (message consumer). Kitchen (being the message consumer) makes arrangements and issues directions to have the order (task) prepared as required by Mr A (message producer).   

Pulsar Message Queue: Pub-Sub Messaging

Message producers create a message, whereas message consumers being receivers of the message take responsibility and charge to have the task completed as per the instructions issued in the message by the producers. In the whole process, the messages are carried by the Pulsar message queues, which take messages from producers to the consumers. To understand, we can take the example of a restaurant.

A person namely, A (message producer), places his order (message request) and communicates to the waiter (Pulsar message queue), who carries the task and delivers it to the kitchen (message consumer). Kitchen (being the message consumer) makes arrangements and issues directions to have the order prepared as required by Mr A (message producer).   

The quality of a pub-sub messaging system is that it has a broker component in its system, which after receiving messages from the producers, organizes them into topics. When the messages are organized into topics they are automatically received by their subscribers, owing to which publishers don’t have to wait for subscribers to receive them. 

Among other functions, the broker component of the pub-sub system has the responsibility to determine whether messages are pushed out to subscribers or if the subscribers will pull the messages down from the broker.

Besides, the broker component of the pub-sub system maintains the order and sequence in which topic messages are received by the broker so that they may be available to the subscribers in the same order. Until and unless the subscriber has not acknowledged that it has successfully received and processed the message, each message is preserved by the pulsar.

Pulsar Message Queue: Using Pulsar as a Message Queue

The below-mentioned diagram shows two types of use cases in which pulsar is used, without special configuration or adjustment, as a Pulsar message queue.

In the first use case, pub-sub producers and the pub-sub consumers communicate with one another through a pub-sub topic, whereas in the second use case, queue producers and queue consumers communicate with each other through a queued topic. Pulsar can run both cases side by side without getting them mixed up. Topics of both the cases need not be marked or pre-destined about their status as real-time or queued.

The difference between the topics lies in that message queue topic will need consumers to use shared subscriptions rather than exclusive or failover subscriptions (plus, all consumers must use the same subscription name, or else it’s simply not the same subscription). When consumers establish a shared subscription on a topic, Pulsar automatically load balances between consumers receiving messages, which is optimal for Pulsar message queues.

Pulsar Message Queue: Controlling Message Dispatch

There is no negation to the importance that throughput has a relation to a Pulsar message queue. A message queue that is devoid of throughput can be highly harmful as in the absence of throughput, the queue would not have any idea about the requirements of its surrounding data pipeline. If you’re using Pulsar as a message queue, you can fine-tune processing throughput by adjusting your consumers’ configuration.

While using Apache Pulsar, consumers can process multiple messages at any instant through the receiver queue. The default size of the consumer’s receiver queue is 1000 messages, however, the configuration can be performed, and the size can be increased or decreased to meet the requirement. One of the most realistic approaches is to have the queue size of a receiver according to the speed with which the consumer tends to process the message.

If processing tasks, for instance, require only milliseconds, then extend the receiver queue, enabling consumers to accelerate the processing throughput.

On the flip side, if processing tasks need more time than a millisecond, it’s recommended to set a smaller receiver queue size. To illustrate, we can have an example of consumers who are performing CPU-intensive batch processing operations. In the illustrated scenario, operations may require from several seconds to a longer period to be performed.

Therefore, in order to have a load balancer distribute messages in an orderly way across consumers, we need to minimize the size of the receiver queue.  

An example of a client with a minimum size of receive is here:

Pulsar Message Queue: Pulsar Java Client Configuration

Java producer, consumer, readers, TableView of messages and administrative tasks can be created through a Pulsar Java client. All the processes in producer, consumer, readers and TableView of a Java client are protected and safe. To perform the Java client configuration, install the updated version of the Pulsar Java client through Maven central. 

<!-- in your <properties> block -->
<pulsar.version>2.10.0</pulsar.version>

<!-- in your <dependencies> block -->
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>${pulsar.version}</version>
</dependency>

A pulsar protocol URL is required to connect pulsar through client libraries.

def pulsarVersion = '2.10.0'

dependencies {
    compile group: 'org.apache.pulsar', name: 'pulsar-client', version: pulsarVersion
}

To initiate a PulsarClient object using a URL, add the following lines:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

The Java consumer configuration that wants to utilize a shared subscription is shown as follows: 

import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.SubscriptionType;

String SERVICE_URL = "pulsar://localhost:6650";
String TOPIC = "persistent://public/default/mq-topic-1";
String subscription = "sub-1";

PulsarClient client = PulsarClient.builder()
        .serviceUrl(SERVICE_URL)
        .build();

Consumer consumer = client.newConsumer()
        .topic(TOPIC)
        .subscriptionName(subscription)
        .subscriptionType(SubscriptionType.Shared)
        // If you'd like to restrict the receiver queue size
        .receiverQueueSize(10)
        .subscribe();

Conclusion

Pulsar acts as a single-stop provider to offer you messaging and streaming services. Small and medium-sized enterprises deploy this platform to solve the existing pain points and achieve limited operational costs compared to other competitive solutions. With Pulsar, it is easy to manage, maintain and integrate several systems as it continuously keeps an eye over pub-sub, message queueing, event streaming and all of your processing need in one go. 

However, as a Developer, extracting complex data from a diverse set of data sources like Databases, CRMs, Project management Tools, Streaming Services, and Marketing Platforms to your Database can seem to be quite challenging. If you are from non-technical background or are new in the game of data warehouse and analytics, Hevo Data can help!

Visit our Website to Explore Hevo

Hevo Data will automate your data transfer process, hence allowing you to focus on other aspects of your business like Analytics, Customer Management, etc. This platform allows you to transfer data from 100+ multiple sources to Cloud-based Data Warehouses like Snowflake, Google BigQuery, Amazon Redshift, etc. It will provide you with a hassle-free experience and make your work life much easier.

Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite first hand.

You can also have a look at our unbeatable pricing that will help you choose the right plan for your business needs!