Understanding RabbitMQ Queue & Messaging Simplified 101

on message broker, Message Queue, message queuing, MQ, Python, RabbitMQ, Tutorials • April 22nd, 2022

RabbitMQ Queue

Companies are now shifting towards the adoption of microservice-based architecture in modern applications. To handle and distribute the workload for smooth and fast functioning of the application, using decoupled modules became a necessity. Message brokers allow applications to communicate and decouple from each other.

RabbitMQ is a widely used open-source message broker that helps in scaling the application by deploying a message queuing mechanism in between the two applications. It offers temporary storage for data preventing data loss. RabbitMQ Queue takes messages from the publisher and sends them to the consumer.

RabbitMQ works as an intermediary platform that ensures the message is delivered to the right destination. In this article, you will learn about RabbitMQ, its basic terminologies, and how to message through RabbitMQ. You will also go through the steps to create a Python program to send messages through RabbitMQ Queue.

Table of Contents

What is RabbitMQ?

RabbitMQ Logo - RabbitMQ Queue
Image Source

RabbitMQ is an open-source message broker software written in Erlang. It is commonly called message-oriented middleware that implements the AMQP (Advanced Message Queuing Protocol). It is extended with a plug-in architecture to provide support for MQ Telemetry Transport (MQTT), Streaming Text Oriented Messaging Protocol (STOMP), and other protocols. RabbitMQ is highly scalable and ensures data availability all the time with fault-tolerant architecture.

RabbitMQ supports many programming languages and can run on various Cloud environments and operating systems. It offers browser-based UI for monitoring and management, and also comes with HTTP UI and CLI tools for operations. RabbitMQ is lightweight and easy to deploy on Cloud and premises. Moreover, it can be deployed in distributed as well as federated configurations to provide high scalability and availability to meet business requirements.

Basic concepts of RabbitMQ:

  • Producer: A producer is the one who sends messages to a queue based on the queue name.
  • Queue: A Queue is a sequential data structure that is a medium through which messages are transferred and stored.
  • Consumer: A consumer is the one who subscribes to and receives messages from the broker and uses that message for other defined operations.
  • Exchange: An exchange is an entry point for the broker because it takes messages from a publisher and routes those messages to the appropriate queue.
  • Broker: It is a message broker which provides storage for data produced. The data is meant to be consumed or received by another application that connects to the broker with the given parameters or connection strings.
  • Channel: Channels offer a lightweight connection to a broker via a shared TCP connection.

Key Features of RabbitMQ

Some of the main features of RabbitMQ are listed below:

  • Distributed Deployment: Users can deploy RabbitMQ as clusters for high availability and throughput as well. The clusters are federated across multiple availability zones and regions to be available every time.
  • Tools and Plugins: RabbitMQ offers a plethora of tools and plugins for continuous integration, operational metrics, and integration to other enterprise systems. 
  • Enterprise and Cloud Ready: RabbitMQ is lightweight and easy to deploy on the public as well as private clouds using pluggable authentication authorization.
  • Asynchronous Messaging: RabbitMQ supports multiple messaging protocols, message queuing, delivery acknowledgment, routing to queues, and different exchange types.

To learn more about RabbitMQ, click here.

Simplify Data Streaming Using Hevo’s No Code Data Pipeline

Hevo Data, an Automated No Code Data Pipeline, a Fully-managed Data Pipeline platform, can help you automate, simplify & enrich your data replication process in a few clicks. With Hevo’s wide variety of connectors and blazing-fast Data Pipelines, you can extract & load data from 100+ data sources straight into your Data Warehouse or any Databases. To further streamline and prepare your data for analysis, you can process and enrich raw granular data using Hevo’s robust & built-in Transformation Layer without writing a single line of code!

Get Started with Hevo for Free

Hevo is the fastest, easiest, and most reliable data replication platform that will save your engineering bandwidth and time multifold. Try our 14-day full access free trial today to experience an entirely automated hassle-free Data Replication!

What is RabbitMQ Queue?

RabbitMQ Queue Example
Image Source

A RabbitMQ Queue is a sequential data structure in which an item can be enqueued at the last or dequeued from the head. Publishers and Consumers communicate using a queue-like storage mechanism. RabbitMQ Queues follows FIFO (First in First out) but it also offers other queue features such as priorities and re-queueing for changing the order.

RabbitMQ Queues have names so that applications can reference them. The application picks a RabbitMQ Queue by name or asks the broker to generate a name to distinguish 2 RabbitMQ queues from each other.

Properties of RabbitMQ Queue

RabbitMQ Queues have properties that define how they behave. There are a few mandatory as well as optional properties of RabbitMQ Queue, listed below:

  • RabbitMQ Queue must have a name.
  • RabbitMQ Queue is durable so it can survive a broker restart.
  • RabbitMQ Queue can be Exclusive that is used by only connection and the Queue will be deleted when the connection is closed.
  • The RabbitMQ Queue auto-deletes when the last consumer unsubscribes.
  • RabbitMQ Queue has optional arguments such as queue length limit, message TTL, etc.

How does RabbitMQ Messaging Queue Work?

RabbitMQ Queue: Working of RabbitMQ Messaging Queue
Image Source

In this section, you will learn how the message flow RabbitMQ takes place. The following procedure is given below:

  1. The producer first publishes a message to an exchange with a specified type.
  2. Once the exchange receives the message, it is responsible for routing the message. The exchange routes the messages taking into consideration other parameters such as exchange type, routing key, etc.
  3. Now bindings are created from exchange to the RabbitMQ Queues. Each Queue has a name that helps in distinguishing the two. Then the exchange routes the message into Queues based on the attributes of the message.
  4. Once the messages are enqueued in the Queue, it remains there until they are handled by a consumer.
  5. The consumer then handles the message from the RabbitMQ Queue.

Steps to Send Messages Through RabbitMQ

Now that you have understood what RabbitMQ is, its basic terminologies, and how messages flow through RabbitMQ Queue. In this section, you will go through the process to send messages through RabbitMQ using Python. The following steps to send a message through RabbitMQ are listed below:

  • Sending Message
  • Receiving Message 

For this tutorial, the Pika Python client which is a Python library will be used. Here, you need to write two programs: one for sending messages and the other for receiving messages. The producer or sender will send the message and RabbitMQ Queue will keep the message as a buffer on behalf of the consumer and then send the message to the consumer.

Sending Message

  • First, install the Pika Python client using the command given below:
python -m pip install pika --upgrade
  • Create a new Python file with the name send.py for sending messages.
  • First, you need to establish a connection with your RabbitMQ Queue or Server using the following code given below.
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
  • Once you run the program, it will connect you to the broker on localhost. If you want to connect to a broker on a different machine, simply provide its name or IP address in place of localhost.
  • Let’s create a RabbitMQ Queue with the name “queue1”. You can provide any other name of your choice. The following code is given below.
channel.queue_declare(queue=’queue1’)
  • It will ensure that the RabbitMQ Queue exists and the message will not drop by the broker.
  • A RabbitMQ Message never travels to the RabbitMQ Queue directly. It needs to go through an exchange. The exchange should know the RabbitMQ Queue name where the message needs to be sent.
  • The following code to publish a message to exchange is given below.
channel.basic_publish(exchange='',
                      routing_key='queue1',
                      body='Hi, Msg 1')
print(" [x] Sent 'Hi, Msg 1'")
  • Here, the RabbitMQ Queue name should be specified in the routing_key parameter.
  • Now, let’s close the connection to ensure the network buffers were flushed and the message was delivered to RabbitMQ.
  • You can close the connection using the code given below.
connection.close()
  • The send.py file will look like this in the end.
import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='queue1')

channel.basic_publish(exchange='', routing_key='queue1', body='Hi, Msg 1')
print(" [x] Sent 'Hi, Msg 1'")
connection.close()

What Makes Hevo’s Data Streaming and Loading Unique

Providing a high-quality ETL solution can be a difficult task if you have a large volume of data. Hevo Data‘s automated, No-code platform empowers you with everything you need to have for a smooth data replication experience.

Check out what makes Hevo amazing:

  • Fully Managed: Hevo requires no management and maintenance as it is a fully automated platform.
  • Data Transformation: Hevo provides a simple interface to perfect, modify, and enrich the data you want to transfer.
  • Faster Insight Generation: Hevo offers near real-time data replication so you have access to real-time insight generation and faster decision making. 
  • Schema Management: Hevo can automatically detect the schema of the incoming data and map it to the destination schema.
  • Scalable Infrastructure: Hevo has in-built integrations for 100+ sources (with 40+ free sources) that can help you scale your data infrastructure as required.
  • Live Support: Hevo team is available round the clock to extend exceptional support to its customers through chat, email, and support calls.

Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo.

Receiving Message 

  • Now, let’s create another new Python file with the name receive.py for receiving messages from the Queue.
  • First, import the modules required by using the code given below.
import pika, sys, os
  • Then, establish the connection in the main() function similarly, done before while sending the message. The following code is given below.
def main():
    connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()
  • Now, let’s add more code to the main() function.
  • Create a Queue with the name you used before. The code is given below.
channel.queue_declare(queue='queue1’)
  • Receiving RabbitMQ Message from the RabbitMQ Queue works by subscribing a callback function to a Queue.
  • The code for the callback function is given below.
def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
  • Next, you need to tell RabbitMQ that the particular callback function should receive messages from the “queue1” Queue. The code to consume the message is given below.
channel.basic_consume(queue=’queue1’,
                      auto_ack=True,
                      on_message_callback=callback)
  • Let’s start consuming the messages using the code given below.
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
  • Also, let’s add all the code in the infinite loop which will stop on keyboard interrupt and else run a callback function whenever required to wait for data. The code is given below.
if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print('Interrupted')
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)
  • The receive.py file will look like this in the end.
import pika, sys, os

def main():
    connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()

    channel.queue_declare(queue='queue1')

    def callback(ch, method, properties, body):
        print(" [x] Received %r" % body)

    channel.basic_consume(queue=’queue1’, on_message_callback=callback, auto_ack=True)

    print(' [*] Waiting for messages. To exit press CTRL+C')
    channel.start_consuming()

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print('Interrupted')
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)
  • Let’s run the receive.py file first to wait for messages and then run the send.py file. 
  • Once both the programs are running, it will show the similar output shown below.

python send.py

# => [x] Sent 'Hello World!'

python receive.py

# => [*] Waiting for messages. To exit press CTRL+C
# => [x] Received 'Hi, Msg 1'

That’s it! You have successfully sent a message through the RabbitMQ.

Conclusion

In this article, you learnt about RabbitMQ Queues and Messaging, its essential terminologies such as Producer, Exchange, Consumer, Queue, etc. You also read how message flows through RabbitMQ from producer to consumer and went through the steps to create a Python program to send and receive messages through RabbitMQ. RabbitMQ is the most widely deployed message broker that helps companies develop and manage scalable applications.

It is essential to store these data streams in Data Warehouses and run Analytics on them to generate insights. Hevo Data is a No-code Data Pipeline solution that helps to transfer data from Kafka and 100+ sources to desired Data Warehouse. It fully automates the process of transforming and transferring data to a destination without writing a single line of code.

Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand.

Share your experience of learning about RabbitMQ Queue and Messaging in the comments section below!

Data Engineering
Survey 2022
Calling all data engineers – fill out this short survey to help us build an industry report for our data engineering community.
TAKE THE SURVEY
Amazon Gift Cards of $25 each are on offer for all valid and complete survey submissions.