Understanding RabbitMQ Queue & Messaging Simplified 101

Companies are now shifting towards the adoption of microservice-based architecture in modern applications. To handle and distribute the workload for smooth and fast functioning of the application, using decoupled modules became a necessity. Message brokers allow applications to communicate and decouple from each other.

RabbitMQ is a widely used open-source message broker that helps in scaling the application by deploying a message queuing mechanism in between the two applications. It offers temporary storage for data preventing data loss. RabbitMQ Queue takes messages from the publisher and sends them to the consumer.

RabbitMQ works as an intermediary platform that ensures the message is delivered to the right destination. In this article, you will learn about RabbitMQ, its basic terminologies, and how to message through RabbitMQ. You will also go through the steps to create a Python program to send messages through RabbitMQ Queue.

Table of Contents

• What is RabbitMQ?
  • Key Features of RabbitMQ
• What is RabbitMQ Queue?
• Properties of RabbitMQ Queue
• How does RabbitMQ Messaging Queue Work?
• Steps to Send Messages Through RabbitMQ
  • Sending Message
  • Receiving Message
• Conclusion

What is RabbitMQ?

RabbitMQ is an open-source message broker software written in Erlang. It is commonly called message-oriented middleware that implements the AMQP (Advanced Message Queuing Protocol). It is extended with a plug-in architecture to provide support for MQ Telemetry Transport (MQTT), Streaming Text Oriented Messaging Protocol (STOMP), and other protocols. RabbitMQ is highly scalable and ensures data availability all the time with fault-tolerant architecture.

RabbitMQ supports many programming languages and can run on various Cloud environments and operating systems. It offers browser-based UI for monitoring and management, and also comes with HTTP UI and CLI tools for operations. RabbitMQ is lightweight and easy to deploy on Cloud and premises. Moreover, it can be deployed in distributed as well as federated configurations to provide high scalability and availability to meet business requirements.

Basic concepts of RabbitMQ:

• Producer: A producer is the one who sends messages to a queue based on the queue name.
• Queue: A Queue is a sequential data structure that is a medium through which messages are transferred and stored.
• Consumer: A consumer is the one who subscribes to and receives messages from the broker and uses that message for other defined operations.
• Exchange: An exchange is an entry point for the broker because it takes messages from a publisher and routes those messages to the appropriate queue.
• Broker: It is a message broker which provides storage for data produced. The data is meant to be consumed or received by another application that connects to the broker with the given parameters or connection strings.
• Channel: Channels offer a lightweight connection to a broker via a shared TCP connection.

Key Features of RabbitMQ

Some of the main features of RabbitMQ are listed below:

• Distributed Deployment: Users can deploy RabbitMQ as clusters for high availability and throughput as well. The clusters are federated across multiple availability zones and regions to be available every time.
• Tools and Plugins: RabbitMQ offers a plethora of tools and plugins for continuous integration, operational metrics, and integration to other enterprise systems. 
• Enterprise and Cloud Ready: RabbitMQ is lightweight and easy to deploy on the public as well as private clouds using pluggable authentication authorization.
• Asynchronous Messaging: RabbitMQ supports multiple messaging protocols, message queuing, delivery acknowledgment, routing to queues, and different exchange types.

To learn more about RabbitMQ, click here.

What is RabbitMQ Queue?

A RabbitMQ Queue is a sequential data structure in which an item can be enqueued at the last or dequeued from the head. Publishers and Consumers communicate using a queue-like storage mechanism. RabbitMQ Queues follows FIFO (First in First out) but it also offers other queue features such as priorities and re-queueing for changing the order.

RabbitMQ Queues have names so that applications can reference them. The application picks a RabbitMQ Queue by name or asks the broker to generate a name to distinguish 2 RabbitMQ queues from each other.

Properties of RabbitMQ Queue

RabbitMQ Queues have properties that define how they behave. There are a few mandatory as well as optional properties of RabbitMQ Queue, listed below:

• RabbitMQ Queue must have a name.
• RabbitMQ Queue is durable so it can survive a broker restart.
• RabbitMQ Queue can be Exclusive that is used by only connection and the Queue will be deleted when the connection is closed.
• The RabbitMQ Queue auto-deletes when the last consumer unsubscribes.
• RabbitMQ Queue has optional arguments such as queue length limit, message TTL, etc.

How does RabbitMQ Messaging Queue Work?

In this section, you will learn how the message flow RabbitMQ takes place. The following procedure is given below:

1. The producer first publishes a message to an exchange with a specified type.
2. Once the exchange receives the message, it is responsible for routing the message. The exchange routes the messages taking into consideration other parameters such as exchange type, routing key, etc.
3. Now bindings are created from exchange to the RabbitMQ Queues. Each Queue has a name that helps in distinguishing the two. Then the exchange routes the message into Queues based on the attributes of the message.
4. Once the messages are enqueued in the Queue, it remains there until they are handled by a consumer.
5. The consumer then handles the message from the RabbitMQ Queue.

Steps to Send Messages Through RabbitMQ

Now that you have understood what RabbitMQ is, its basic terminologies, and how messages flow through RabbitMQ Queue. In this section, you will go through the process to send messages through RabbitMQ using Python. The following steps to send a message through RabbitMQ are listed below:

• Sending Message
• Receiving Message 

For this tutorial, the Pika Python client which is a Python library will be used. Here, you need to write two programs: one for sending messages and the other for receiving messages. The producer or sender will send the message and RabbitMQ Queue will keep the message as a buffer on behalf of the consumer and then send the message to the consumer.

Sending Message

• First, install the Pika Python client using the command given below:

python -m pip install pika --upgrade

• Create a new Python file with the name send.py for sending messages.
• First, you need to establish a connection with your RabbitMQ Queue or Server using the following code given below.

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

• Once you run the program, it will connect you to the broker on localhost. If you want to connect to a broker on a different machine, simply provide its name or IP address in place of localhost.
• Let’s create a RabbitMQ Queue with the name “queue1”. You can provide any other name of your choice. The following code is given below.

channel.queue_declare(queue=’queue1’)

• It will ensure that the RabbitMQ Queue exists and the message will not drop by the broker.
• A RabbitMQ Message never travels to the RabbitMQ Queue directly. It needs to go through an exchange. The exchange should know the RabbitMQ Queue name where the message needs to be sent.
• The following code to publish a message to exchange is given below.

channel.basic_publish(exchange='',
                      routing_key='queue1',
                      body='Hi, Msg 1')
print(" [x] Sent 'Hi, Msg 1'")

• Here, the RabbitMQ Queue name should be specified in the routing_key parameter.
• Now, let’s close the connection to ensure the network buffers were flushed and the message was delivered to RabbitMQ.
• You can close the connection using the code given below.

connection.close()

• The send.py file will look like this in the end.

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='queue1')

channel.basic_publish(exchange='', routing_key='queue1', body='Hi, Msg 1')
print(" [x] Sent 'Hi, Msg 1'")
connection.close()

Receiving Message 

• Now, let’s create another new Python file with the name receive.py for receiving messages from the Queue.
• First, import the modules required by using the code given below.

import pika, sys, os

• Then, establish the connection in the main() function similarly, done before while sending the message. The following code is given below.

def main():
    connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()

• Now, let’s add more code to the main() function.
• Create a Queue with the name you used before. The code is given below.

channel.queue_declare(queue='queue1’)

• Receiving RabbitMQ Message from the RabbitMQ Queue works by subscribing a callback function to a Queue.
• The code for the callback function is given below.

def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

• Next, you need to tell RabbitMQ that the particular callback function should receive messages from the “queue1” Queue. The code to consume the message is given below.

channel.basic_consume(queue=’queue1’,
                      auto_ack=True,
                      on_message_callback=callback)

• Let’s start consuming the messages using the code given below.

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

• Also, let’s add all the code in the infinite loop which will stop on keyboard interrupt and else run a callback function whenever required to wait for data. The code is given below.

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print('Interrupted')
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)

• The receive.py file will look like this in the end.

import pika, sys, os

def main():
    connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()

    channel.queue_declare(queue='queue1')

    def callback(ch, method, properties, body):
        print(" [x] Received %r" % body)

    channel.basic_consume(queue=’queue1’, on_message_callback=callback, auto_ack=True)

    print(' [*] Waiting for messages. To exit press CTRL+C')
    channel.start_consuming()

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print('Interrupted')
        try:
            sys.exit(0)
        except SystemExit:
            os._exit(0)

• Let’s run the receive.py file first to wait for messages and then run the send.py file. 
• Once both the programs are running, it will show the similar output shown below.

python send.py

# => [x] Sent 'Hello World!'

python receive.py

# => [*] Waiting for messages. To exit press CTRL+C
# => [x] Received 'Hi, Msg 1'

That’s it! You have successfully sent a message through the RabbitMQ.

Conclusion

In this article, you learnt about RabbitMQ Queues and Messaging, its essential terminologies such as Producer, Exchange, Consumer, Queue, etc. You also read how message flows through RabbitMQ from producer to consumer and went through the steps to create a Python program to send and receive messages through RabbitMQ. RabbitMQ is the most widely deployed message broker that helps companies develop and manage scalable applications.

It is essential to store these data streams in Data Warehouses and run Analytics on them to generate insights. Hevo Data is a No-code Data Pipeline solution that helps to transfer data from Kafka and 100+ sources to desired Data Warehouse. It fully automates the process of transforming and transferring data to a destination without writing a single line of code.

Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand.

Share your experience of learning about RabbitMQ Queue and Messaging in the comments section below!