RabbitMQ Queue Types 101: A Comprehensive Guide to Messaging in RabbitMQ

Business around the world uses several SaaS applications that need to communicate seamlessly with each other. With several network protocols and every application having its unique architecture, you need a solution that can provide a simplified asynchronous (delayed) messaging system. RabbitMQ is such an open-source message queuing solution that acts as a message broker. 

According to your use case, there are several RabbitMQ Queue Types that allow several services in your application to interact with each other without worrying about message loss while ensuring the quality of service (QoS) requirements. It also enables better control and efficient message routing facilitating extensive decoupling of applications.

In this article, you will learn about various RabbitMQ Queue Types in detail.  

Table of Contents

• What is RabbitMQ?
  • When and Why should you use RabbitMQ?
• What are RabbitMQ Queues?
• Understanding RabbitMQ Queue Types
  • RabbitMQ Queue Types: Distributed and Replicated Queues
  • RabbitMQ Queue Types: Temporary Queues
  • RabbitMQ Queue Types: Exclusive Queues
  • RabbitMQ Queue Types: Server-named Queues
• Key Considerations for RabbitMQ Queues
• Conclusion

What is RabbitMQ?

RabbitMQ is a popular open-source Message Queue software that allows several applications to interact and scale up. Acting as a messaging broker, RabbitMQ provides you with a common platform that supports Asynchronous messaging. Here, applications can securely send and receive messages and RabbitMQ ensures a safe environment for your messages to live until consumed.

RabbitMQ is designed using Erlang for building a stable, reliable, fault-tolerant, and highly scalable system that effectively handles multiple operations in parallel. It also supports different types of protocols(method of data transportation between devices). Introduced with support for Advanced Message Queuing Protocol (AMQP), it now also provides complete support for Message Queuing Telemetry Transport (MQTT), Streaming Text Oriented Messaging Protocol (STOMP), and several other common protocols.

Internally, the architecture of all the RabbitMQ Queue types is simple. There is a client application called a producer that creates a message and sends it to a broker (message queue). Other applications, called consumers, link to the Message queues and subscribe to the messages that are processed.  All of the Queued messages are stored until the consumer extracts the message. The RabbitMQ software can act as a message producer or consumer, or both a consumer and a producer.

When and Why should you use RabbitMQ?

Using RabbitMQ Message Queues, you can avoid unnecessary response delays as it allows web servers to respond to requests quickly rather than being forced to perform resource-intensive tasks on the spot. Employing any of the RabbitMQ Queue Types is often a good choice when you need to distribute a message among multiple consumers(message receivers) or even for balancing the load between workers. Owing to the flexibility of all the RabbitMQ Queue Types, you can send a request in one programming language and handle it in some other programming language. You can use RabbitMQ in the following scenarios:

• When you need a finer and more constant per-message control.
• For applications that use several messaging techniques such as pub/sub, point-to-point, and request/response. 
• Message Queuing is helpful if you need to perform the same job on a specific server, a group of servers, or all servers. Your application might send the message and, exchange forwards it. All of this is achievable by the robust routing features of RabbitMQ.
• For cases when you want to combine your messaging broker with your existing IT infrastructure.

What are RabbitMQ Queues?

RabbitMQ Queues is an ordered data structure that allows you to add(enqueue) at the tail and retrieve(dequeue) from the head. Many messaging protocols and platforms consider publishers and consumers to interact using a queue-like storage method. It is to be noted that all the RabbitMQ Queue Types are based on the FIFO (“first-in, first-out”) principle. To understand the RabbitMQ Queues, you can go through the following aspects: 

• Naming Conventions
• Queue Properties
• Property Equivalence and Declaration
• Durability
• Message Ordering

Naming Conventions

All the Rabbit Queues Types have to be properly named allowing the applications to reference them easily. These applications can either choose queue names or let the broker generate a name for them. While naming the queue, the following guidelines are followed:

• Queue names can be up to 255 bytes of UTF-8 characters.
• The names starting with “amq.” are reserved for internal use by the broker. If you try to state a queue with a name that violates this rule will result in a channel-level exception with reply code 403 (ACCESS_REFUSED).

Queue Properties

Queue Properties determine how a queue behaves. You can note down the following set of mandatory and optional properties:

• Queue Name
• Durable: Determines if a queue will survive a broker restart.
• Exclusive: You can use it with only one connection and the queue will be deleted when that connection is closed.
• Auto-delete: As soon as the last consumer exits, the queue that has had at least one consumer is deleted.
• Arguments: This is an optional property. Mainly utilized by plugins and broker-specific features such as Queue type (e.g. quorum or classic), Message and queue TTL, Queue length limit, Legacy classic queue mirroring settings, max number of priorities, consumer priorities, etc.

When queues like auto-delete or exclusive use a known (static) name and the client disconnects and reconnects immediately, the RabbitMQ node will delete such queues and then try to reconnect them. This leads to a natural race condition of recovering the client. This can lead to errors and exceptions during client-side reconnection, causing unnecessary confusion and impacting application availability.

Property Equivalence and Declaration

You must declare a queue before you can use it. When you declare a queue, it is created if the queue does not already exist. In case a queue already exists and its attributes match the attributes in the declaration, the declaration has no impact. If the existing queue attribute does not match the attribute of the declaration, a channel-level exception with code 406 (PRECONDITION_FAILED) will be thrown.

Durability

You can either choose durable or transient RabbitMQ Queues. Metadata of a durable queue is stored on disk, while metadata of a transient queue is stored in memory when possible. You can assume a similar difference for messages at publishing time in some protocols, e.g. AMQP 0-9-1 and MQTT.

For cases where durability is essential, use durable queues and ensure that publish mark published messages as persisted. All the transient queues are deleted when the node boots and therefore, by design, it does not survive a node reboot. This also results in discarding all the Messages in transient queues On the other hand, Durable queues will be recovered on node boot along with the messages in them published as persistent. It is to be noted that even if the messages are stored in a durable queue, they will be deleted during the recovery if they were published as transient.

Message Ordering

You can define Queues RabbitMQ as sequenced collections of messages. Messages are enqueued and dequeued (delivered to consumers) in the FIFO manner. However, you may not always get the FIFO order, for instance, for priority and sharded RabbitMQ Queue Types. 

• Maintaining the specific order can be difficult, especially when there are several competing consumers, consumer priorities, and message redeliveries. This is applicable across all kinds of redeliveries i.e automatic after-channel closure and negative consumer acknowledgments. 
• An application can expect messages published on a single channel to be queued in the sequence they were published on all queues to which they are routed. If the publication occurs on multiple connections or channels, those message sequences are forwarded and interleaved at the same time. 
• A consumer application can assume that the first delivery to a single consumer (with the redelivered property set to false) will run in the same FIFO order as it was queued. For repetitive deliveries (if the redelivered property is set to true), the original order can be affected by consumer confirmation and redelivery timing and cannot be guaranteed. 
• For several consumers, the messages are dequeued for delivery in FIFO order, but the actual delivery is to several consumers. If all consumers have the same priority, they will be selected on a round-robin basis. Only consumers on channels that do not exceed the prefetch value (the number of pending deliveries) are taken into consideration.

Understanding RabbitMQ Queue Types

Check out the following RabbitMQ Queue Types:

• RabbitMQ Queue Types: Distributed and Replicated Queues
• RabbitMQ Queue Types: Temporary Queues
• RabbitMQ Queue Types: Exclusive Queues
• RabbitMQ Queue Types: Server-named Queues

RabbitMQ Queue Types: Distributed and Replicated Queues

• You can consider Quorum queues as replicated, data security, and consistency-oriented RabbitMQ queue types. 
• Classic queues were previously used to support replication. However, they have been deprecated and you should avoid them. 
• Queues can also be federated between loosely joined nodes or clusters. Although, it is to be noted that intra-cluster replication and federation are orthogonal functions and thus, you should not consider them as direct alternatives. 
• Streams are another replicated data structure supported by RabbitMQ with a unique combination of supported operations and functions.

RabbitMQ Queue Types: Temporary Queues

You will often encounter that with some workloads, their respective queues don’t last that much. You also may choose to manually delete the queues you have declared before disconnection, though this is not an effective method. Moreover, the client connections often fail, thereby leading to unused resources(queues) being left behind.

Thus, you can consider Temporary Queues as another of the RabbitMQ Queue Types. There are potentially three ways to make the queue deleted automatically:

• Exclusive queues 
• TTLs 
• Auto-delete queues

An auto-delete queue will be deleted when its last consumer is canceled (e.g. using the basic.cancel in AMQP 0-9-1) or exits (closed channel or connection, or lost TCP connection with the server).

There will be cases where a queue never had any consumers. For example, when all consumption happens using the basic.get method (the “pull” API), it won’t be automatically deleted. For such cases, you have to use exclusive queues or queue TTL.

RabbitMQ Queue Types: Exclusive Queues

• You can only use Exclusive queues (consumed, cleaned up, deleted, etc.) by declaring a connection. If you try to use an exclusive queue from another connection, you will get the channel-level exception RESOURCE_LOCKED with an error message stating that exclusive access to the locked queue is not possible. 
• You will observe that exclusive queues get deleted when the declaration of a connection is closed or gone. For example, your exclusive queue is deleted because the underlying TCP connection is lost. Therefore, they are only suitable for customer-specific transient states. 
• It is good practice to declare the exclusive queues as server-named. 
• Coming under the popular RabbitMQ Queue Types, exclusive queues are declared on the “client local” node (the node to which the client declaring the queue is connected), regardless of the queue_master_locator value.

RabbitMQ Queue Types: Server-named Queues

In AMQP 0-9-1, your application can ask the broker to generate a unique queue name. You can employ this feature by sending an empty string as the queue name argument. You can get the same generated name through various methods in the same channel by using the empty string where a queue name is required. This concept is applicable as the channel remembers the last server-generated queue name.

Server-named queues are ideal for a state that is transient and applicable to a particular consumer (application instance). This allows the applications to send such names in message metadata to let other applications respond to them. Or else, you should know the names of server-named queues and utilized them only by the declaring application instance. The instance is also responsible for properly setting up appropriate bindings (routing) for the queue so that publishers can use well-known exchanges rather than using the server-generated queue name directly.

Key Considerations for RabbitMQ Queues

While dealing with various RabbitMQ Queue Types, you need to consider the following aspects:

• In-Memory Storage
• Length Limit and Time-to-Live
• Monitoring and Metrics
• Parallelism and CPU Utilization
• Acknowledgments and Consumers

In-Memory Storage

• All RabbitMQ Queue Types store their messages in RAM and/or on disk. For some protocols such as AMQP 0-9-1, the client can also decide where to save this data. In AMQP 0-9-1, you can configure this via a message property (delivery_mode or, in some clients, persistent).
• Whenever you are publishing messages as transient, it represents that RabbitMQ should keep as many messages as possible in RAM. Although, Queues will even send the transient messages to the disk when they are under memory pressure.
• Persistent messages sent to durable queues are persisted in batches or when a specific amount of time passes (fraction of a second).
• RabbitMQ Queue Types like Lazy queues send messages out to disk more aggressively regardless of their persistence property.

Length Limit and Time-to-Live

For all the RabbitMQ Queue Types, you can easily limit their length. Also, both Queues and messages can have a TTL(Time-to-Live). You can use both of these functions for data expiration as well as a method to limit the resources (RAM, disk space) a queue can use at most. For example, when consumers go offline or their throughput falls behind publishers.

Monitoring and Metrics

RabbitMQ records several parameters about all the RabbitMQ Queue Types. You can easily access them via RabbitMQ HTTP API and management UI, which is designed for monitoring. These metrics include queue length, ingress and egress rates, number of consumers, number of messages in various states (for instance, ready for delivery or unacknowledged), number of messages in RAM vs. on disk, etc.

You can also use the rabbitmqctl to list queues and some basic metrics. By using the rabbitmq-top plugin, you can easily monitor runtime metrics like VM scheduler usage, queue (Erlang) process GC activity, amount of RAM used by the queue process, queue process mailbox length, etc. You can also view the individual queue pages in the management UI.

Parallelism and CPU Utilization

Whether your queue replica is a leader or follower, it is limited to a single CPU core on the hot code path. Therefore, by design, it assumes that most practical systems use several queues. On a general note, a single queue is usually assumed as an anti-pattern (not just because of resource usage).  If there is a need to replace the order of messages with parallelism (improved CPU core utilization), rabbitmqsharding provides a great way for clients to do that transparently.

Acknowledgments and Consumers

Messages can be used by registering a consumer (subscription). That is, RabbitMQ either sends the message to the client or retrieves it separately for a protocol that supports the message (such as the basic.get AMQP 091 method), as with HTTP GET. Delivered messages are explicitly or automatically verified by the consumer as soon as the delivery is written to the connection socket. 

 Auto acknowledgment mode typically provides higher throughput and uses less network bandwidth. However, with regard to failure, the warranty is the least. As a good practice, you should first use the manual verification mode.

Conclusion

In this article, you have learned about various RabbitMQ Queue Types. RabbitMQ is a widely used messaging broker that provides a safe environment for your applications to send or receive messages securely. This open-source tool provides you with several RabbitMQ Queue Types that can be used according to your business needs. With auto-delete queues, TTL & exclusive queues you can easily delete queues automatically when required. All the RabbitMQ Queue Types can be configured and monitored based on various parameters and metrics respectively.

As you collect and manage your data in RabbitMQ and across several applications and databases in your business, it is important to consolidate it for a complete performance analysis of your business. However, it is a time-consuming and resource-intensive task to continuously monitor the Data Connectors. To achieve this efficiently, you need to assign a portion of your engineering bandwidth to Integrate data from all sources, Clean & Transform it, and finally, Load it to a Cloud Data Warehouse, BI Tool, or a destination of your choice for further Business Analytics. All of these challenges can be comfortably solved by a Cloud-based ETL tool such as Hevo Data.   

Visit our Website to Explore Hevo

Hevo Data, a No-code Data Pipeline can transfer data in Real-Time from a vast sea of 150+ Data sources to a Data Warehouse, BI Tool, or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!  

If you are using Messaging systems like Kafka and searching for a no-fuss alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo, with its strong integration with 100+ sources and BI tools(Including 40+ Free Sources), allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.

Share your experience of learning about various RabbitMQ Queue Types! Let us know in the comments section below!