Redis Message Queue: 4 Easy Steps to Build a Message Broker

A message broker is a software that enables systems, applications, and services to communicate and exchange information. It translates messages between formal messaging protocols and allows interdependent services to “talk” directly, even if written in different languages or implemented on various platforms. It is common to use a message broker when using asynchronous communication for microservices.

A message broker ensures reliable and stable communication where the messages are managed and monitored to ensure they aren’t lost. There are a few brokers you can choose from, ranging in scale and data capabilities. Redis or REmote DIctionary Server is an advanced NoSQL key-value data store used as a cache, database, and message broker. It provides tools like Redis message queue for message broking. It is known for its rich data types, fast read and writes operations, and advanced memory structure. Redis is ideal for developing scalable, high-performance web applications. 

In this article, you will learn about Message brokers, Redis, and the steps to build a message broker using the Redis message queue. 

Benefits of Message Broker

Here are the benefits of using a message broker:

Better Performance

Message queues allow for asynchronous communication, meaning the endpoints produce and consume messages that interact with the queue. Message producers can add requests to the queue without waiting for processing. Message consumers process messages when they become available, optimizing data flow.

Increased Reliability

Queues increase reliability by improving data persistence and reducing errors that arise when different parts within a system are offline. Separating various system components via message queues creates more fault tolerance. If one piece is unreachable, the other component can continue interacting with the queue.

Granular Scalability

Queues offer granular scalability. As workloads peak, multiple components can continue to add requests to the queue without risking collision. You can distribute the workload across numerous message consumers. It allows message queues, message producers, and message consumers to scale up and down on demand.

Simplified Decoupling

Message queues eliminate dependencies between system components and simplify decoupling. When software components aren’t burdened by communication code, they can be designed to perform a discrete business function. Queues are a simple way to decouple any distributed system, including monolithic applications, microservices, or serverless architectures. 

How do message brokers work?

Here are some basic concepts of a message broker:

• Producer – The application responsible for sending messages. It’s called publishers in the publish/subscribe pattern. 
• Consumer – The endpoint takes messages waiting in the message broker. In the publish/subscribe pattern, they are called subscribers. 
• Queue/topic – A folder in a filesystem used by message brokers to store messages.

Introduction to Redis

Redis, or Remote Dictionary Server, is a single-threaded, in-memory, and open-source NoSQL datastore. It is written in C by Salvatore Sanfilippo and launched in 2009. The high performance of Redis makes it a favorite data store, cache, and even lightweight message broker by using the tools like Redis message queue. According to DB-Engines ranking, Redis is the most popular key-value data store. It falls in the middle of a data continuum from relatively unstructured to structured data, where the key can be any binary sequence, a string, or a JPEG file’s contents.

Features of Redis

In-memory datastore

Redis data is stored in the server’s main memory, unlike other conventional relational databases. In-memory data stores don’t suffer the same penalty as non-memory databases. It gives applications super-fast performance and support for millions of operations per second.

Data Persistence

Data Persistence means that the data survives even if the server fails. For a data store to be persistent, it must write on permanent storage (non-volatile, such as a hard disk). Redis supports point-in-time backups and RDB, AOF persistence mechanism to persist the data to the non-volatile memory. 

Rich Data Structures

Redis is more than just a standard key-value datastore technology. It offers a wide variety of data structures to meet applications’ needs. It enables users to implement applications according to the client’s requirements not associated with the technology limitations. Available data structures are listed below:

• Strings: A text up to 512MB.
• Sets: An unordered group of strings with the ability to do set operations.
• Sorted sets: Unique and non-repeating string elements.
• Lists: A collection of strings that preserves the adding order.
• Hashes: For storing a list of fields and values.
• Bitmaps: For bit-level operations carried on the primary Redis data structure strings.
• HyperLogLogs: Probabilistic data structure to estimate the unique items in a data set.

High availability and scalability

Redis offers a primary-replica architecture in a single node primary or clustered topology. It allows you to build highly available solutions providing consistent performance and reliability. As Redis is single-threaded, only one thread is associated with I/O operations. Redis server uses master-slave architecture to increase the load.

Simplicity

Redis Supported languages include C, C++, C#, Java, JavaScript, Node.js, Python, PHP, Ruby, R, etc. The Redis comes with native data structures, as mentioned above. Redis has extensive open community support. Redis is also a very simplified technology. We need fewer lines integrated to store, access, and use data with our applications than other technology. 

Format of pushed messages in Redis message queue Pub/Sub

A message is an array-reply with three elements in the Redis message queue:

• Subscribe: means the receiver is successfully subscribed to the channel given as the second element in the reply. The third argument will represent the number of channels to which the receiver is currently subscribed.
• unsubscribe: The receiver has successfully unsubscribed from the channel given as the second element in the reply. The third argument will represent the number of channels to which the receiver is currently subscribed. The last argument will be zero when you are no longer subscribed to any channel.
• message: an announcement received from a PUBLISH command issued by another client. The second element is the name of the originating channel, and the third argument is the actual message payload.

Steps to Build a Message Broker Using Redis message queue

Redis message queue supports Pub/Sub messaging with pattern matching and many varieties of data structures such as sorted sets, lists, and hashes. A subscriber in the Redis message queue Pub/Sub system can subscribe to any number of channels, and a publisher of the Redis message queue can publish the message to any channel. This decoupling of publisher and subscriber in Redis message queue Pub/Sub messaging allows scalability and flexibility.

Step 1: Connecting to Redis message queue

import redis
# Connect to a local redis instance
r = redis.Redis(host='localhost', port=6379, db=0)

Step 2: Writing/Reading to a stream:

Code for writing a stream directly:

event = {"eventType": "purchase", "amount": 5, "item_id": "XXX"}
r.xadd("stream_key", '*', event)
# the `*` means that redis generates and event id automatically

Code for reading a stream directly:

last_id = '$' # `$` means only new messages
while True:
events = r.xread({"stream_key": last_id}, block=0, count=10)
for _, e in events:
print(f"new event, amount: {e['amount']}")
last_id = e['id']

Step 3: Publishing on Redis message queue Pub/Sub:

# publish a message to the `redis` channel
r.publish("redis", "hello world")

Step 4: Subscribing to a channel on Redis message queue Pub/Sub:

sub = r.pubsub()
sub.subscribe("redis")
while True:
msg = sub.get_message()
print(f"new message: {msg['data']}")

Conclusion

In this post, we’ve looked at message brokers, Redis, and set up a message broker using the Redis message queue. One of the main advantages of message brokers using Redis message queue is that you can communicate with clients irrespective of their languages. Redis message queue includes popular solutions and clients such as Ruby, Python, PHP, Java, Objective-C, Node.js, Clojure, C++, C#, etc. Using Redis message queue Pub/Sub within your company’s infrastructure, you will enjoy quick messaging and communication between processes and applications. The Redis message queue’s simplicity and performance make it a popular choice for a wide range of use cases. 

There are various trusted sources that companies use as it provides many benefits but transferring data from it into a data warehouse is a hectic task. The Automated data pipeline helps in solving this issue and this is where Hevo comes into the picture. Hevo Data is a No-code Data Pipeline and has awesome 100+ pre-built Integrations that you can choose from.

visit our website to explore hevo

Hevo can help you Integrate your data from 100+ data sources and load them into a destination to Analyze real-time data. It will make your life easier and data migration hassle-free. It is user-friendly, reliable, and secure.

SIGN UP for a 14-day free trial and see the difference!

Share your experience of learning about the Redis message queue in the comments section below.