AWS Simple Queue Service (SQS) lets you decouple and communicate between components of a distributed system through a fully managed messaging system. Messages can be sent, stored, and received at any volume with no loss of data or dependency on other systems.
As a managed service, Amazon SQS also eliminates the overhead of managing and operating message-based middleware, enabling developers to focus on key tasks instead of application development or infrastructure management.
This article explains how Amazon SQS works, with a specific focus on the Amazon SQS ReceiveMessage method, that can retrieve any messages (up to 10) from a queue. We’ll also have a look at how you can send and receive multiple messages in Amazon SQS using some examples.
How Does Message Queuing Work?
A Message Queue enables multiple processes to communicate asynchronously. Message Queuing Systems have two basic components:
It is more efficient to send data in the form of messages asynchronously, instead of directly calling each other and awaiting a response from the other. This level of dependencies creates bottlenecks for message senders since they have to wait for message consumers to process the message and send back a response.
Message Queues solve this dependency bottleneck by storing messages in sequential order in a queue. When a sender sends a message, it gets stored in the queue, and the sender is given an acknowledgment that the message has been received successfully. The same messages are then transferred to consumers when they go available.
In this system, messages can be retrieved from the queue by consumers and processed without the knowledge of the producer programs. As a result, the communicating programs are able to operate independently of each other, at different times and speeds, and without direct communication or interdependency between them.
AWS SQS Core Concepts
AWS Simple Queue Service (SQS) provides Distributed Message Queuing under the control of Amazon Web Services. As part of the SQS service, the messages are redundantly stored across multiple Amazon SQS servers.
In Amazon SQS, two kinds of Message Queues are available:
- Standard Queues: In Standard Queues, message throughput is maximum, ordering is best-effort, and delivery is guaranteed. By default, SQS uses the Standard Queue. When using Standard Queues, your applications should be designed to be idempotent so that processing a message more than once does not negatively impact the system.
- FIFO Queues: A FIFO Queue (First-In-First-Out) is used when an application wants to communicate in a specific order, or when it wants to avoid re-processing of duplicate messages. With FIFO Queues, each message will be processed only once, in the same order as it was sent.
Are you looking for an ETL tool to migrate your data from AWS sources? Migrating your data can become seamless with Hevo’s no-code intuitive platform. With Hevo, you can:
- Automate Data Extraction: Effortlessly pull data from various sources and destinations with 150+ pre-built connectors.
- Transform Data effortlessly: Use Hevo’s drag-and-drop feature to transform data with just a few clicks.
- Seamless Data Loading: Quickly load your transformed data into your desired destinations, such as BigQuery.
- Transparent Pricing: Hevo offers transparent pricing with no hidden fees, allowing you to budget effectively while scaling your data integration needs.
Try Hevo and join a growing community of 2000+ data professionals who rely on us for seamless and efficient migrations.
Get Started with Hevo for Free
Ordering and Deduplication With FIFO Queues
Messages in a FIFO Queue are delivered in the order in which they are received, and they are delivered exactly once. All these messages are ordered based on a Message Group ID. Amazon SQS stores messages that arrive for processing in the order they are sent to a FIFO Queue. To ensure sequential storage and transmission, each producer has to use a unique Message Group ID to send all its messages.
Every message belonging to the same Message Group is processed in a strict order one by one. FIFO queues also prevent duplicate messages from being sent to a queue. If you send the same message within a 5-minute deduplication interval, it won’t be added to the Message Queue.
There are two ways to configure deduplication:
- Enabling Content-Based Deduplication: SQS generates a Message Deduplication ID using SHA-256 hashes based on the content of the body of a message when this property is enabled.
- Providing the Message Deduplication ID: After a message has been sent with a particular Message Deduplication ID, any messages sent later with the same Message Deduplication ID are accepted successfully but are not delivered within the 5-minute deduplication period.
Queue Configurations
After creating a Message Queue, we must configure it with specific attributes according to our message processing requirements. Let’s look at a few of them:
1. Dead-letter Queue: This is a queue for messages that are not successfully consumed by one or more source queues. We can use them to debug our applications or messaging systems, as they let us identify unconsumed messages and determine why their processing did not succeed.
2. Dead-letter Queue Redrive: This configuration allows us to specify how long unconsumed messages were held in Dead-Letter Queues before they were moved to their Source Queues.
3. Visibility Timeout: Visibility timeout prevents other consumers from viewing messages received from a queue by a single consumer. In the visibility timeout period, Amazon SQS does not allow other consumers to receive and process the message.
4. Message Retention Period: The message retention period points to the time messages are held in the queue. Before this time is crossed, the queued messages should be received and processed. After the retention period, the messages are automatically deleted from the queue.
5. DelaySeconds: The period of delay from the time a message is queued to the time it is delivered.
6. MaximumMessageSize: The maximum size an SQS message can be in bytes before getting rejected.
7. ReceiveMessageWaitTimeSeconds: Message receivers wait for a message to arrive for a specific amount of time. In this value, 0 is the default value and any value between 0 and 20 seconds can be used.
8. Short and Long Polling: To receive messages from a queue, Amazon SQS uses short polling and long polling mechanisms. In short polling, a response sampled on SQS ReceiveMessage call is returned immediately, even if the message queue is empty. But in long polling, a response sampled on SQS ReceiveMessage call is returned after a message is received in the queue, or after the long polling period has expired. Short polling is the default for the SQS client. In most cases, long polling is preferred.
SQS ReceiveMessage Options and Parameters
AttributeNames — (Array<String>)
There should be a list of attributes that each message should contain. Some of these attributes are:
- All – Returns the entire set of values.
- ApproximateFirstReceiveTimestamp – Indicates when the messages were first received from the queue (in milliseconds).
- ApproximateReceiveCount – This function counts how many times a message was received across all queues, but not deleted.
- AWSTraceHeader – This function gets the AWS X-Ray trace header string.
- SenderId
- Gives the user’s IAM ID, for instance, ABCDEFGHI1JKLMNOPQ23R.
- Returns the ID of an IAM role, for instance, ABCDE1F2GH3I4JK5LMNOP:i-a123b456.
- SentTimestamp – This function returns the time at which the message was passed to the queue (in milliseconds).
- SqsManagedSseEnabled –Uses the SQS-owned encryption keys to enable server-side queue encryption. One server-side encryption method per queue is supported (e.g. SSE-KMS or SSE-SQS).
- MessageDeduplicationId – Provides a value that is provided by the producer when SendMessage is called.
- MessageGroupId – The SendMessage action returns the value provided by the producer. The message groups with the same IDs are returned sequentially.
- SequenceNumber – Returns the value provided by Amazon SQS.
MessageAttributeNames — (Array<String>)
The name of the message attribute, where N is its index.
- Names can contain alphanumeric characters and underscores (_), hyphens (-), and periods (.).
- Names for attributes in a message must be unique and case-sensitive.
- The name cannot begin with an AWS-reserved prefix like AWS, or Amazon. (or any other casing variant).
- Names should not begin with or end with a period (.), nor should they have consecutive periods (..).
- Up to 256 characters can be used in the name.
SQS ReceiveMessage
SQS ReceiveMessage allows you to specify an attribute list to receive, or all attributes by specifying All or .* in your request. Additionally, you can use message attributes that start with a prefix, for example, bar.*.
Type: Array of strings
Required: No
QueueUrl — (String)
QueueUrl is the URL for the Amazon SQS ReceiveMessage Queue where messages are received. The URLs and names of queues are case-sensitive.
Type: String
Required: Yes
ReceiveRequestAttemptId — (String)
This parameter is applicable only to FIFO Queues. Tokens are used to deduplicate SQS ReceiveMessage calls. In the event that a networking issue occurs after an SQS ReceiveMessage action and you receive a generic error message instead of a response, you can retry the action using the same ReceiveRequestAttemptId to retrieve the same set of messages, regardless of whether their visibility timeout has ended.
- ReceiveRequestAttemptId can be used only for 5 minutes after an SQS ReceiveMessage action.
- You can specify a ReceiveRequestAttemptId explicitly in the FIFO Queue action when setting it.
- Amazon SQS generates a ReceiveRequestAttemptId if a caller of the SQS ReceiveMessage action does not supply a ReceiveRequestAttemptId.
- You can retry the SQS ReceiveMessage action with the same ReceiveRequestAttemptId if none of the messages have been modified (deleted or had their visibility changed).
- When a visibility timeout occurs, subsequent messages and receipt handle with the same ReceiveRequestAttemptId are returned. The visibility timeout is reset when a retry occurs in the deduplication interval.
- After a given MessageGroupId becomes invisible, no further messages belonging to that group are returned until the visibility timeout expires. As long as it is visible, you can still receive messages with another MessageGroupId.
- ReceiveMessage cannot retry until the visibility timeout expires if CallerReceiveMessage cannot track the ReceiveRequestAttemptId. Delays may occur as a result, but messages continue to be queued in a strict order.
ReceiveRequestAttemptId can have up to 128 characters. ReceivingRequestAttemptId can contain alphanumeric characters (a-z, A-Z, 0-9) and punctuation (!”#$%&'()*+,-./:;<=>?@[]^_`{|}~).
SQS ReceiveMessage Example
Creating a FIFO SQS Queue
You can create FIFO queues and Standard Queues using the Amazon SQS console. Except for queue names, the console uses default values when setting up the queue.
In our examples, we will create the queues and send and receive messages by using the AWS SDK for Java. The Amazon Web Services SDK provides a set of Java libraries that are based on common design patterns familiar to Java developers for easy integration with AWS Services.
We must first add the following Maven dependencies to pom.xml in order to set up the AWS SDK for Java:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
</dependency>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.17.116</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Next, let’s create a FIFO Queue and put non-duplicate messages into it in sequential order. This will be done in the createFifoQueue() method:
public class ResourceHelper {
private static Logger logger
= Logger.getLogger(ResourceHelper.class.getName());
public static void main(String[] args) {
createFifoQueue();
}
public static void createFifoQueue() {
SqsClient sqsClient = getSQSClient();
// A FIFO queue's attributes are defined by an attribute map
Map<QueueAttributeName, String> attributeMap
= new HashMap<QueueAttributeName, String>();
// Set the FIFO_QUEUE attribute to true to mark the queue as FIFO
attributeMap.put(
QueueAttributeName.FIFO_QUEUE, "true");
// Scope of DEDUPLICATION is set to messageGroup
attributeMap.put(
QueueAttributeName.DEDUPLICATION_SCOPE, "messageGroup");
// CONTENT_BASED_DEDUPLICATION is disabled
attributeMap.put(
QueueAttributeName.CONTENT_BASED_DEDUPLICATION, "false");
// Preparing a queue creation request, and ending the queue name with a fifo
CreateQueueRequest createQueueRequest
= CreateQueueRequest.builder()
.queueName("myfifoqueue.fifo")
.attributes(attributeMap)
.build();
// Seting up First-In-First-Out queue
CreateQueueResponse createQueueResponse
= sqsClient.createQueue(createQueueRequest);
// Returns the URL of the queue
logger.info("url " + createQueueResponse.queueUrl());
}
private static String getQueueArn(
final String queueName,
final String region) {
return "arn:aws:sqs:" + region + ":"
+ AppConfig.ACCOUNT_NO + ":"
+ queueName;
}
private static SqsClient getSQSClient() {
...
}
}
We have created a queue named myfifoqueue.fifo. FIFO Queue names must include the .fifo extension.
SQS does not detect duplicate messages sent to our FIFO Queue by checking their content, since the property: contentBasedDeduplication is set to false. As a result, SQS will look in the message for a property named messageDeduplicationId, which needs to be explicitly sent when a message is sent to a FIFO Queue. Messages with the same value of messageDeduplicationId will be considered duplicates by SQS.
In addition, the deduplicationScope property of the queue is set to MESSAGE_GROUP, which means that duplicate messages will be detected by the message group. You can also set deduplicationScope to QUEUE.
How to Send a Message to a FIFO Queue
In a FIFO Queue, messages are sent and received in the order they are received.
Here are five messages you can send to the FIFO Queue to test this behavior:
public class MessageSender {
private static Logger logger
= Logger.getLogger(MessageSender.class.getName());
public static void sendMessageToFifo() {
SqsClient sqsClient = getSQSClient();
Map<String, MessageAttributeValue> messageAttributes
= new HashMap<String, MessageAttributeValue>();
...
final String queueURL = "https://sqs.us-east-1.amazonaws.com/"
+ AppConfig.ACCOUNT_NO
+ "/myfifoqueue.fifo";
Contains a list of duplicate IDs that should be sent with different messages.
List<String> dedupIds = List.of("dedupid1",
"dedupid2",
"dedupid3",
"dedupid2",
"dedupid1");
String messageGroupId = "signup";
// Sending a list of messages. 2 of them are duplicates
List<String> messages = List.of(
"My fifo message1",
"My fifo message2",
"My fifo message3",
"My fifo message2", // Duplicate message
"My fifo message1"); // Duplicate message
short loop = 0;
Sending the above messages sequentially. There will be duplicate messages, but they won't be received for (String message : messages) {
// message is identified as duplicate
// if deduplication id is already used
SendMessageRequest sendMessageRequest
= SendMessageRequest.builder()
.queueUrl(queueURL)
.messageBody(message)
.messageAttributes(messageAttributes)
.messageDeduplicationId(dedupIds.get(loop))
.messageGroupId(messageGroupId)
.build();
SendMessageResponse sendMessageResponse
= sqsClient
.sendMessage(sendMessageRequest);
logger.info("message id: " + sendMessageResponse.messageId());
loop += 1;
}
sqsClient.close();
}
}
The following is an example of the output generated by this program:
message id and sequence no.: 9529ddac-8946-4fee-a2dc-7be428666b63 | 18867399222923248640
message id and sequence no.: 2ba4d7dd-877c-4982-b41e-817c99633fc4 | 18867399223023088896
message id and sequence no.: ad354de3-3a89-4400-83b8-89a892c30526 | 18867399223104239872
message id and sequence no.: 2ba4d7dd-877c-4982-b41e-817c99633fc4 | 18867399223023088896
message id and sequence no.: 9529ddac-8946-4fee-a2dc-7be428666b63 | 18867399222923248640
The message identifier is returned along with the sequence number when SQS accepts the message. We can see that the Sequence number assigned to each message by Amazon SQS is a large and non-consecutive number.
Five messages are being sent to the queue myfifoqueue.fifo, two of which are duplicates. The messageDeduplicationId property in the message is calculated by SQS when duplicate messages are detected because the contentBasedDeduplication property has been set to false.
In My fifo message1 and My fifo message2, the messageDeduplicationId is the same while in My fifo message3, the messageDeduplicationId is different.
After the messages are consumed, we will only receive three unique messages in the same order as when we sent the messages to the queue.
Recommended
Conclusion
The Simple Queue Service (SQS) offered by Amazon enables decoupling and communication among distributed system components. The purpose of this article was to discuss Amazon SQS, Message Queuing as a form of asynchronous communication between two or more processes, and how messages and queues are the core components of a message queuing system. Additionally, we examined the queue, the sending and receiving of messages using SQS ReceiveMessage, and worked through some examples.
Most companies need to analyze their business data stored in multiple data sources. The data needs to be loaded to the Data Warehouse to get a holistic view of the data. While building in-house ETL pipelines requires a ton of engineering bandwidth and huge maintenance costs in the long run, it’s far better to employ the use of fast & reliable third-party ETL solutions like Hevo Data.
Hevo Data is a No-code Data Pipeline Platform that helps to transfer data from 150+ Data Sources to desired Data Warehouse. It fully automates the process of transforming and transferring data to a destination without writing a single line of code.
FAQ on AWS SQS ReceiveMessage
1. How do I receive SQS messages?
To receive SQS (Simple Queue Service) messages, use the AWS SDK or AWS Management Console to create a SQS queue listener, then implement polling or subscribe to receive messages from the queue and process them programmatically in your application or service.
2. What is the receive message wait time in SQS?
The receive message wait time in SQS (Simple Queue Service) is the maximum amount of time (in seconds) that a long polling request can wait for a message to arrive in the queue before returning an empty response.
3. What is the maximum number of messages received in SQS?
The maximum number of messages that can be received in a single SQS (Simple Queue Service) receive message API call is 10 messages per request.
4. What is an SQS empty receive?
An SQS empty receive refers to a receive message operation on an SQS queue that returns no messages because the queue is temporarily empty at the time of the request.
5. How to check if the SQS queue is empty or not?
To check if an SQS queue is empty, use the AWS SDK or AWS Management Console to retrieve the ApproximateNumberOfMessages attribute of the queue.
Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand.
Share your experience of learning about AWS SQS ReceiveMessage in the comments section below!
Samuel is a versatile writer specializing in the data industry. With over seven years of experience, he excels in data science, data integration, and data analysis, crafting engaging content on these topics. He is also adept at WordPress development. Samuel holds a Bachelor's degree in Computer Science from Lagos State University.