AWS SQS ReceiveMessage: Syntax, Importance, Parameters & Examples

AWS Simple Queue Service (SQS) lets you decouple and communicate between components of a distributed system through a fully managed messaging system. Messages can be sent, stored, and received at any volume with no loss of data or dependency on other systems. 

As a managed service, Amazon SQS also eliminates the overhead of managing and operating message-based middleware, enabling developers to focus on key tasks instead of application development or infrastructure management.

This article explains how Amazon SQS works, with a specific focus on the Amazon SQS ReceiveMessage method, that can retrieve any messages (up to 10) from a queue. We’ll also have a look at how you can send and receive multiple messages in Amazon SQS using some examples.

How Does Message Queuing Work?

A Message Queue enables multiple processes to communicate asynchronously. Message Queuing Systems have two basic components: 

• Messages, and 
• Queues 

It is more efficient to send data in the form of messages asynchronously, instead of directly calling each other and awaiting a response from the other. This level of dependencies creates bottlenecks for message senders since they have to wait for message consumers to process the message and send back a response.

Message Queues solve this dependency bottleneck by storing messages in sequential order in a queue. When a sender sends a message, it gets stored in the queue, and the sender is given an acknowledgment that the message has been received successfully. The same messages are then transferred to consumers when they go available. 

In this system, messages can be retrieved from the queue by consumers and processed without the knowledge of the producer programs. As a result, the communicating programs are able to operate independently of each other, at different times and speeds, and without direct communication or interdependency between them.

AWS SQS Core Concepts

AWS Simple Queue Service (SQS) provides Distributed Message Queuing under the control of Amazon Web Services. As part of the SQS service, the messages are redundantly stored across multiple Amazon SQS servers. 

In Amazon SQS, two kinds of Message Queues are available:

• Standard Queues: In Standard Queues, message throughput is maximum, ordering is best-effort, and delivery is guaranteed. By default, SQS uses the Standard Queue. When using Standard Queues, your applications should be designed to be idempotent so that processing a message more than once does not negatively impact the system.

• FIFO Queues: A FIFO Queue (First-In-First-Out) is used when an application wants to communicate in a specific order, or when it wants to avoid re-processing of duplicate messages. With FIFO Queues, each message will be processed only once, in the same order as it was sent.

Ordering and Deduplication With FIFO Queues

Messages in a FIFO Queue are delivered in the order in which they are received, and they are delivered exactly once. All these messages are ordered based on a Message Group ID. Amazon SQS stores messages that arrive for processing in the order they are sent to a FIFO Queue. To ensure sequential storage and transmission, each producer has to use a unique Message Group ID to send all its messages. 

Every message belonging to the same Message Group is processed in a strict order one by one. FIFO queues also prevent duplicate messages from being sent to a queue. If you send the same message within a 5-minute deduplication interval, it won’t be added to the Message Queue.

There are two ways to configure deduplication:

• Enabling Content-Based Deduplication: SQS generates a Message Deduplication ID using SHA-256 hashes based on the content of the body of a message when this property is enabled.
• Providing the Message Deduplication ID: After a message has been sent with a particular Message Deduplication ID, any messages sent later with the same Message Deduplication ID are accepted successfully but are not delivered within the 5-minute deduplication period.

Queue Configurations

After creating a Message Queue, we must configure it with specific attributes according to our message processing requirements. Let’s look at a few of them:

Dead-letter Queue: This is a queue for messages that are not successfully consumed by one or more source queues. We can use them to debug our applications or messaging systems, as they let us identify unconsumed messages and determine why their processing did not succeed.

Dead-letter Queue Redrive: This configuration allows us to specify how long unconsumed messages were held in Dead-Letter Queues before they were moved to their Source Queues.

Visibility Timeout: Visibility timeout prevents other consumers from viewing messages received from a queue by a single consumer. In the visibility timeout period, Amazon SQS does not allow other consumers to receive and process the message.

Message Retention Period: The message retention period points to the time messages are held in the queue. Before this time is crossed, the queued messages should be received and processed. After the retention period, the messages are automatically deleted from the queue.

DelaySeconds: The period of delay from the time a message is queued to the time it is delivered.

MaximumMessageSize: The maximum size an SQS message can be in bytes before getting rejected.

ReceiveMessageWaitTimeSeconds: Message receivers wait for a message to arrive for a specific amount of time. In this value, 0 is the default value and any value between 0 and 20 seconds can be used.

Short and Long Polling: To receive messages from a queue, Amazon SQS uses short polling and long polling mechanisms. In short polling, a response sampled on SQS ReceiveMessage call is returned immediately, even if the message queue is empty. But in long polling, a response sampled on SQS ReceiveMessage call is returned after a message is received in the queue, or after the long polling period has expired. Short polling is the default for the SQS client. In most cases, long polling is preferred.

SQS ReceiveMessage Options and Parameters

AttributeNames — (Array<String>)

There should be a list of attributes that each message should contain. Some of these attributes are:

• All – Returns the entire set of values.
• ApproximateFirstReceiveTimestamp – Indicates when the messages were first received from the queue (in milliseconds).
• ApproximateReceiveCount – This function counts how many times a message was received across all queues, but not deleted.
• AWSTraceHeader – This function gets the AWS X-Ray trace header string.
• SenderId
  • Gives the user’s IAM ID, for instance, ABCDEFGHI1JKLMNOPQ23R.
  • Returns the ID of an IAM role, for instance, ABCDE1F2GH3I4JK5LMNOP:i-a123b456.
• SentTimestamp – This function returns the time at which the message was passed to the queue (in milliseconds).
• SqsManagedSseEnabled –Uses the SQS-owned encryption keys to enable server-side queue encryption. One server-side encryption method per queue is supported (e.g. SSE-KMS or SSE-SQS).
• MessageDeduplicationId – Provides a value that is provided by the producer when SendMessage is called.
• MessageGroupId – The SendMessage action returns the value provided by the producer. The message groups with the same IDs are returned sequentially.
• SequenceNumber – Returns the value provided by Amazon SQS.

MessageAttributeNames — (Array<String>)

The name of the message attribute, where N is its index.

• Names can contain alphanumeric characters and underscores (_), hyphens (-), and periods (.).
• Names for attributes in a message must be unique and case-sensitive.
• The name cannot begin with an AWS-reserved prefix like AWS, or Amazon. (or any other casing variant).
• Names should not begin with or end with a period (.), nor should they have consecutive periods (..).
• Up to 256 characters can be used in the name.

SQS ReceiveMessage

SQS ReceiveMessage allows you to specify an attribute list to receive, or all attributes by specifying All or .* in your request. Additionally, you can use message attributes that start with a prefix, for example, bar.*.

Type: Array of strings

Required: No

QueueUrl — (String)

QueueUrl is the URL for the Amazon SQS ReceiveMessage Queue where messages are received. The URLs and names of queues are case-sensitive.

Type: String

Required: Yes

ReceiveRequestAttemptId — (String)

This parameter is applicable only to FIFO Queues. Tokens are used to deduplicate SQS ReceiveMessage calls. In the event that a networking issue occurs after an SQS ReceiveMessage action and you receive a generic error message instead of a response, you can retry the action using the same ReceiveRequestAttemptId to retrieve the same set of messages, regardless of whether their visibility timeout has ended.

• ReceiveRequestAttemptId can be used only for 5 minutes after an SQS ReceiveMessage action. 
• You can specify a ReceiveRequestAttemptId explicitly in the FIFO Queue action when setting it.
• Amazon SQS generates a ReceiveRequestAttemptId if a caller of the SQS ReceiveMessage action does not supply a ReceiveRequestAttemptId.
• You can retry the SQS ReceiveMessage action with the same ReceiveRequestAttemptId if none of the messages have been modified (deleted or had their visibility changed).
• When a visibility timeout occurs, subsequent messages and receipt handle with the same ReceiveRequestAttemptId are returned. The visibility timeout is reset when a retry occurs in the deduplication interval. 
• After a given MessageGroupId becomes invisible, no further messages belonging to that group are returned until the visibility timeout expires. As long as it is visible, you can still receive messages with another MessageGroupId.
• ReceiveMessage cannot retry until the visibility timeout expires if CallerReceiveMessage cannot track the ReceiveRequestAttemptId. Delays may occur as a result, but messages continue to be queued in a strict order.

ReceiveRequestAttemptId can have up to 128 characters. ReceivingRequestAttemptId can contain alphanumeric characters (a-z, A-Z, 0-9) and punctuation (!”#$%&'()*+,-./:;<=>?@[]^_`{|}~).

SQS ReceiveMessage Example

Creating a FIFO SQS Queue

You can create FIFO queues and Standard Queues using the Amazon SQS console. Except for queue names, the console uses default values when setting up the queue.

In our examples, we will create the queues and send and receive messages by using the AWS SDK for Java. The Amazon Web Services SDK provides a set of Java libraries that are based on common design patterns familiar to Java developers for easy integration with AWS Services.

We must first add the following Maven dependencies to pom.xml in order to set up the AWS SDK for Java:

  <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>sqs</artifactId>
    </dependency>
    <dependencyManagement>
        <dependencies>
          <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.17.116</version>
            <type>pom</type>
            <scope>import</scope>
          </dependency>
        </dependencies>
   </dependencyManagement>

Next, let’s create a FIFO Queue and put non-duplicate messages into it in sequential order. This will be done in the createFifoQueue() method:

public class ResourceHelper {
  private static Logger logger
      = Logger.getLogger(ResourceHelper.class.getName());

  public static void main(String[] args) {
    createFifoQueue();
  }

  public static void createFifoQueue() {
    SqsClient sqsClient = getSQSClient();

    // A FIFO queue's attributes are defined by an attribute map
    Map<QueueAttributeName, String> attributeMap
        = new HashMap<QueueAttributeName, String>();

    // Set the FIFO_QUEUE attribute to true to mark the queue as FIFO
    attributeMap.put(
        QueueAttributeName.FIFO_QUEUE, "true");

    // Scope of DEDUPLICATION is set to messageGroup
    attributeMap.put(
        QueueAttributeName.DEDUPLICATION_SCOPE, "messageGroup");

    // CONTENT_BASED_DEDUPLICATION is disabled
    attributeMap.put(
        QueueAttributeName.CONTENT_BASED_DEDUPLICATION, "false");

    // Preparing a queue creation request, and ending the queue name with a fifo
    CreateQueueRequest createQueueRequest
        = CreateQueueRequest.builder()
        .queueName("myfifoqueue.fifo")
        .attributes(attributeMap)
        .build();

    // Seting up First-In-First-Out queue
    CreateQueueResponse createQueueResponse
        = sqsClient.createQueue(createQueueRequest);

    // Returns the URL of the queue
    logger.info("url " + createQueueResponse.queueUrl());
  }

  private static String getQueueArn(
      final String queueName,
      final String region) {
    return "arn:aws:sqs:" + region + ":"
        + AppConfig.ACCOUNT_NO + ":"
        + queueName;
  }

  private static SqsClient getSQSClient() {
    ...
  }
}

We have created a queue named myfifoqueue.fifo. FIFO Queue names must include the .fifo extension.

SQS does not detect duplicate messages sent to our FIFO Queue by checking their content, since the property: contentBasedDeduplication is set to false. As a result, SQS will look in the message for a property named messageDeduplicationId, which needs to be explicitly sent when a message is sent to a FIFO Queue. Messages with the same value of messageDeduplicationId will be considered duplicates by SQS.

In addition, the deduplicationScope property of the queue is set to MESSAGE_GROUP, which means that duplicate messages will be detected by the message group. You can also set deduplicationScope to QUEUE.

How to Send a Message to a FIFO Queue

In a FIFO Queue, messages are sent and received in the order they are received.

Here are five messages you can send to the FIFO Queue to test this behavior:

public class MessageSender {

  private static Logger logger
      = Logger.getLogger(MessageSender.class.getName());

  public static void sendMessageToFifo() {
    SqsClient sqsClient = getSQSClient();

    Map<String, MessageAttributeValue> messageAttributes
        = new HashMap<String, MessageAttributeValue>();
  ...

    final String queueURL = "https://sqs.us-east-1.amazonaws.com/"
        + AppConfig.ACCOUNT_NO
        + "/myfifoqueue.fifo";  

Contains a list of duplicate IDs that should be sent with different messages.

List<String> dedupIds = List.of("dedupid1",
        "dedupid2",
        "dedupid3",
        "dedupid2",
        "dedupid1");

String messageGroupId = "signup";

    // Sending a list of messages. 2 of them are duplicates
    List<String> messages = List.of(
        "My fifo message1",
        "My fifo message2",
        "My fifo message3",
        "My fifo message2", // Duplicate message
        "My fifo message1"); // Duplicate message

    short loop = 0;

Sending the above messages sequentially. There will be duplicate messages, but they won't be received for (String message : messages) {

    
  // message is identified as duplicate 
      // if deduplication id is already used 
      SendMessageRequest sendMessageRequest
          = SendMessageRequest.builder()
          .queueUrl(queueURL)
          .messageBody(message)
          .messageAttributes(messageAttributes)
          .messageDeduplicationId(dedupIds.get(loop))
          .messageGroupId(messageGroupId)
          .build();

      SendMessageResponse sendMessageResponse
          = sqsClient
          .sendMessage(sendMessageRequest);

      logger.info("message id: " + sendMessageResponse.messageId());

      loop += 1;
    }


    sqsClient.close();
  }
}

The following is an example of the output generated by this program:

message id and sequence no.: 9529ddac-8946-4fee-a2dc-7be428666b63 | 18867399222923248640
message id and sequence no.: 2ba4d7dd-877c-4982-b41e-817c99633fc4 | 18867399223023088896
message id and sequence no.: ad354de3-3a89-4400-83b8-89a892c30526 | 18867399223104239872
message id and sequence no.: 2ba4d7dd-877c-4982-b41e-817c99633fc4 | 18867399223023088896
message id and sequence no.: 9529ddac-8946-4fee-a2dc-7be428666b63 | 18867399222923248640

The message identifier is returned along with the sequence number when SQS accepts the message. We can see that the Sequence number assigned to each message by Amazon SQS is a large and non-consecutive number.

Five messages are being sent to the queue myfifoqueue.fifo, two of which are duplicates. The messageDeduplicationId property in the message is calculated by SQS when duplicate messages are detected because the contentBasedDeduplication property has been set to false.

In My fifo message1 and My fifo message2, the messageDeduplicationId is the same while in My fifo message3, the messageDeduplicationId is different.

After the messages are consumed, we will only receive three unique messages in the same order as when we sent the messages to the queue.

Recommended

• AWS SQS APIs: Create Queues, Send & Receive Messages Simplified 101
• Beginners Guide to Message Queues: Benefits, 2 Types & Use Cases
• How to Build a Message Queue using Python? | Made Easy
• Message Brokers: Key Models, Use Cases & Tools Simplified 101

Conclusion

The Simple Queue Service (SQS) offered by Amazon enables decoupling and communication among distributed system components. The purpose of this article was to discuss Amazon SQS, Message Queuing as a form of asynchronous communication between two or more processes, and how messages and queues are the core components of a message queuing system. Additionally, we examined the queue, the sending and receiving of messages using SQS ReceiveMessage, and worked through some examples.

Most companies need to analyze their business data stored in multiple data sources. The data needs to be loaded to the Data Warehouse to get a holistic view of the data. While building in-house ETL pipelines requires a ton of engineering bandwidth and huge maintenance costs in the long run, it’s far better to employ the use of fast & reliable third-party ETL solutions like Hevo Data.

Hevo Data is a No-code Data Pipeline Platform that helps to transfer data from 150+ Data Sources to desired Data Warehouse. It fully automates the process of transforming and transferring data to a destination without writing a single line of code.

FAQ on AWS SQS ReceiveMessage

Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand.

Share your experience of learning about AWS SQS ReceiveMessage in the comments section below!