Ultimate Guide to Python SQS: 13+ Operations With Easy Examples

Amazon Web Services (AWS) has become the world’s most comprehensive and broadly adopted platform for enterprises and software developers. The power of the AWS Cloud brings serverless architectures that are scalable, highly available, and fully managed. Python is a popular and versatile high-level programming language that is used in a range of applications including Data Analysis and Machine Learning, Software and Web Development, Automation or Scripting, etc. 

Whether you are building a brand new application or hosting services on AWS, using Python on AWS simplifies the use of AWS Services. This combination provides you with a set of libraries that are consistent and familiar to Python developers. Boto3 is the AWS SDK for Python, which allows Python developers to write software on AWS Cloud and make use of Amazon Services like S3 and EC2. Boto provides an easy-to-use, object-oriented API as well as low-level direct service access.

This ultimate guide will teach you how to make use of Python scripts to interact with Simple Queue Service (SQS) provided by Amazon Web Services (AWS). You’ll learn to create Python SQS Queues using the Boto3 library, set up Queue permissions, attributes, and tags, configure Queue URLs, and list all your AWS SQS Python Queues. Later, you’ll learn how to programmatically send, receive and remove messages from Python SQS Queues. 

​​Take a closer look at how helpful Python SQS is, what it can do, and how you can use it to perform a variety of operations.

Table of Contents

• What Is Amazon SQS?
  • Key Benefits of Amazon SQS
• What Is Python?
• What Is Boto3 SDK for AWS & Python?
• Prerequisites for Python SQS Guide
• How to Perform SQS Actions Using Python & Boto3?
• How to Process SQS Queue Messages?
• Bonus: How to Run Scheduled Messages Using SQS & Python?
• Challenges of Using AWS SQS Queues
• Conclusion

What Is Amazon SQS?

Connected web applications and program instances require an interface to communicate with each other. This interface helps them pass data or logic which is useful for executing another set of tasks. The interface must offer reliable message delivery so that messages are delivered and processed on time, and at the same time allow decoupling, so that the sender application doesn’t need to wait for the consumer to process the message and pass an acknowledgment.

Amazon SQS or Simple Queue Service is a Distributed Message Broker Service from Amazon which helps establish reliable, secure, and decoupled communication between web applications, services, and program instances. Decoupling allows multiple application components to run independently, which eases message processing and management. 

One of the best features of SQS is that it lets you transmit any volume of data, at any level of throughput. SQS always ensures that your message is delivered at least once, and also allows multiple consumers and senders to communicate using the same Message Queue. It offers support for both Standard and FIFO Queues. 

• Standard Queues give the highest throughput, best-effort ordering, and at least one delivery.
• FIFO Queues or First-In, First-Out Queues ensure that messages are processed only once, in the sequence in which they are sent.

To gain in-depth information on AWS SQS Message Queue Service and its working, do check out What is AWS SQS (Simple Queue Service)?: 5 Comprehensive Aspects. You can know more about Message Queue here – Beginners Guide to Message Queues: Benefits, 2 Types & Use Cases.

Key Benefits of Amazon SQS

• High Durability: Amazon SQS offers extremely high message durability by storing your messages on multiple servers so that you don’t lose them. 
• Unmatched Security: With Amazon SQS, you gain full control over who can send and receive your messages. Amazon SQS provides a lot of security features like Server-Side Encryption (SSE), encrypted connections over HTTPS (TLS), IAM roles, and multiple user account access levels. 
• Built for Ultimate Scale and Low Latency: Without any provisioning instructions, Amazon SQS can process each buffered request individually, and scale transparently to manage any demand surges or spikes.
• Unlimited Queues and Messages: When you use Amazon SQS, there are no restraints on the number of Queues that can be used or messages that can be sent and received by your application components.
• Multiple Writers and Readers: Amazon SQS allows you to send and receive messages from various parts of your system (or threads of the same process) at the same time.
• Customization: SQS lets you customize your Message Queues by setting Delay Queues that let you postpone the delivery of new messages to consumers for some seconds.

What Is Python?

Python is an object-oriented, high-level programming language that is currently used by close to 80% of developers worldwide. It’s a powerful, easy to learn & use language with elegant syntax and dynamic typing. Python stands out because it uses English terms in its syntax and uses fewer syntactical structures than other languages.

Equipped with a combination of interactive features and easy integration with C, C++, COM, ActiveX, CORBA, and Java, Python is an ideal language for scripting and rapid application development in many areas on most platforms. Python comes with a large collection of standard modules that you can use as the basis of your programs, and the coolest part about these is that they are portable and work cross-platform. 

More:

• Understanding Data Modelling in Python: 4 Critical Aspects
• AWS Lambda Python Integration: 2022’s Complete Guide to Get Started
• How to Build a Message Queue using Python? | Made Easy

What Is Boto3 SDK for AWS & Python?

Boto3 is an AWS Software Development Kit (SDK) for Python. It allows Python developers to write programs in the cloud and make use of AWS Services like Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). 

Boto3 is the latest version of AWS SDK offering support for Python. It’s based on Botocore, a module that offers low-level functionality that the Python SDK and the AWS CLI share. Boto3 is an object-oriented API that offers low-level access to Amazon Web Services and is published and maintained by Amazon Web Services.

To install Boto3, you may use the following command:

pip install boto3

If your project requires a specific version of Boto3 or has compatibility concerns, you may choose to specify the same using the following constraints when installing:

# Install Boto3 version 1.0 specifically
pip install boto3==1.0.0

# Make sure Boto3 is no older than version 1.15.0
pip install boto3>=1.15.0

# Avoid versions of Boto3 newer than version 1.15.3
pip install boto3<=1.15.3

Prerequisites for Python SQS Guide

The following are the prerequisites required in this Python SQS guide:

• Python 
• Boto3 SDK
• AWS Credentials

How to Perform SQS Actions Using Python & Boto3?

In the upcoming sections, we discuss how to use the Boto3 library to perform various operations on AWS SQS. 

• Create a New Python SQS Queue: Standard
• Create a New Python SQS Queue: FIFO
• How to Fetch the SQS Queue URL?
• How to Set Up SQS Python Queue Attributes?
• How to Set Up AWS SQS Python Tags Attributes?
• Delete AWS SQS Python Queue
• List AWS SQS Python Queues
• Set Up Permissions for Python AWS SQS Queues
• Remove Permissions for Python AWS SQS Queues

Create a New Python SQS Queue: Standard

AWS Standard SQS Queues are the default type of Message Queues. They provide an unlimited number of transactions per second, high throughput, and at least one message delivery. In AWS Standard Queues, messages are delivered in the same order as they are sent.

To create a new Python SQS Standard Queue, you can use the create_queue() command. You do this by first importing the required module and then instantiating the SQS client.

After that, you can create SQS Queue using the following function definition.

And finally, you specify your Python SQS Standard Queue with SQS Queue attributes as follows:

Note: A Queue_Name (type: string) points out the name of the SQS Queue and it can have up to 80 characters containing alphanumeric characters, hyphens (-), and underscores (_).

Once you execute the following set of commands, you’ll be displayed with an output something like this:

2021-05-01 17:17:17, 735: INFO: Standard Queue hevo-data-standard-queue created. Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-standard-queue

Create a New Python SQS Queue: FIFO

SQS FIFO Queues ensure that the messages that come first are processed first; i.e., in the order in which they arrive. FIFO Queues ensure that a message is sent precisely once and stays available until it is processed and deleted by a consumer; duplicate messages are not added to the Message Queue. They can handle up to 3000 messages per second. 

To create a new Python SQS FIFO Queue, you can use the create_queue() command as defined in the previous section. All you have to do is change the previous definition to something like this:

Again, after code execution, you’ll receive an output something like this:

2021-05-01 17:17:43, 876: INFO: FIFO Queue hevo-data-fifo-queue.fifo created. Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-fifo-queue.fifo

In both cases, you have to provide the attribute values for the Queue. These include:

• DelaySeconds: Time value by which messages are delayed.
• RedrivePolicy: Dead-letter Queue functionality is specified through this.
• VisibilityTimeout: The Queue’s visibility timeout is measured in seconds. This is the amount of time that a single customer can see a specific message. 
• MaximumMessageSize: Specifies the maximum message size limit.
• FifoQueue: A parameter that defines a Python AWS SQS Queue as FIFO. It accepts two values: true or false. 

More information about SQS Queue attributes can be found here AWS SQS ReceiveMessage: Syntax, Importance, Parameters & Examples.

How to Fetch the SQS Queue URL?

For obtaining SQS Queue URL, you can use the get_queue_url() method. If you would like to access a Queue that belongs to another AWS account, you need to get permission from the respective Queue owner.

Here’s the request syntax to fetch the SQS Queue URL:

response = client.get_queue_url(
    QueueName='string',
    QueueOwnerAWSAccountId='string'
)

Parameters:

• QueueName: Name of the Queue whose URL has to be obtained.
• QueueOwnerAWSAccountId: AWS account ID that created the Queue.

Output:

https://us-east-1.queue.amazonaws.com/xxxx/hevo-data-new-queue

How to Set Up SQS Python Queue Attributes?

You can set or update Python SQS Queue attributes using the set_queue_attributes() method. The set_queue_attributes() method sets the value of one or more Queue attributes, and it might take up to 60 seconds for most Queue properties to propagate throughout the Amazon SQS system once you modify them.

Here’s you can use the set_queue_attributes() method:

The set_queue_attributes() method utilizes the following names, descriptions, and values for specific request parameters:

• DelaySeconds: The period of delay from the time a message is queued to the time it is delivered.
• MaximumMessageSize: The maximum size (in bytes) an SQS message can be before it gets rejected from SQS Queue.
• ReceiveMessageWaitTimeSeconds: Message receivers wait for a message to arrive for a specific amount of time. In this value, 0 is the default value and any value between 0 and 20 seconds can be used.
• VisibilityTimeout: Visibility timeout prevents other consumers from viewing messages received from a Queue by a single consumer. In the visibility timeout period, Amazon SQS does not allow other consumers to receive and process the message.

Get more information about available Python AWS SQS Queue attributes from the official documentation page here- CreateQueue.

Once executed, you’ll receive an output like this:

2021-05-01 17:30:17, 567: INFO: Queue https://queue.amazonaws.com/967745397581/hevo-data-standard-queue attributes created.

How to Set Up AWS SQS Python Tags Attributes?

AWS tags are used to organize and identify your Amazon SQS Queues for cost allocation. To use SQS Queue tags, you need to use the tag_queue() method. 

While using tags, please be mindful that adding more than 50 tags to an SQS Queue isn’t recommended. 

Once executed, you’ll receive an output like this:

2021-05-01 17:31:43, 867: INFO: Resource tags applied to the queue -  https://queue.amazonaws.com/967745397581/hevo-data-standard-queue.

Delete AWS SQS Python Queue 

Using the delete_queue() method, you can delete any AWS SQS Queue regardless of the Queue’s contents. Please note that when you delete a Queue, it might take up to 60 seconds to take effect. 

Here’s what the output would look like:

2021-05-01 17:34:16,149: INFO: https://queue.amazonaws.com/967745397581/hevo-data-standard-queue deleted successfully.

List AWS SQS Python Queues

You can use the list_queues() method to get a list of all of your Python SQS Queues. This method produces a list of your SQS Queues in the current region with a maximum of 1000 results. If the optional QueueNamePrefix argument is set, only Queues with names that begin with the provided value are returned.

Here’s a simple example to explain how you can use the list_queues() method.

import boto3

# Create SQS client
sqs = boto3.client('sqs')

# List SQS queues
response = sqs.list_queues()

print(response['QueueUrls'])

Once executed, you’ll receive an output like this:

2021-05-01 17:35:29, 673: INFO: Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-fifo-queue.fifo
2021-05-01 17:35:29, 673: INFO: Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-standard-queue

Set Up Permissions for Python AWS SQS Queues

With the help of the add_permission() method, you can add permission to a Queue for a specific principal. As an owner of the SQS Queue, you gain full control access rights for the Queue where you have complete freedom to grant or deny Queue access permissions to other AWS users. 

Here’s how you use the add permission() method to set up permissions for Python SQS Queues:

And here’s what the execution output would look like:

2021-05-01 17:36:22, 155: INFO: Permissions added to the queue with the label HevoDataSendMessage.

Remove Permissions for Python AWS SQS Queues

AWS SQS has provisions for both adding and removing permissions from SQS Queues. To remove permissions, you can use the remove_permission() method which revokes any permissions in the Queue policy that matches the specified Label parameter. Do keep in mind that only the SQS Queue owner has the right to remove permissions.

Once executed, here’s what the output would look like:

2021-05-01 17:36:52, 695: INFO: Permissions HevoDataSendMessage removed from the queue.

How to Process Python SQS Queue Messages?

Boto3 library gives you the capability to send, read and delete messages from a Python SQS Queue. The following sections explain how to use Boto3 library methods to process SQS Queue messages.

• Send Python SQS Messages
• Receive Python SQS Messages
• Delete Python SQS Messages
• Remove All SQS Queue Messages

Send Python SQS Messages 

To send a message, you can make use of the send_message() Boto3 method. The send_message() SQS Python method delivers a message to your specified Queue by adding it to the end of the Queue. 

Please Note: Your messages can only contain XML, JSON, and unformatted text, as well as the following Unicode characters. The Python AWS SQS Queue will reject your message if it contains any other characters.

#x9 | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000 to #x10FFFF

Here’s the code to send messages in Python SQS:

Parameters:

• QueueUrl: The URL of the Amazon SQS Queue.
• MessageBody: The message that you wish to send. The minimum allowed size for a message is one character and the maximum is 256 KB.
• MessageAttributes: Message attributes are optional and separate from the message body. Each message attribute consists of a Name, Type, and Value, and they tell a consumer how to handle a message. Amazon SQS allows up to 10 message attributes per message.
  • StringValue (string): Unicode strings with UTF-8 binary encoding.
  • BinaryValue (bytes): Any binary data, such as compressed data, encrypted data, or pictures, can be stored in binary type attributes.
  • StringListValues (list): Not implemented. Reserved for future use.
  • BinaryListValues (list): Not implemented. Reserved for future use.
  • DataType (string): AWS SQS Python supports the following logical data types: String, Number, and Binary.
• MessageSystemAttributes: Message system attributes are the user-specified message system attribute values. At present, the only supported message system attribute is AWSTraceHeader.
  • StringValue (string): Unicode strings with UTF-8 binary encoding.
  • BinaryValue (bytes): Any binary data, such as compressed data, encrypted data, or pictures, can be stored in binary type attributes.
  • StringListValues (list): Not implemented. Reserved for future use.
  • BinaryListValues (list): Not implemented. Reserved for future use.
  • DataType (string): AWS SQS Python supports the following logical data types: String, Number, and Binary.
• MessageDeduplicationId: This option applies to FIFO SQS Python Queues. It prevents duplication of messages by allocating a unique MessageDeduplicationId to each message. Any other messages with the same MessageDeduplicationId are accepted but not delivered during the 5-minute deduplication interval. The maximum length of MessageDeduplicationId is 128 characters.

When executed successfully, the output would look like this:

{'MD5OfMessageBody': '88bac95f31528d13a072c05f2a1cf371', 'MessageId': '2ce1541b-0472-4715-8375-f8a8587c16e9', 'ResponseMetadata': {'RequestId': '02a7b659-c044-5357-885e-ee0c398e24b0', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '02a7b659-c044-5357-885e-ee0c398e24b0', 'date': 'Fri, 18 Dec 2020 00:27:54 GMT', 'content-type': 'text/xml', 'content-length': '378'}, 'RetryAttempts': 0}}

Receive Python SQS Messages

To receive a message from an AWS SQS Python Queue, you can use receive_message() Boto3 method. The receive_message() SQS Python method retrieves one or more messages (up to 10), from your specified SQS Queue.

Here’s the code to receive messages in Python SQS:

Parameters:

• QueueUrl: The URL of the Amazon SQS Queue to which you wish to send a message.
• AttributeNames: A list of attributes that are to be returned along with each message.
• MaxNumberOfMessages: The maximum number of messages to retrieve.
• WaitTimeSeconds: Waiting time for a message to arrive in the Queue. This option is useful for the long-polling of messages.
• MessageAttributeNames: The name of the message attribute.
• VisibilityTimeout: The Queue’s visibility timeout is measured in seconds. This is the amount of time that a single customer can see a specific message. 
• ReceiveRequestAttemptId: Applied to FIFO SQS Queues for deduplication of receive_message() calls. 

When executed successfully, the output would look like this:

Number of messages received: 1
Message body: {'key': 'value'}

Delete Python SQS Messages

To delete a message from an AWS SQS Python Queue, you can use the delete_message() Boto3 method. The delete_message() SQS Python method deletes up to ten messages from the specified Queue.

Here’s the code to delete messages in Python SQS:

When executed successfully, the output would look like this:

{'ResponseMetadata': {'RequestId': 'd9a860cb-45ff-58ec-8232-389eb8d7c2c6', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'd9a860cb-45ff-58ec-8232-389eb8d7c2c6', 'date': 'Fri, 18 Dec 2020 00:42:16 GMT', 'content-type': 'text/xml', 'content-length': '215'}, 'RetryAttempts': 0}}

Remove All SQS Queue Messages

If you would like to remove all messages from your AWS SQS Python Queue, you can use the purge_queue() Boto3 method. The purge_queue() Boto3 method then deletes all the messages in a Queue specified by the QueueURL parameter.

Here’s the code to remove all messages:

It might take up to 60 seconds for the message deletion procedure to take effect. Before you run purge_queue(), all messages delivered to the Queue will be received but erased within the following minute. While the queue is being purged, all messages delivered to it after you use purge_queue() may be removed.

Bonus: How to Run Scheduled Messages Using SQS & Python?

Newscatcher is an API that scrapes the internet and combines over 1,000,000 news articles every day. Using the SQS Queues and AWS Lambda architecture, they are able to acquire and send referential data in CSV file formats and get updates on articles. This process involves importing a CSV file into a DynamoDB table using the Boto3 Python package. One method that you can use to do this is by using import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types) Python function as follows:

import boto

MY_ACCESS_KEY_ID = 'copy your access key ID here'
MY_SECRET_ACCESS_KEY = 'copy your secret access key here'


def do_batch_write(items, table_name, dynamodb_table, dynamodb_conn):
    '''
    From https://gist.github.com/griggheo/2698152#file-gistfile1-py-L31
    '''
    batch_list = dynamodb_conn.new_batch_write_list()
    batch_list.add_batch(dynamodb_table, puts=items)
    while True:
        response = dynamodb_conn.batch_write_item(batch_list)
        unprocessed = response.get('UnprocessedItems', None)
        if not unprocessed:
            break
        batch_list = dynamodb_conn.new_batch_write_list()
        unprocessed_list = unprocessed[table_name]
        items = []
        for u in unprocessed_list:
            item_attr = u['PutRequest']['Item']
            item = dynamodb_table.new_item(
                    attrs=item_attr
            )
            items.append(item)
        batch_list.add_batch(dynamodb_table, puts=items)


def import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types):
    '''
    Import a CSV file to a DynamoDB table
    '''        
    dynamodb_conn = boto.connect_dynamodb(aws_access_key_id=MY_ACCESS_KEY_ID, aws_secret_access_key=MY_SECRET_ACCESS_KEY)
    dynamodb_table = dynamodb_conn.get_table(table_name)     
    BATCH_COUNT = 2 # 25 is the maximum batch size for Amazon DynamoDB
    
    items = []
    
    count = 0
    csv_file = open(csv_file_name, 'r')
    for cur_line in csv_file:
        count += 1
        cur_line = cur_line.strip().split(',')
        
        row = {}
        for colunm_number, colunm_name in enumerate(colunm_names):
            row[colunm_name] = column_types[colunm_number](cur_line[colunm_number])
         
        item = dynamodb_table.new_item(
                    attrs=row
            )           
        items.append(item)
        
        if count % BATCH_COUNT == 0:
            print 'batch write start ... ', 
            do_batch_write(items, table_name, dynamodb_table, dynamodb_conn)
            items = []
            print 'batch done! (row number: ' + str(count) + ')'
    
    # flush remaining items, if any
    if len(items) > 0: 
        do_batch_write(items, table_name, dynamodb_table, dynamodb_conn)

        
    csv_file.close() 


def main():
    '''
    Demonstration of the use of import_csv_to_dynamodb()
    We assume the existence of a table named `test_persons`, with
    - Last_name as primary hash key (type: string)
    - First_name as primary range key (type: string)
    '''
    colunm_names = 'Last_name First_name'.split()
    table_name = 'test_persons'
    csv_file_name = 'test.csv'
    column_types = [str, str]
    import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types)
    

if __name__ == "__main__":
    main()
    #cProfile.run('main()') # if you want to do some profiling

Code Credits: RIPTutorial

Challenges of Using AWS SQS Queues

While AWS Simple Queue Service comes with its own set of advantages like unlimited Python SQS Queues and messages, high durability, unmatched security, and pay-for-what-you-use pricing, it’s not to say that it’s devoid of any performance challenges. Consider the following aspects that AWS SQS Queues face:

• Latency Problem: Once you’ve added a message to the Queue. It takes around one minute for the message to become available for reading. Also, the order in which a message is received after being queued is not guaranteed.
• High Scalability Costs: With pay-per-use pricing, your SQS cost might quickly grow if you send many messages. SQS price includes data transport charges, which can mount up if you send big messages or process messages from outside the primary AWS area where the Queue is situated.

Conclusion

This guide discussed AWS Simple Queue Service (SQS) which enables decoupling and communication among distributed system components. Using the Boto3 Python Package, you can perform a range of operations like creating a new Python SQS Queue, fetching Queue URLs, and setting Queue attributes, tags, and permissions. Additionally, we examined different ways to process AWS SQS Python Queues like how to send, receive, and delete certain messages from SQS Python Queue and how to completely remove all SQS Queue messages. 

Amazon Web Services (AWS) provides services and infrastructure to build reliable, fault-tolerant, and highly available systems in the cloud. Just like AWS, Hevo Data, a No-Code Automation and Data Pipeline Creation Tool helps you to build your own fault-tolerant, reliable, and zero-data loss Data Pipelines in the cloud. 

Hevo Data is a comprehensive ETL platform that allows you to migrate data from 100+ Data Sources like Amazon S3, Amazon Relational Database Services like Amazon RDS on PostgreSQL, Amazon RDS on MySQL, Oracle on Amazon RDS, MySQL on Amazon Aurora, and many more. Our connector inventory now includes 40+ Free Data Sources from which you may obtain and move data without incurring any costs.

The best part about Hevo is that setting up Data Pipelines is a cakewalk; select your source, provide credentials and choose your target destination. And you are done!

Visit our Website to Explore Hevo

Hevo can connect your frequently used applications to Data Warehouses like Amazon Redshift, Snowflake, Google BigQuery, Firebolt, or even Database Destinations like PostgreSQL, MySQL, or MS SQL Server in a matter of minutes. Using Hevo requires little to no training, and you will be able to set up your Data Pipelines without any help from your engineering teams. 

Why not try Hevo and see the magic for yourself? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also check our unbeatable pricing and make a decision on your best-suited plan. 

Have any questions on Python SQS Queues? Do let us know in the comment section below. Also, share any other AWS Services or features you’d want us to cover. We’d be happy to know your opinions.