Ultimate Guide to Python SQS: 13+ Operations With Easy Examples

Whether you are building a brand new application or hosting services on AWS, using Python on AWS simplifies the use of AWS Services. This combination provides you with a set of libraries that are consistent and familiar to Python developers. Boto3 is the AWS SDK for Python, which allows Python developers to write software on AWS Cloud and make use of Amazon Services like S3 and EC2. Boto provides an easy-to-use, object-oriented API as well as low-level direct service access.

This ultimate guide will teach you how to make use of Python scripts to interact with Simple Queue Service (SQS) provided by Amazon Web Services (AWS). You’ll learn to create Python SQS Queues using the Boto3 library, set up Queue permissions, attributes, and tags, configure Queue URLs, and list all your AWS SQS Python Queues. Later, you’ll learn how to programmatically send, receive and remove messages from Python SQS Queues. 

​​Take a closer look at how helpful Python SQS is, what it can do, and how you can use it to perform a variety of operations.

What Is Amazon SQS?

Amazon SQS or Simple Queue Service is a Distributed Message Broker Service from Amazon that helps establish reliable, secure, and decoupled communication between web applications, services, and program instances. Decoupling allows multiple application components to run independently, which eases message processing and management.

One of the best features of SQS is that it lets you transmit any volume of data, at any level of throughput. SQS always ensures that your message is delivered at least once, and also allows multiple consumers and senders to communicate using the same Message Queue. It offers support for both Standard and FIFO Queues. 

• Standard Queues give the highest throughput, best-effort ordering, and at least one delivery.
• FIFO Queues or First-In, First-Out Queues ensure that messages are processed only once, in the sequence in which they are sent.

To gain in-depth information on AWS SQS Message Queue Service and how it works, check out What is AWS SQS (Simple Queue Service)?: 5 Comprehensive Aspects. You can learn more about Message Queue with Beginners Guide to Message Queues: Benefits, 2 Types & Use Cases.

What Is Python?

Python is an object-oriented, high-level programming language that is currently used by close to 80% of developers worldwide. Python’s dynamic capabilities make building a message queue straightforward and efficient, enabling applications to handle large volumes of data reliably and in real time.

Equipped with a combination of interactive features and easy integration with C, C++, COM, ActiveX, CORBA, and Java, Python is an ideal language for scripting and rapid application development in many areas on most platforms. Python’s versatility makes it ideal for data modeling, offering four critical aspects to streamline the structuring, querying, and manipulation of data in applications.

What Is Boto3 SDK for AWS & Python?

Boto3 is an AWS Software Development Kit (SDK) for Python. It allows Python developers to write programs in the cloud and make use of AWS Services like Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). 

Boto3 is the latest version of AWS SDK offering support for Python. It’s based on Botocore, a module that offers low-level functionality that the Python SDK and the AWS CLI share. Boto3 is an object-oriented API that offers low-level access to Amazon Web Services and is published and maintained by Amazon Web Services.

To install Boto3, you may use the following command:

pip install boto3

If your project requires a specific version of Boto3 or has compatibility concerns, you may choose to specify the same using the following constraints when installing:

# Install Boto3 version 1.0 specifically
pip install boto3==1.0.0

# Make sure Boto3 is no older than version 1.15.0
pip install boto3>=1.15.0

# Avoid versions of Boto3 newer than version 1.15.3
pip install boto3<=1.15.3

Prerequisites for Python SQS Guide

The following are the prerequisites required in this Python SQS guide:

• Python 
• Boto3 SDK
• AWS Credentials

How to Perform SQS Actions Using Python & Boto3?

In the upcoming sections, we discuss how to use the Boto3 library to perform various operations on AWS SQS. 

• Create a New Python SQS Queue: Standard
• Create a New Python SQS Queue: FIFO
• How to Fetch the SQS Queue URL?
• How to Set Up SQS Python Queue Attributes?
• How to Set Up AWS SQS Python Tags Attributes?
• Delete AWS SQS Python Queue
• List AWS SQS Python Queues
• Set Up Permissions for Python AWS SQS Queues
• Remove Permissions for Python AWS SQS Queues

Create a New Python SQS Queue: Standard

AWS Standard SQS Queues are the default type of Message Queues. They provide an unlimited number of transactions per second, high throughput, and at least one message delivery. In AWS Standard Queues, messages are delivered in the same order as they are sent.

To create a new Python SQS Standard Queue, you can use the create_queue() command. You do this by first importing the required module and then instantiating the SQS client.

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_resource = boto3.resource("sqs", region_name=AWS_REGION)

def create_queue(queue_name, delay_seconds, visiblity_timeout):
    """
    Create a standard SQS queue
    """
    try:
        response = sqs_resource.create_queue(QueueName=queue_name,
                                             Attributes={
                                                 'DelaySeconds': delay_seconds,
                                                 'VisibilityTimeout': visiblity_timeout
                                             })
    except ClientError:
        logger.exception(f'Could not create SQS queue - {queue_name}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_NAME = 'hevo-data-standard-queue'
    DELAY_SECONDS = '0'
    VISIBLITY_TIMEOUT = '60'
    output = create_queue(QUEUE_NAME, DELAY_SECONDS, VISIBLITY_TIMEOUT)
    logger.info(
        f'Standard Queue {QUEUE_NAME} created. Queue URL - {output.url}')

Note: A Queue_Name (type: string) points out the name of the SQS Queue and it can have up to 80 characters containing alphanumeric characters, hyphens (-), and underscores (_).

Once you execute the following set of commands, you’ll be displayed with an output something like this:

2021-05-01 17:17:17, 735: INFO: Standard Queue hevo-data-standard-queue created. Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-standard-queue

Create a New Python SQS Queue: FIFO

SQS FIFO Queues ensure that the messages that come first are processed first; i.e., in the order in which they arrive. FIFO Queues ensure that a message is sent precisely once and stays available until it is processed and deleted by a consumer; duplicate messages are not added to the Message Queue. They can handle up to 3000 messages per second. 

To create a new Python SQS FIFO Queue, you can use the create_queue() command as defined in the previous section. All you have to do is change the previous definition to something like this:

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_resource = boto3.resource("sqs", region_name=AWS_REGION)

def create_queue(queue_name, delay_seconds, visiblity_timeout):
    """
    Create a First In First Out (FIFO) SQS queue
    """
    try:
        response = sqs_resource.create_queue(QueueName=queue_name,
                                             Attributes={
                                                 'DelaySeconds': delay_seconds,
                                                 'VisibilityTimeout': visiblity_timeout,
                                                 'FifoQueue': 'true'
                                             })
    except ClientError:
        logger.exception(f'Could not create SQS queue - {queue_name}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_NAME = 'hevo-data-fifo-queue.fifo'
    DELAY_SECONDS = '0'
    VISIBLITY_TIMEOUT = '60'
    output = create_queue(QUEUE_NAME, DELAY_SECONDS, VISIBLITY_TIMEOUT)
    logger.info(f'FIFO Queue {QUEUE_NAME} created. Queue URL - {output.url}')

Again, after code execution, you’ll receive an output something like this:

2021-05-01 17:17:43, 876: INFO: FIFO Queue hevo-data-fifo-queue.fifo created. Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-fifo-queue.fifo

In both cases, you have to provide the attribute values for the Queue. These include:

• DelaySeconds: Time value by which messages are delayed.
• RedrivePolicy: Dead-letter Queue functionality is specified through this.
• VisibilityTimeout: The Queue’s visibility timeout is measured in seconds. This is the amount of time that a single customer can see a specific message. 
• MaximumMessageSize: Specifies the maximum message size limit.
• FifoQueue: A parameter that defines a Python AWS SQS Queue as FIFO. It accepts two values: true or false. 

More information about SQS Queue attributes can be found in AWS SQS ReceiveMessage: Syntax, Importance, Parameters & Examples.

How to Fetch the SQS Queue URL?

For obtaining SQS Queue URL, you can use the get_queue_url() method. If you would like to access a Queue that belongs to another AWS account, you need to get permission from the respective Queue owner.

Here’s the request syntax to fetch the SQS Queue URL:

response = client.get_queue_url(
    QueueName='string',
    QueueOwnerAWSAccountId='string'
)

Parameters:

• QueueName: Name of the Queue whose URL has to be obtained.
• QueueOwnerAWSAccountId: AWS account ID that created the Queue.

Output:

https://us-east-1.queue.amazonaws.com/xxxx/hevo-data-new-queue

How to Set Up SQS Python Queue Attributes?

You can set or update Python SQS Queue attributes using the set_queue_attributes() method. The set_queue_attributes() method sets the value of one or more Queue attributes, and it might take up to 60 seconds for most Queue properties to propagate throughout the Amazon SQS system once you modify them.

Here’s you can use the set_queue_attributes() method:

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def configure_queue_attributes(queue_url, delay_seconds, max_msg_size):
    """
    Configure queue attributes.
    """
    try:
        response = sqs_client.set_queue_attributes(QueueUrl=queue_url,
                                                   Attributes={
                                                       'DelaySeconds':
                                                       delay_seconds,
                                                       'MaximumMessageSize':
                                                       max_msg_size
                                                   })
    except ClientError:
        logger.exception(f'Could not set attributes on - {queue_url}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_URL = '<your-queue-url>'
    DELAY_SECONDS = '15'
    MAX_MSG_SZIE = '2048'
    queue = configure_queue_attributes(QUEUE_URL, DELAY_SECONDS, MAX_MSG_SZIE)
    logger.info(f'Queue {QUEUE_URL} attributes created.'

Once executed, you’ll receive an output like this:

2021-05-01 17:30:17, 567: INFO: Queue https://queue.amazonaws.com/967745397581/hevo-data-standard-queue attributes created.

How to Set Up AWS SQS Python Tags Attributes?

AWS tags are used to organize and identify your Amazon SQS Queues for cost allocation. To use SQS Queue tags, you need to use the tag_queue() method. 

Note: While using tags, please be mindful that adding more than 50 tags to an SQS Queue isn’t recommended. 

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def apply_queue_tags(queue_url, tags):
    """
    Add resource tags to the specified Amazon SQS queue.
    """
    try:
        response = sqs_client.tag_queue(QueueUrl=queue_url, Tags=tags)
    except ClientError:
        logger.exception(f'Could not set tags on - {queue_url}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_URL = '<your-queue-url>'
    TAGS = {
        'Name': 'hands-on-cloud-standard-queue',
        'Team': 'hands-on-cloud',
        'Type': 'standard'
    }
    queue = apply_queue_tags(QUEUE_URL, TAGS)
    logger.info(f'Resource tags applied to the queue - {QUEUE_URL}.')

Once executed, you’ll receive an output like this:

2021-05-01 17:31:43, 867: INFO: Resource tags applied to the queue -  https://queue.amazonaws.com/967745397581/hevo-data-standard-queue.

Delete AWS SQS Python Queue 

Using the delete_queue() method, you can delete any AWS SQS Queue regardless of the Queue’s contents. Please note that when you delete a Queue, it might take up to 60 seconds to take effect. 

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def delete_queue(queue_name):
    """
    Deletes the queue specified by the QueueUrl.
    """
    try:
        response = sqs_client.delete_queue(QueueUrl=queue_name)
    except ClientError:
        logger.exception(f'Could not delete the {queue_name} queue.')
        raise
    else:
        return response

if __name__ == '__main__':
    # Constants
    QUEUE_URL = '<your-queue-url>'
    queue = delete_queue(QUEUE_URL)
    logger.info(f'{QUEUE_URL} deleted successfully.')

Here’s what the output would look like:

2021-05-01 17:34:16,149: INFO: https://queue.amazonaws.com/967745397581/hevo-data-standard-queue deleted successfully.

List AWS SQS Python Queues

You can use the list_queues() method to get a list of all of your Python SQS Queues. This method produces a list of your SQS Queues in the current region with a maximum of 1000 results. If the optional QueueNamePrefix argument is set, only Queues with names that begin with the provided value are returned.

Here’s a simple example to explain how you can use the list_queues() method.

import boto3

# Create SQS client
sqs = boto3.client('sqs')

# List SQS queues
response = sqs.list_queues()

print(response['QueueUrls'])

Once executed, you’ll receive an output like this:

2021-05-01 17:35:29, 673: INFO: Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-fifo-queue.fifo
2021-05-01 17:35:29, 673: INFO: Queue URL - https://queue.amazonaws.com/967745397581/hevo-data-standard-queue

Set Up Permissions for Python AWS SQS Queues

With the help of the add_permission() method, you can add permission to a Queue for a specific principal. As an owner of the SQS Queue, you gain full control access rights for the Queue where you have complete freedom to grant or deny Queue access permissions to other AWS users. 

Here’s how you use the add permission() method to set up permissions for Python SQS Queues:

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def add_access_permissions(queue_url, label, account_ids, actions):
    """
    Adds permission to a queue for a specific principal.
    """
    try:
        response = sqs_client.add_permission(QueueUrl=queue_url,
                                             Label=label,
                                             AWSAccountIds=account_ids,
                                             Actions=actions)
    except ClientError:
        logger.exception(f'Could not add permissions for - {queue_url}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_URL = '<your-queue-url>'
    LABEL = 'HandOnCloudSendMessage'
    ACCOUNT_IDS = ['979450158315']
    ACTIONS = ['SendMessage', 'DeleteMessage']
    permissions = add_access_permissions(QUEUE_URL, LABEL, ACCOUNT_IDS,
                                         ACTIONS)
    logger.info(f'Permissions added to the queue with the label {LABEL}.')

And here’s what the execution output would look like:

2021-05-01 17:36:22, 155: INFO: Permissions added to the queue with the label HevoDataSendMessage.

Remove Permissions for Python AWS SQS Queues

AWS SQS has provisions for both adding and removing permissions from SQS Queues. To remove permissions, you can use the remove_permission() method which revokes any permissions in the Queue policy that matches the specified Label parameter. Do keep in mind that only the SQS Queue owner has the right to remove permissions.

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def remove_access_permissions(queue_url, label):
    """
    Revokes any permissions in the queue policy.
    """
    try:
        response = sqs_client.remove_permission(QueueUrl=queue_url,
                                                Label=label)
    except ClientError:
        logger.exception(f'Could not remove permissions for - {queue_url}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_URL = '<your-queue-url>'
    LABEL = 'HandOnCloudSendMessage'
    permissions = remove_access_permissions(QUEUE_URL, LABEL)
    logger.info(f'Permissions {LABEL} removed from the queue.')

Once executed, here’s what the output would look like:

2021-05-01 17:36:52, 695: INFO: Permissions HevoDataSendMessage removed from the queue.

How to Process Python SQS Queue Messages?

Boto3 library gives you the capability to send, read and delete messages from a Python SQS Queue. The following sections explain how to use Boto3 library methods to process SQS Queue messages.

• Send Python SQS Messages
• Receive Python SQS Messages
• Delete Python SQS Messages
• Remove All SQS Queue Messages

Send Python SQS Messages 

To send a message, you can make use of the send_message() Boto3 method. The send_message() SQS Python method delivers a message to your specified Queue by adding it to the end of the Queue. 

Please Note: Your messages can only contain XML, JSON, and unformatted text, as well as the following Unicode characters. The Python AWS SQS Queue will reject your message if it contains any other characters.

#x9 | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000 to #x10FFFF

Here’s the code to send messages in Python SQS:

import logging
import boto3
from botocore.exceptions import ClientError
import json
AWS_REGION = 'us-east-1'
# logger config
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s: %(levelname)s: %(message)s')
sqs_client = boto3.client("sqs", region_name=AWS_REGION)

def send_queue_message(queue_url, msg_attributes, msg_body):
    """
    Sends a message to the specified queue.
    """
    try:
        response = sqs_client.send_message(QueueUrl=queue_url,
                                           MessageAttributes=msg_attributes,
                                           MessageBody=msg_body)
    except ClientError:
        logger.exception(f'Could not send meessage to the - {queue_url}.')
        raise
    else:
        return response

if __name__ == '__main__':
    # CONSTANTS
    QUEUE_URL = '<your-queue-url>'
    MSG_ATTRIBUTES = {
        'Title': {
            'DataType': 'String',
            'StringValue': 'Working with SQS in Python using Boto3'
        },
        'Author': {
            'DataType': 'String',
            'StringValue': 'Abhinav D'
        }
    }
    MSG_BODY = 'Learn how to create, receive, delete and modify SQS queues and see the other functions available within the AWS.'
    msg = send_queue_message(QUEUE_URL, MSG_ATTRIBUTES, MSG_BODY)
    json_msg = json.dumps(msg, indent=4)
    logger.info(f'''
        Message sent to the queue {QUEUE_URL}.
        Message attributes: \n{json_msg}''')

When executed successfully, the output would look like this:

{'MD5OfMessageBody': '88bac95f31528d13a072c05f2a1cf371', 'MessageId': '2ce1541b-0472-4715-8375-f8a8587c16e9', 'ResponseMetadata': {'RequestId': '02a7b659-c044-5357-885e-ee0c398e24b0', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '02a7b659-c044-5357-885e-ee0c398e24b0', 'date': 'Fri, 18 Dec 2020 00:27:54 GMT', 'content-type': 'text/xml', 'content-length': '378'}, 'RetryAttempts': 0}}

Receive Python SQS Messages

To receive a message from an AWS SQS Python Queue, you can use receive_message() Boto3 method. The receive_message() SQS Python method retrieves one or more messages (up to 10), from your specified SQS Queue.

Here’s the code to receive messages in Python SQS:

def receive_message():
    sqs_client = boto3.client("sqs", region_name="us-west-2")
    response = sqs_client.receive_message(
        QueueUrl="https://us-west-2.queue.amazonaws.com/xxx/my-new-queue",
        MaxNumberOfMessages=1,
        WaitTimeSeconds=10,
    )

    print(f"Number of messages received: {len(response.get('Messages', []))}")

    for message in response.get("Messages", []):
        message_body = message["Body"]
        print(f"Message body: {json.loads(message_body)}")
        print(f"Receipt Handle: {message['ReceiptHandle']}")

When executed successfully, the output would look like this:

Number of messages received: 1
Message body: {'key': 'value'}

Delete Python SQS Messages

To delete a message from an AWS SQS Python Queue, you can use the delete_message() Boto3 method. The delete_message() SQS Python method deletes up to ten messages from the specified Queue.

Here’s the code to delete messages in Python SQS:

def delete_message(receipt_handle):
    sqs_client = boto3.client("sqs", region_name="us-west-2")
    response = sqs_client.delete_message(
        QueueUrl="https://us-west-2.queue.amazonaws.com/xxx/my-new-queue",
        ReceiptHandle=receipt_handle,
    )
    print(response)

When executed successfully, the output would look like this:

{'ResponseMetadata': {'RequestId': 'd9a860cb-45ff-58ec-8232-389eb8d7c2c6', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'd9a860cb-45ff-58ec-8232-389eb8d7c2c6', 'date': 'Fri, 18 Dec 2020 00:42:16 GMT', 'content-type': 'text/xml', 'content-length': '215'}, 'RetryAttempts': 0}}

Remove All SQS Queue Messages

If you would like to remove all messages from your AWS SQS Python Queue, you can use the purge_queue() Boto3 method. The purge_queue() Boto3 method then deletes all the messages in a Queue specified by the QueueURL parameter.

Here’s the code to remove all messages:

def purge_queue():
    sqs_client = boto3.client("sqs", region_name="us-west-2")
    response = sqs_client.purge_queue(
        QueueUrl="https://us-west-2.queue.amazonaws.com/xxx/my-new-queue",
    )
    print(response)

It might take up to 60 seconds for the message deletion procedure to take effect. Before you run purge_queue(), all messages delivered to the Queue will be received but erased within the following minute. While the queue is being purged, all messages delivered to it after you use purge_queue() may be removed.

Bonus: How to Run Scheduled Messages Using SQS & Python?

Newscatcher is an API that scrapes the internet and combines over 1,000,000 news articles every day. Using the SQS Queues and AWS Lambda architecture, they are able to acquire and send referential data in CSV file formats and get updates on articles. This process involves importing a CSV file into a DynamoDB table using the Boto3 Python package. One method that you can use to do this is by using import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types) Python function as follows:

import boto

MY_ACCESS_KEY_ID = 'copy your access key ID here'
MY_SECRET_ACCESS_KEY = 'copy your secret access key here'


def do_batch_write(items, table_name, dynamodb_table, dynamodb_conn):
    '''
    From https://gist.github.com/griggheo/2698152#file-gistfile1-py-L31
    '''
    batch_list = dynamodb_conn.new_batch_write_list()
    batch_list.add_batch(dynamodb_table, puts=items)
    while True:
        response = dynamodb_conn.batch_write_item(batch_list)
        unprocessed = response.get('UnprocessedItems', None)
        if not unprocessed:
            break
        batch_list = dynamodb_conn.new_batch_write_list()
        unprocessed_list = unprocessed[table_name]
        items = []
        for u in unprocessed_list:
            item_attr = u['PutRequest']['Item']
            item = dynamodb_table.new_item(
                    attrs=item_attr
            )
            items.append(item)
        batch_list.add_batch(dynamodb_table, puts=items)


def import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types):
    '''
    Import a CSV file to a DynamoDB table
    '''        
    dynamodb_conn = boto.connect_dynamodb(aws_access_key_id=MY_ACCESS_KEY_ID, aws_secret_access_key=MY_SECRET_ACCESS_KEY)
    dynamodb_table = dynamodb_conn.get_table(table_name)     
    BATCH_COUNT = 2 # 25 is the maximum batch size for Amazon DynamoDB
    
    items = []
    
    count = 0
    csv_file = open(csv_file_name, 'r')
    for cur_line in csv_file:
        count += 1
        cur_line = cur_line.strip().split(',')
        
        row = {}
        for colunm_number, colunm_name in enumerate(colunm_names):
            row[colunm_name] = column_types[colunm_number](cur_line[colunm_number])
         
        item = dynamodb_table.new_item(
                    attrs=row
            )           
        items.append(item)
        
        if count % BATCH_COUNT == 0:
            print 'batch write start ... ', 
            do_batch_write(items, table_name, dynamodb_table, dynamodb_conn)
            items = []
            print 'batch done! (row number: ' + str(count) + ')'
    
    # flush remaining items, if any
    if len(items) > 0: 
        do_batch_write(items, table_name, dynamodb_table, dynamodb_conn)

        
    csv_file.close() 


def main():
    '''
    Demonstration of the use of import_csv_to_dynamodb()
    We assume the existence of a table named `test_persons`, with
    - Last_name as primary hash key (type: string)
    - First_name as primary range key (type: string)
    '''
    colunm_names = 'Last_name First_name'.split()
    table_name = 'test_persons'
    csv_file_name = 'test.csv'
    column_types = [str, str]
    import_csv_to_dynamodb(table_name, csv_file_name, colunm_names, column_types)
    

if __name__ == "__main__":
    main()
    #cProfile.run('main()') # if you want to do some profiling

Challenges of Using AWS SQS Queues

While AWS Simple Queue Service comes with its own set of advantages like unlimited Python SQS Queues and messages, high durability, unmatched security, and pay-for-what-you-use pricing, it’s not to say that it’s devoid of any performance challenges. Consider the following aspects that AWS SQS Queues face:

• Latency Problem: Once you’ve added a message to the Queue. It takes around one minute for the message to become available for reading. Also, the order in which a message is received after being queued is not guaranteed.
• High Scalability Costs: With pay-per-use pricing, your SQS cost might quickly grow if you send many messages. SQS price includes data transport charges, which can mount up if you send big messages or process messages from outside the primary AWS area where the Queue is situated.

Conclusion

This guide discussed AWS Simple Queue Service (SQS) which enables decoupling and communication among distributed system components. Using the Boto3 Python Package, you can perform a range of operations like creating a new Python SQS Queue, fetching Queue URLs, and setting Queue attributes, tags, and permissions. Additionally, we examined different ways to process AWS SQS Python Queues like how to send, receive, and delete certain messages from SQS Python Queue and how to completely remove all SQS Queue messages. 

Amazon Web Services (AWS) provides services and infrastructure to build reliable, fault-tolerant, and highly available systems in the cloud. Just like AWS, Hevo Data, a No-Code Automation and Data Pipeline Creation Tool helps you to build your own fault-tolerant, reliable, and zero-data loss Data Pipelines in the cloud. 

Hevo Data is a comprehensive ETL platform that allows you to migrate data from 150+ Data Sources like Amazon S3, Amazon Relational Database Services like Amazon RDS on PostgreSQL, Amazon RDS on MySQL, Oracle on Amazon RDS, MySQL on Amazon Aurora, and many more.

Why not try Hevo and see the magic for yourself? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite firsthand. You can also check our unbeatable pricing and make a decision on your best-suited plan.

Have any questions on Python SQS Queues? Do let us know in the comment section below. Also, share any other AWS Services or features you’d want us to cover. We’d be happy to know your opinions.

FAQs