How to Connect SQS to DynamoDB?

Enterprise applications receive and work with a high volume of messages and notifications, which can put a lot of stress on the system and hence affect the performance negatively. Enterprise applications thus require a complex messaging system to manage such high volumes of data.

Amazon SQS (Simple Queue Service) is one such service that helps enterprises avoid any overloading of their systems and applications. It stores the messages in the form of queues and makes it possible for the applications to handle the data messages on time.

This article aims at providing you with a step-by-step guide to help you connect Amazon SQS to DynamoDB in a hassle-free manner to transfer your data messages efficiently.

Prerequisites

• Working knowledge of Amazon SQS.
• Working knowledge of Amazon DynamoDB.
• A general idea of Python.
• A general idea of APIs.

What is Amazon SQS?

Amazon SQS is a message queuing service that allows enterprises to run business services and applications in a way that messaging remains independent of the IT infrastructure. This way, messages can run and fail independently of each other to avoid slowdowns, disturbance, or system-wide faults within the application.

Amazon SQS allows enterprises to decouple and scale distributed systems, microservices, and serverless applications. It further eliminates the complexity associated with operating and managing message-oriented middleware and allows developers to focus on differentiating work.

Amazon SQS makes use of two different kinds of message queues, namely Standard and FIFO queues. “Standard queues” ensure maximum throughput, best-effort ordering, and at-least-once delivery, whereas the “FIFO queues” ensure that messages are processed exactly once and in the exact order of their arrival. It further sends server-side encrypted messages, reducing the security risks involved when sending messages between applications running on the cloud.

Key Features of Amazon SQS

Some of the main features of Amazon SQS are listed below:

• Long Polling: Amazon SQS helps in minimizing the cost while receiving messages by reducing extraneous polling. The long poll messages wait for upto 20 seconds for the next message to arrive.
• Scalability: Amazon SQS decouples the processes that make it easier to scale up sending or receiving rates of messages by simply adding another process.
• Message Locking: when Amazon SQS receives a message it gets locked while being processed which allows other computers to process messages simultaneously.
• Server-side Encryption: Amazon SQS protects the messages using the AWS-managed Key Management Service (KMS). It also encrypts messages through SSE as soon as Amazon SQS receives them.

For further information on Amazon SQS, you can check the official website here.

What is Amazon DynamoDB?

Amazon DynamoDB is a managed NoSQL database offered by Amazon. It is available as a part of Amazon’s data warehousing suite called Amazon Web Services (AWS). Being a NoSQL database, it stores data in the form of collections, which further contain documents that store data in the form of a key-value pair. Each record is accessed with the help of a primary key and key-value pairs, using its proprietary querying layer.

Amazon DynamoDB makes use of a collection of nodes, each of which contains several primary keys, so when a query executes, only those nodes which contain those primary keys get activated and fetch data. It is known for its scalability, ease of use, reliability & no compulsion for using a fixed schema among all stored documents, giving them the ability to have varying fields (columns).

Amazon DynamoDB is connected with different Datases or Data Warehouses and many organizations load data from Amazon DynamoDB to Amazon S3 for storing and analytics purposes, you can automate this process using Hevo, a fully managed No-Code Data Pipelined.

Key Features of Amazon DynamoDB 

Some of the main features of Amazon DynamoDB are listed below:

• Auto-Scaling: Amazon DynamoDB offers auto-scaling of throughput and storage for the tables using provisioned capacity. If the traffic increases it will automatically increase the throughput to accommodate the load.
• Point-time Recovery: It helps protect DynamoDB from accidental write or delete operations. It offers a continuous backup option that users can use to restore from the second data changed to upto 35 days.
• On-demand Mode: DynamoDB can automatically adjust itself to accommodate the workloads as they ramp up or down to any previously reached traffic.
• DynamoDB Accelerator: DynamoDB can deliver fast read performance of tables using DynamoDB Accelerator which is an in-memory cache allowing you to use fully-managed in-memory cache.

For further information on Amazon DynamoDB, you can check the official website here.

Why Migrate Data from Amazon SQS to DynamoDB?

1. Integration of Amazon SQS to DynamoDB can build a fast, scalable, and reliable layer to receive and process high volumes of messages using its distributed and highly available architecture.
2. Connecting Amazon SQS DynamoDB can be used to develop a full system, capable of handling any level of throughput or amount of data, without requiring other services to be always available.
3. With an Amazon SQS DynamoDB integration, applications can also process messages asynchronously. SQS DynamoDB connection further lets you add more workers and resources to improve the performance, depending upon the resources enqueued.
4. Manually performing the Data Migration process to Amazon DynamoDB can be tedious and time-consuming and to combat this challenge, Automated No-code Data Pipelines like Hevo exist.

Steps to Migrate Messages from SQS to DynamoDB

One popular & efficient way of transferring messages from SQS to DynamoDB is by making use of Python-based Lambda functions. Since Amazon SQS generally places all the data messages and notifications generated by applications in a queue, the Python-based Lambda function will be able to write the data messages from the Amazon SQS to DynamoDB table.

You can implement the migration from SQS to DynamoDB using the following steps:

• Step 1: Creating a Table in DynamoDB
• Step 2: Setting up AWS SQS Event Source using a Lambda Function

Step 1: Creating a Table in DynamoDB

To start loading your messages from Amazon SQS to DynamoDB, you first need to create a table, with attributes that match your data messages. To do this, go to the AWS console for DynamoDB and click on the “create table” option.

Once you’ve clicked on it, the DynamoDB create table page will now open up on your screen, where you need to provide the necessary information as follows:

Once you’ve added the information related to the table name and primary key, select the default settings option and click on Create to create table to get messages from SQS to DynamoDB.

This is how you can create a table for SQS to DynamoDB connection.

Step 2: Setting up AWS SQS Event Source using a Lambda Function

Once you’ve created your DynamoDB table, you now need to code the Python-based Lambda function that will transfer the messages from SQS to DynamoDB. To do this, you will need to make use of Boto3, the official AWS SDK for Python, that allows you to integrate your Python scripts with AWS services such as SQS so that you can send messages from SQS to DynamoDB.

You can create your Lambda function to send messages from SQS to DynamoDB using the following lines of code:

import boto3
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.info('Loading function')
dynamo_client = boto3.client('dynamodb')
def lambda_handler(event, context):
    operations = {
        'DELETE': lambda dynamo, x: dynamo.delete_item(**x),
        'POST': lambda dynamo, x: dynamo.put_item(**x),
        'PUT': lambda dynamo, x: dynamo.update_item(**x),
        'GET': lambda dynamo, x: dynamo.get_item(**x),
        'GET_ALL': lambda dynamo, x: dynamo.scan(**x),
    }
    for record in event['Records']:
        payload = loads(record['body'], parse_float=str)
        operation = record['messageAttributes']['Method']['stringValue']
        if operation in operations:
            try:
                operations[operation](dynamo_client, payload)
            except Exception as e:
                logger.error(e)
        else:
            logger.error('Unsupported method '{}''.format(operation))
 return 0

With your Lambda function now ready and functional, click on the SQS Lambda triggers tab, you will now be able to see your SQS events facilitate the Lambda function as follows:

Now, every time a new message comes into the Amazon SQS queue, Amazon SQS will automatically trigger the Lambda function and transfer the message from SQS to DynamoDB.

This is how you can transfer messages from SQS to DynamoDB using Python-based Lambda functions.

Limitations of Amazon SQS to DynamoDB

Though there are many applications of trasferring messages from Amazon SQS to DynamoDB using Lambda function but it has some limitations that need to be considered. The following limitations of trasferring messages from Amazon SQS to DynamoDB are listed below:

• For trasferring messages from Amazon SQS to DynamoDB with this method requires you to have an in-depth knowledge of AWS & its services and further requires you to write a lot of custom code.
• Limitations related to execution time, storage, and run-time memory for Amazon SQS to DynamoDB, may not be practical for real-life scenarios. 

Conclusion

This article teaches you how to connect SQS to DynamoDB. It also provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. You also read about the limitations of trasferring messages from Amazon SQS to DynamoDB using Lambda function. Now, the manual approach of connecting SQS to DynamoDB will add complex overheads in terms of time, and resources. Such a solution will require skilled engineers and regular data updates.

Hevo Data provides an Automated No-code Data Pipeline that empowers you to overcome the above-mentioned limitations. Hevo caters to 100+ data sources (including 40+ free sources) and can seamlessly transfer DynamoDB data to the Data Warehouse of your choice in real-time. Hevo’s Data Pipeline enriches your data and manages the transfer process in a fully automated and secure manner without having to write any code. It will make your life easier and make data migration hassle-free.

Learn more about Hevo

Frequently Asked Question