DynamoDB Azure Connection: A Comprehensive Guide 101

on Data Integration, Data Migration, Database Management Systems, DynamoDB, Microsoft Azure, NoSQL • July 15th, 2022 • Write for Hevo

DynamoDB Azure - Featured Image

Modern-day businesses deal with a massive amount of semi-structured data on a daily basis. To effectively handle the exponentially growing data, No-SQL databases such as AWS DynamoDB and Azure Cosmos DB are popular choices among data engineers. Based on the various business & engineering requirements, organizations are required to migrate their applications and data from one database to other. 

Setting up the DynamoDB Azure connection is also a fairly simple process. Using the Data Migration Tool provided by Azure for free, you can complete the DynamoDB Azure Migration process. With the right credentials and connection strings, Data Migration Tool imports your DynamoDB table data quickly into Azure Cosmos DB.

In this article, you will learn how to effectively set up the DynamoDB Azure Integration. 

Table of Contents

What is Azure?

DynamoDB Azure - Azure Logo
Image Source

Microsoft Azure is a public Cloud Computing platform that allows you to access and manage Microsoft’s cloud services and resources. The cloud service includes best-in-class computation, Data Analytics, Storage, networking, IoT, migration, artificial intelligence, and other machine learning, integration, management tools, developer tools, security, databases, DevOps, media identity, and web services. For instance, Microsoft Azure SQL Database is a completely managed PaaS(Platform as a Service) offering continuous upgrading, patching, backups, and monitoring of the database.

Key Features of Azure

  • Improved Backup & Data Recovery: Azure allows you to back your data from any language or operating system & also lets you decide the frequency of the data backup cycle. As a preventive measure, Azure stores 3 copies of your in 3 different places as well as three separate copies in a remote Azure data center.  
  • Manageability: Features such as Automatic Patch Management for virtual machines allow you to focus on your core objectives. Azure scales automatically as your business scales, thereby providing a seamless experience.
  • Analytics Capabilities: Cortana Analytics, Stream Analytics, Machine Learning, and SQL services are some of the brilliant analytics tools provided by Azure to assist you in discovering new business opportunities, improving customer service, and making informed decisions.
  • Flexibility: Microsft Azure allows you to work with multiple programming languages, including Java, Node Js, and C#. After developing your applications, Azure also provides a platform to test and deploy them.

Scale your data integration effortlessly with Hevo’s Fault-Tolerant No Code Data Pipeline

As the ability of businesses to collect data explodes, data teams have a crucial role to play in fueling data-driven decisions. Yet, they struggle to consolidate the data scattered across sources into their warehouse to build a single source of truth. Broken pipelines, data quality issues, bugs and errors, and lack of control and visibility over the data flow make data integration a nightmare.

1000+ data teams rely on Hevo’s Data Pipeline Platform to integrate data from over 150+ sources like DynamoDB in a matter of minutes. Billions of data events from sources as varied as SaaS apps, Databases, File Storage and Streaming sources can be replicated in near real-time with Hevo’s fault-tolerant architecture. Hevo currently does not support Azure as destination. What’s more – Hevo puts complete control in the hands of data teams with intuitive dashboards for pipeline monitoring, auto-schema management, and custom ingestion/loading schedules. 

All of this combined with transparent pricing and 24×7 support makes us the most loved data pipeline software on review sites.

Take our 14-day free trial to experience a better way to manage data pipelines.

Get started for Free with Hevo!

What is Amazon DynamoDB?

DynamoDB Azure - DynamoDB Logo
Image Source

Amazon DynamoDB is a completely managed NoSQL database service that you can use to create database tables that can store and retrieve any amount of data. It completely handles all the administrative tasks such as hardware provisioning, setup, and configuration, replication, software patches, or cluster scaling. Using your AWS Management Console, you can simply monitor resource usage and performance metrics.

Key Features of DynamoDB

DynamoDB offers the following eye-catching features:

  • Managed: Automatically manages table traffic between multiple servers to maintain performance. 
  • Security: Your data is secured with DynamoDB’s inbuilt encryption capabilities. This eliminates the operational overhead and complexity associated with protecting sensitive data.
  • Flexible: AWS allows you to easily scale up or down the throughput capacity of a table as you require without any downtime or performance degradation.
  • On-Demand Backups: DynamoDB provides you with the ability to create full backups of your tables for long-term retention and archival for regulatory compliance needs. 

Why set up the DynamoDB Azure Connection?

Moving your data to Azure CosmosDB and setting up the DynamoDB Azure Connection has the following benefits:

  • The JSON structure of Azure Cosmos DB is much simpler in comparison to that of DynamoDB.
  • While setting up DynamoDB, you are required to provide the read compute units & write compute units. However, you only need to specify the throughput as Request Units (RU/s) in Cosmos DB, which can be used for any operations dynamically.
  • To query documents in DynamoDB, you need API functions. In Cosmos DB, you can easily do projection and filter inside a simple SQL query.
  • As Azure Cosmos DB supports natural queries (SQL), operations are asynchronous and much easier. 

Prerequisites

  • Working knowledge of DynamoDB & CosmosDB.
  • An AWS account.
  • An Azure CosmosDB account.

All of the capabilities, none of the firefighting:-

Using manual scripts and custom code to move data into the warehouse is cumbersome. Frequent breakages, pipeline errors, and lack of data flow monitoring make scaling such a system a nightmare. Hevo’s reliable data pipeline platform enables you to set up zero-code and zero-maintenance data pipelines that just work.

  • Reliability at Scale – With Hevo, you get a world-class fault-tolerant architecture that scales with zero data loss and low latency. 
  • Monitoring and Observability – Monitor pipeline health with intuitive dashboards that reveal every stat of pipeline and data flow. Bring real-time visibility into your ELT with Alerts and Activity Logs 
  • Stay in Total Control – When automation isn’t enough, Hevo offers flexibility – data ingestion modes, ingestion, and load frequency, JSON parsing, destination workbench, custom schema management, and much more – for you to have total control.    
  • Auto-Schema Management – Correcting improper schema after the data is loaded into your warehouse is challenging. Hevo automatically maps source schema with destination warehouse so that you don’t face the pain of schema errors.
  • 24×7 Customer Support – With Hevo you get more than just a platform, you get a partner for your pipelines. Discover peace with round-the-clock “Live Chat” within the platform. What’s more, you get 24×7 support even during the 14-day full-featured free trial.
  • Transparent Pricing – Say goodbye to complex and hidden pricing models. Hevo’s Transparent Pricing brings complete visibility to your ELT spend. Choose a plan based on your business needs. Stay in control with spend alerts and configurable credit limits for unforeseen spikes in the data flow. 
Get started for Free with Hevo!

How to set up DynamoDB Azure CosmosDB Connection?

Using the Data Migration Tool, you can easily complete the DynamoDB Azure Migration process. To get started with the DynamoDB Azure connection, you can follow the simple steps given below:

  • Step 1: Firstly, log in to your AWS DynamoDB account. In this DynamoDB Azure connection article, a sample “Orders” table containing 5 records is considered for replicating data from DynamoDB to Azure CosmosDB. A read capacity of 5 is set for this table.
DynamoDB Azure - Sample Table
Image Source

Depending on your table, you can configure the partition key and the read capacity. You can do this by clicking on the Table name and navigating to the respective tab for configuring your table settings. 

DynamoDB Azure - Items in the Table
Image Source
  • Step 2: In this DynamoDB Azure Connection step, navigate to Services> IAM > Users.

Since you need to send data from DynamoDB to Azure CosmosDB, you need a user that has read-only access to your DynamoDB Tables. 

DynamoDB Azure - Adding DynamoDB User with Read-only access
Image Source

You can click on the Add User button and then click on the Add permissions button to configure user access settings. In this article, a user named “DynamoDBUser” has been added with read-only permissions. 

DynamoDB Azure - Read-only access
Image Source
  • Step 3: To set up the DynamoDB Azure Integration, you need the connection strings for both of these databases. The format for your DynamoDB connection string is:
ServiceURL=<Service Address>;AccessKey=<Access Key>;SecretKey=<Secret Key>;

The service URL is the endpoint for a specific AWS region that acts as an entry point for an AWS web service. You can find the endpoint for your AWS region on this official AWS DynamoDB documentation page. For your Access Key and Secret Key, you can follow the steps in the documentation.

  • Step 4: In a new tab, login to your Azure CosmosDB Account and click on the New Container button. Name your Database and enter your AWS DynamoDB details like the table name & the partition key.
DynamoDB Azure - New CosmosDB Container
Image Source

Enter the throughput according to your table size. For this example, 400 is selected as it is the minimum option available. Once done, click on the OK button.

DynamoDB Azure - Throughput
Image Source
  • Step 5: In the side vertical menu, click on the Keys option and copy the Primary Connection String to a notepad. Now, add the database to the string in the format Database=<databse name>. In this example, it will be Database=MyCompany
DynamoDB Azure - CosmosDb Keys option
Image Source
  • Step 6: In this DynamoDb Azure Connection Step, download the Data Migration Tool from here. Unzip the directory and open the DTUI.exe file. A security warning window will pop up. Click on the Run button to proceed.
  • Step 7: The Azure Cosmos DB Data Migration Tool window will appear on your scream To begin with the DynamoDB Azure Connection, click on the Source Information option available on the left vertical menu. 
DynamoDB Azure - data migration tool Intro page
Image Source
  • Step 8: Select DynamoDB from the “Import from” drop-down menu. Paste your DynamoDB connection string and click on the Verify button to validate the connection to the database. At last, enter your table name in the Enter Request field and click on the Next button to continue with DynamoDB Azure Connection.  
DynamoDB Azure - Source Information
Image Source
  • Step 9: Select Azure CosmosDB – Sequential Record Import from the drop-down menu. Paste the Azure Connection string & click on the Verify button. Fill in your input table details and click on the Next button to proceed with the DynamoDB Azure migration process.
DynamoDB Azure - Target information
Image Source
  • Step 10: If you want, you can provide the log file. Select the Update interval and click on the Next button. Finally, a summary of your DynamoDB Azure Connection will be presented. Click on the Import button to complete the DynamoDb Azure Migration process. 
DynamoDB Azure - Import button
Image Source

Conclusion

In this article, you have learned how to easily set up the DynamoDB Azure Migration process. Azure provides you with a Data Migration Tool that allows you to import data from DynamoDB into Azure in just a few clicks. To connect the 2 databases, you need authentic connection strings for both of them. After entering the database and table details, the Data Migration Tool starts fetching your data.

As you collect and manage your data across several applications and databases in your business, it is important to consolidate it for complete performance analysis of your business. However, it is a time-consuming and resource-intensive task to continuously monitor the Data Connectors. To achieve this efficiently, you need to assign a portion of your engineering bandwidth to Integrate data from all sources, Clean & Transform it, and finally, Load it to a Cloud Data Warehouse, BI Tool, or a destination of your choice for further Business Analytics. All of these challenges can be comfortably solved by a Cloud-based ETL tool such as Hevo Data.   

Visit our Website to Explore Hevo

Hevo Data, a No-code Data Pipeline can Ingest Data in Real-Time from a vast sea of 100+ sources like DynamoDB to a Data Warehouse or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!  

If you are using a NoSQL Database like DynamoDB and searching for a seamless alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo’s strong integration with 150+ Connectors (Including 40+ Free Sources), allows you to export, load, transform & enrich your data to make it analysis-ready.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.

Share your experience of setting up the DynamoDB Azure Connection! Let us know in the comments section below!

No-code Data Pipeline For DynamoDB