Modern-day businesses deal with a massive amount of semi-structured data on a daily basis. To effectively handle the exponentially growing data, No-SQL databases such as AWS DynamoDB and Azure Cosmos DB are popular choices among data engineers. Based on the various business & engineering requirements, organizations are required to migrate their applications and data from one database to other.
Setting up the DynamoDB Azure connection is also a fairly simple process. Using the Data Migration Tool provided by Azure for free, you can complete the DynamoDB Azure Migration process. With the right credentials and connection strings, Data Migration Tool imports your DynamoDB table data quickly into Azure Cosmos DB.
In this article, you will learn how to effectively set up the DynamoDB Azure Integration.
What is Azure?
Image Source
Microsoft Azure is a public Cloud Computing platform that allows you to access and manage Microsoft’s cloud services and resources. The cloud service includes best-in-class computation, Data Analytics, Storage, networking, IoT, migration, artificial intelligence, and other machine learning, integration, management tools, developer tools, security, databases, DevOps, media identity, and web services. For instance, Microsoft Azure SQL Database is a completely managed PaaS(Platform as a Service) offering continuous upgrading, patching, backups, and monitoring of the database.
Key Features of Azure
- Improved Backup & Data Recovery: Azure allows you to back your data from any language or operating system & also lets you decide the frequency of the data backup cycle. As a preventive measure, Azure stores 3 copies of your in 3 different places as well as three separate copies in a remote Azure data center.
- Manageability: Features such as Automatic Patch Management for virtual machines allow you to focus on your core objectives. Azure scales automatically as your business scales, thereby providing a seamless experience.
- Analytics Capabilities: Cortana Analytics, Stream Analytics, Machine Learning, and SQL services are some of the brilliant analytics tools provided by Azure to assist you in discovering new business opportunities, improving customer service, and making informed decisions.
- Flexibility: Microsft Azure allows you to work with multiple programming languages, including Java, Node Js, and C#. After developing your applications, Azure also provides a platform to test and deploy them.
As the ability of businesses to collect data explodes, data teams have a crucial role to play in fueling data-driven decisions. Yet, they struggle to consolidate the data scattered across sources into their warehouse to build a single source of truth. Broken pipelines, data quality issues, bugs and errors, and lack of control and visibility over the data flow make data integration a nightmare.
1000+ data teams rely on Hevo’s Data Pipeline Platform to integrate data from over 150+ sources like DynamoDB in a matter of minutes. Billions of data events from sources as varied as SaaS apps, Databases, File Storage and Streaming sources can be replicated in near real-time with Hevo’s fault-tolerant architecture. Hevo currently does not support Azure as destination. What’s more – Hevo puts complete control in the hands of data teams with intuitive dashboards for pipeline monitoring, auto-schema management, and custom ingestion/loading schedules.
All of this combined with transparent pricing and 24×7 support makes us the most loved data pipeline software on review sites.
Take our 14-day free trial to experience a better way to manage data pipelines.
Get started for Free with Hevo!
What is Amazon DynamoDB?
Image Source
Amazon DynamoDB is a completely managed NoSQL database service that you can use to create database tables that can store and retrieve any amount of data. It completely handles all the administrative tasks such as hardware provisioning, setup, and configuration, replication, software patches, or cluster scaling. Using your AWS Management Console, you can simply monitor resource usage and performance metrics.
Key Features of DynamoDB
DynamoDB offers the following eye-catching features:
- Managed: Automatically manages table traffic between multiple servers to maintain performance.
- Security: Your data is secured with DynamoDB’s inbuilt encryption capabilities. This eliminates the operational overhead and complexity associated with protecting sensitive data.
- Flexible: AWS allows you to easily scale up or down the throughput capacity of a table as you require without any downtime or performance degradation.
- On-Demand Backups: DynamoDB provides you with the ability to create full backups of your tables for long-term retention and archival for regulatory compliance needs.
Why set up the DynamoDB Azure Connection?
Moving your data to Azure CosmosDB and setting up the DynamoDB Azure Connection has the following benefits:
- The JSON structure of Azure Cosmos DB is much simpler in comparison to that of DynamoDB.
- While setting up DynamoDB, you are required to provide the read compute units & write compute units. However, you only need to specify the throughput as Request Units (RU/s) in Cosmos DB, which can be used for any operations dynamically.
- To query documents in DynamoDB, you need API functions. In Cosmos DB, you can easily do projection and filter inside a simple SQL query.
- As Azure Cosmos DB supports natural queries (SQL), operations are asynchronous and much easier.
Prerequisites
- Working knowledge of DynamoDB & CosmosDB.
- An AWS account.
- An Azure CosmosDB account.
How to set up DynamoDB Azure CosmosDB Connection?
Using the Data Migration Tool, you can easily complete the DynamoDB Azure Migration process. To get started with the DynamoDB Azure connection, you can follow the simple steps given below:
- Step 1: Firstly, log in to your AWS DynamoDB account. In this DynamoDB Azure connection article, a sample “Orders” table containing 5 records is considered for replicating data from DynamoDB to Azure CosmosDB. A read capacity of 5 is set for this table.
Image Source
Depending on your table, you can configure the partition key and the read capacity. You can do this by clicking on the Table name and navigating to the respective tab for configuring your table settings.
Image Source
- Step 2: In this DynamoDB Azure Connection step, navigate to Services> IAM > Users.
Since you need to send data from DynamoDB to Azure CosmosDB, you need a user that has read-only access to your DynamoDB Tables.
Image Source
You can click on the Add User button and then click on the Add permissions button to configure user access settings. In this article, a user named “DynamoDBUser” has been added with read-only permissions.
Image Source
- Step 3: To set up the DynamoDB Azure Integration, you need the connection strings for both of these databases. The format for your DynamoDB connection string is:
ServiceURL=<Service Address>;AccessKey=<Access Key>;SecretKey=<Secret Key>;
The service URL is the endpoint for a specific AWS region that acts as an entry point for an AWS web service. You can find the endpoint for your AWS region on this official AWS DynamoDB documentation page. For your Access Key and Secret Key, you can follow the steps in the documentation.
- Step 4: In a new tab, login to your Azure CosmosDB Account and click on the New Container button. Name your Database and enter your AWS DynamoDB details like the table name & the partition key.
Image Source
Enter the throughput according to your table size. For this example, 400 is selected as it is the minimum option available. Once done, click on the OK button.
Image Source
- Step 5: In the side vertical menu, click on the Keys option and copy the Primary Connection String to a notepad. Now, add the database to the string in the format Database=<databse name>. In this example, it will be Database=MyCompany
Image Source
- Step 6: In this DynamoDb Azure Connection Step, download the Data Migration Tool from here. Unzip the directory and open the DTUI.exe file. A security warning window will pop up. Click on the Run button to proceed.
- Step 7: The Azure Cosmos DB Data Migration Tool window will appear on your scream To begin with the DynamoDB Azure Connection, click on the Source Information option available on the left vertical menu.
Image Source
- Step 8: Select DynamoDB from the “Import from” drop-down menu. Paste your DynamoDB connection string and click on the Verify button to validate the connection to the database. At last, enter your table name in the Enter Request field and click on the Next button to continue with DynamoDB Azure Connection.
Image Source
- Step 9: Select Azure CosmosDB – Sequential Record Import from the drop-down menu. Paste the Azure Connection string & click on the Verify button. Fill in your input table details and click on the Next button to proceed with the DynamoDB Azure migration process.
Image Source
- Step 10: If you want, you can provide the log file. Select the Update interval and click on the Next button. Finally, a summary of your DynamoDB Azure Connection will be presented. Click on the Import button to complete the DynamoDb Azure Migration process.
Image Source
Conclusion
In this article, you have learned how to easily set up the DynamoDB Azure Migration process. Azure provides a Data Migration Tool that allows you to import data from DynamoDB into Azure in just a few clicks. To connect the 2 databases, you need authentic connection strings for both of them. After entering the database and table details, the Data Migration Tool fetches your data.
As you collect and manage your data across several applications and databases in your business, it is important to consolidate it for complete performance analysis of your business. However, it is a time-consuming and resource-intensive task to continuously monitor the Data Connectors. To achieve this efficiently, you need to assign a portion of your engineering bandwidth to Integrate data from all sources, Clean & Transform it, and finally, Load it to a Cloud Data Warehouse, BI Tool, or a destination of your choice for further Business Analytics. All of these challenges can be comfortably solved by a Cloud-based ETL tool such as Hevo Data.
Visit our Website to Explore Hevo
Hevo Data, a No-code Data Pipeline can Ingest Data in Real-Time from a vast sea of 150+ sources like DynamoDB to a Data Warehouse or a Destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!
If you are using a NoSQL Database like DynamoDB and searching for a seamless alternative to Manual Data Integration, then Hevo can effortlessly automate this for you. Hevo’s strong integration with 150+ Connectors (Including 40+ Free Sources), allows you to export, load, transform & enrich your data to make it analysis-ready.
Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.
Share your experience of setting up the DynamoDB Azure Connection! Let us know in the comments section below!