Table of Contents AWS DocumentDB: A Brief OverviewSnowflake: A Brief OverviewWhy Migrate to Snowflake?Methods for Integrating AWS DocumentDB and SnowflakeMethod 1: Migrate Data from AWS DocumentDB to Snowflake Using CSV Export/Import MethodMethod 2: Sync AWS DocumentDB to Snowflake Using Hevo DataUse Cases of AWS DocumentDB to Snowflake IntegrationConclusionFrequently Asked Questions (FAQs) Try Hevo for Free Share Share To LinkedIn Share To Facebook Share To X Copy Link Easily move your data from AWS DocumentDB To Snowflake to enhance your analytics capabilities. With Hevo’s intuitive pipeline setup, data flows in real-time—check out our 1-minute demo below to see the seamless integration in action! Your organization utilizes AWS DocumentDB as an operational data store to manage customer information. However, AWS DocumentDB lacks the real-time analytics capabilities needed for timely insights. To achieve this, your organization must implement a real-time migration from AWS DocumentDB to an advanced analytical platform like Snowflake. It offers advanced query processing, data warehousing, and analytics solutions to enhance data management and derive valuable insights. Snowflake’s real-time data streaming lets you make timely decisions to improve business performance. Let’s get into the detailed migration process of AWS DocumentDB to Snowflake. Table of Contents AWS DocumentDB: A Brief OverviewSnowflake: A Brief OverviewWhy Migrate to Snowflake?Methods for Integrating AWS DocumentDB and SnowflakeMethod 1: Migrate Data from AWS DocumentDB to Snowflake Using CSV Export/Import MethodMethod 2: Sync AWS DocumentDB to Snowflake Using Hevo DataUse Cases of AWS DocumentDB to Snowflake IntegrationConclusionFrequently Asked Questions (FAQs) AWS DocumentDB: A Brief Overview AWS DocumentDB is a fully managed, NoSQL JSON document database that is limited in compatibility with MongoDB. It allows you to store, query, index, and aggregate critical JSON data of any size cost-effectively. AWS DocumentDB is known for its scalability, low-latency global reads, durability, and built-in security practices. AWS DocumentDB integrates with vector search, a machine learning technique, to find similar data points using distance metrics. With millisecond response times, this built-in feature helps you search millions of data within your document database based on nuanced meaning and context. Additionally, AWS DocumentDB lets you make your applications smarter and more responsive through integrations with generative AI and other machine-learning capabilities. Snowflake: A Brief Overview Snowflake is a cloud-based data warehousing platform that powers the Data Cloud, offering the flexibility to load, analyze, and securely distribute data. It allows you to aggregate unstructured, semi-structured, and structured data into a centralized location, providing access to all necessary information without data silos. Snowflake helps you effectively manage your workloads with optimized compression and automatic micro-partitioning. As your workload grows, Snowflake’s single elastic performance engine dynamically allocates necessary compute power and storage capacity in near real-time. You can connect to Snowflake using Snowsight, SnowSQL utility, and native connectors like Python or Spark. Why Migrate to Snowflake? Here are a few reasons why you should transfer data from Amazon DocumentDB to Snowflake: No Vendor Lock-in: Unlike AWS DocumentDB, which ties you to the AWS ecosystem, Snowflake’s data warehouse can be deployed on multiple cloud infrastructures, such as Azure, AWS, or Google Cloud Platform. Compatibility with Existing Infrastructure: Snowflake allows you to store the nested objects in a relational table or their native format. Since AWS DocumentDB exclusively supports JSON, Snowflake can directly query or manipulate the JSON data using ANSI-standard SQL. Seamlessly Connect AWS DocumentDB to Snowflake with Hevo Hevo Data is a real-time ELT, no-code pipeline platform that helps you cost-effectively automate data pipelines that are flexible to your needs. By integrating with 150+ data sources, Hevo Data lets you export data from multiple sources, load it to destinations, and transform it for detailed analysis. Why Hevo is the Best: Minimal Learning Curve: Hevo’s simple, interactive UI makes it easy for new users to get started and perform operations. Connectors: With over 150 connectors, Hevo allows you to integrate various data sources into your preferred destination seamlessly. Schema Management: Hevo eliminates the tedious task of schema management by automatically detecting and mapping incoming data to the destination schema. Live Support: The Hevo team is available 24/7, offering exceptional support through chat, email, and calls. Cost-Effective Pricing: Transparent pricing with no hidden fees, helping you budget effectively while scaling your data integration needs. Try Hevo today and experience seamless data transformation and integration. Get Started with Hevo for Free Methods for Integrating AWS DocumentDB and Snowflake To migrate data from AWS DocumentDB to Snowflake, you can utilize Hevo Data or CSV Export/Import Method. Method 1: Migrate Data from AWS DocumentDB to Snowflake Using CSV Export/Import Method If you are wondering, “How do I ingest data from AWS Document DB to Snowflake through a manual CSV Export/Import method?” This section outlines the steps in detail. You must initially export the data from AWS DocumentDB in CSV format using the mongoexport tool. Then, you can load the CSV data into Snowflake using the named internal stage. Step 1: Export Data From AWS DocumentDB in CSV Format Using mongoexport Tool Before you begin, ensure the following prerequisites are in place: Amazon DocumentDB version 4.0 or above. Access to your Amazon DocumentDB cluster. Create an AWS EC2 instance in the same VPC as your DocumentDB cluster. Connect the Amazon EC2 instance to your Amazon DocumentDB. Connect to your EC2 instance and install the mongoexport database tool. Download the global-bundle-pem CA file and save it to an EC2 instance for Amazon DocumentDB to establish secure connections while using MongoDB database tools. Follow the steps to export data from AWS DocumentDB in CSV format: Once you log into your AWS account and go to the AWS EC2 console, start the newly created AWS EC2 instance. Connect to the EC2 instance by clicking the Connect button or executing the following command. ssh -i /path/my-key-pair.pem my_ec2instance_user_name@my_ec2instance_public_dns_name A command prompt will open and check if MongoDB is successfully installed using the following command: mongod –version Navigate to a folder to store the exported CSV file by using the following command: cd path/to/folder Execute the following mongoexport command using the appropriate parameters. Replace your username and password with your login credentials. mongoexport --ssl \ --host="tutorialCluster.node.us-east-1.docdb.amazonaws.com:27017" \ --collection=<collection_name> \ --db=<database_name> \ --out=<outputfilename.csv> \ --username=<yourUsername> \ --password=<yourPassword> \ --sslCAFile global-bundle.pem Exit the AWS EC2 instance by simply typing exit in the command prompt. Run the following command to download the CSV file from the EC2 instance to your local machine: scp -i /path/my-key-pair.pem my_ec2instance_user_name@my_ec2instance_public_dns_name:/path/to/<filename.csv> /C:\Users\yourUsername\yourfolder Step 2: Load the CSV File into Snowflake Using a Named Internal Stage Ensure the following prerequisites are ready before you begin the import: An active Snowflake account Download and install SnowSQL from the Snowflake website. Here are the steps to load the CSV file into Snowflake: Log in to Snowflake using SnowSQL with your Snowflake account credentials: snowsql -a <account_id> -u <user_name> Create a named internal stage using the following SQL command: CREATE OR REPLACE STAGE my_internalstage file_format = (type = 'CSV' FIELD_DELIMITER = '|' SKIP_HEADER = 1); You can also create the name internal stage using Snowsight and Classic Console. Upload the CSV file on your local machine to your named internal stage using the PUT command: PUT file://C:\yourUsername\yourfolder\filename.csv @my_internalstage; Load your staged CSV data into the Snowflake table using the COPY INTO command: COPY INTO mydestinationtable from @my_internalstage; Limitations of AWS DocumentDB to Snowflake Migration Using CSV Export/Import Method Lack of Real-time analytics: If your organization relies on real-time analytics, the CSV Export/Import method would be inadequate because it cannot provide continuous, up-to-date insights into changing data trends. Limited Scalability: As the dataset size increases, the migration through CSV export/import can become time-consuming, resulting in delayed access to important data. Integrate Amazon DocumentDB to SnowflakeGet a DemoTry itIntegrate Amazon DocumentDB to RedshiftGet a DemoTry itIntegrate Amazon DocumentDB to BigQueryGet a DemoTry it Method 2: Sync AWS DocumentDB to Snowflake Using Hevo Data Hevo Data is a real-time ELT, no-code pipeline platform that helps you cost-effectively automate data pipelines that are flexible to your needs. Let’s see how to transfer data from AWS Document DB to Snowflake using Hevo Data. Step 1: Configure AWS DocumentDB as Your Source Before you begin, verify that the following prerequisites are in place: An Active Amazon Web Services (AWS) account. Amazon DocumentDB version 4.0 or above. Connect Amazon EC2 instance to an Amazon DocumentDB database. Whitelist Hevo’s IP addresses. Create a security group for the DocumentDB cluster in your AWS EC2 console. Create a cluster parameter group in your Amazon DocumentDB console. Create an Amazon DocumentDB cluster. Install mongo shell for your operating system. Create a user with the necessary privileges in your Amazon DocumentDB database. Enable Streams on the DocumentDB cluster. Update the Change Stream Log Retention Duration. You must be assigned a Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo. Here are the steps to configure the Amazon DocumentDB as your source in Hevo: Once you sign into your Hevo account, go to the Navigation Bar and click the PIPELINES option. In the Pipelines List View, click the + CREATE button. Choose Amazon DocumentDB as your source type on the Select Source Type page. Provide the necessary information in the Configure your Amazon DocumentDB Source page. Configuring Amazon DocumentDB Source Page Click TEST & CONTINUE. For more information about the source configuration, read the Amazon DocumentDB documentation in Hevo. Step 2: Configure Snowflake as Your Destination Before you start the configuration, check the given prerequisites are in place: An active Snowflake account or set up one. You must be assigned a Team Collaborator or an administrator role in Hevo, excluding the Billing Administrator role. Create and configure a new Hevo role in your Snowflake Destination using Hevo’s ready-to-use script. Obtain the URL of your Snowflake account. Follow the steps to configure the Snowflake as your destination in Hevo: In the Navigation Bar, choose the option DESTINATIONS. Go to the Destinations List View page and click the + CREATE button. Choose Snowflake as your destination type on the Add Destination page. Enter the required information on the Configure your Snowflake Destination page. Configuring your Snowflake Destination Page Click the TEST CONNECTION > SAVE & CONTINUE to complete the destination configuration. For more information about destination configuration, read the Snowflake documentation in Hevo. Use Cases of AWS DocumentDB to Snowflake Integration Backup and Recovery: Your organization can utilize Snowflake’s Time Travel feature to maintain regular backups of large volumes of operational data for up to 90 days. This feature allows you to quickly revert to previous versions of the data set or even recover dropped tables for business continuity. Support Stored Procedures: Snowflake supports stored procedures to meet the needs of analytical workloads. It also allows you to automate a database operation that needs different SQL statements and is performed frequently. Seamlessly Migrate from DocumentDB to Snowflake with Hevo Start For Free No credit card required Conclusion AWS DocumentDB to Snowflake migration offers several benefits for your organization’s data management and analytics. This article highlights two easy methods for migrating data from AWS Document DB to Snowflake, including Hevo Data and the CSV Export/Import method. The Hevo Data method can transfer historical and current data into Snowflake. The CSV Export/Import method is simpler but can migrate only historical data. Additionally, the manual CSV method is inadequate when dealing with large datasets. Choosing a real-time ELT pipeline platform like Hevo Data would allow your organization to manage large data volumes and access real-time data synchronization. Sign up for Hevo’s 14-day free trial and experience seamless data migration. Frequently Asked Questions (FAQs) Why would I choose Snowflake over deploying a database in the cloud and conducting in-house analytics? Some of the key reasons are:Snowflake offers advanced built-in distributed processing that enhances reliability and fault tolerance.You can deploy your Snowflake data lakes or data warehouses in any public cloud platform, such as GCP, Azure, or AWS.Snowflake isolates the compute layer from the storage layer. So, you can process analytical queries in the compute layer while storage operations continue independently. Does Snowflake integrate with AWS? Yes, Snowflake integrates seamlessly with AWS, allowing users to store, analyze, and share data using services like S3 for storage and Redshift for data warehousing. What are the advantages of Snowflake on AWS? Snowflake on AWS provides scalable storage, high-performance data processing, simplified management, strong security, and seamless integration with other AWS services like S3, Lambda, and Redshift for advanced analytics. Dimple M K Customer Experience Engineer, Hevo Data Dimple is an experienced Customer Experience Engineer with four years of industry proficiency, including the last two years at Hevo, where she has significantly refined customer experiences within the innovative data integration platform. She is skilled in computer science, databases, Java, and management. Dimple holds a B.Tech in Computer Science and excels in delivering exceptional consulting services. Her contributions have greatly enhanced customer satisfaction and operational efficiency. Liked the content? Share it with your connections. Share To LinkedIn Share To Facebook Share To X Copy Link Related Articles How to Move Data from AWS DocumentDB to BigQuery : 2 Methods Working With AWS Lambda Java Functions: 6 Easy Steps AWS RDS Oracle to Databricks: Strategic Data Migration Methods