How to Integrate MongoDB to Databricks: 2 Easy Ways

Unlock the full potential of your MongoDB data by integrating it seamlessly with Databricks. With Hevo’s automated pipeline, get data flowing effortlessly—watch our 1-minute demo below to see it in action!

Replicating your data from MongoDB to Databricks is vital for companies that could benefit from features such as historical tracking, offloading analytics, and normalizing schemas. However, integrating these tools could be confusing for some users who are new to creating pipelines or don’t have coding knowledge. Therefore, in this blog, we have provided you with 2 easy step-by-step guides on how to integrate your MongoDB data to Databricks.

Use Cases for Migrating Your Data from MongoDB to Databricks?

• Provides integration of transactional data from different functional groups (Sales, marketing, product, Human Resources) and finding answers
• Enables aggregation of the data of individual interactions of the product for any event. 
• Maps the customer journey within the product.
• Provides both rela-time and and scheduled data processing for real-time insights.
• Provides insights into tresnd patterns such as user retention and features usage.

Methods to Replicate Data from MongoDB to Databricks

Method 1: Using MongoDB Connector
Method 2: Using a No-Code Tool

Method 1: Replicate Data from MongoDB to Databricks Using MongoDB Connector

Step 1: Create a Databricks cluster, go to the Libraries tab, click “Install New,” select “Maven” as the source, and add the appropriate MongoDB Spark connector based on your runtime version:

• Use mongo-spark-connector_2.12:3.0.0 for runtime 7.0.0+
• Use mongo-spark-connector_2.11:2.3.4 for runtime 5.5LTS or 6.x

Step 2: Create a MongoDB Atlas cluster, add a database user with the required permissions, whitelist your Databricks cluster IPs, and upload your data to the Atlas instance.

Step 3: In Databricks, configure your cluster with the MongoDB connection URI. Get the URI from MongoDB Atlas by clicking “Connect” → “Connect Your Application,” choose the Scala driver (v2.2+), and copy the connection string. Then, in your Databricks cluster under “Configuration” > “Advanced Options” > “Spark,” add:

spark.mongodb.input.uri <connection-string>  
spark.mongodb.output.uri <connection-string>

Step 4: To read data from MongoDB into Databricks using the Spark Connector, use the options map like this:

import com.mongodb.spark._

val cars = spark.read.format("com.mongodb.spark.sql.DefaultSource")
  .option("database", "manufacturer")
  .option("collection", "cars")
  .load()

MongoDB’s flexible schema lets you store varied structures, but Spark needs a schema—so it samples documents to infer one. To skip sampling, define your schema explicitly using a case class:

case class Car(carId: Int, carRating: Double, timestamp: Long)

import spark.implicits._
val carsDS = cars.as[Car]
carsDS.cache()
carsDS.show()

Using the MongoDB connector is a great way to replicate data from MongoDB to Databricks effectively. It is optimal for the following scenarios:

• Access to advanced functional programming concepts like higher-kind types, path-dependent types, type classes, currying, multiple parameter lists, etc.
• Data workflows can be automated with connector-like solutions by employing customized scripts with detailed instructions on completing each workflow stage, like MongoDB connector in this scenario. These scripts can be executed by anyone proficient in the chosen programming language.

In the following scenarios, using the MongoDB connector might be cumbersome and not a wise choice:

• Pipeline Management: Managing data pipelines might result in costly expenses across several environments (development, staging, production, etc.). A pipeline needs to be maintained regularly, the settings need to be updated, and data sync should be achieved.
• Time Consuming:  If you plan to export your data frequently, creating instances and clusters, writing custom queries, and mapping and uploading the data with the connector method might not be the best choice since it takes time to carry out these processes. 

Method 2: Automate the Data Replication process using a No-Code Tool

Step 1: Configure MongoDB as a Source

Step 2: Configure Databricks as a Destination

Click “Save and Continue” to run the pipeline. With only 2 simple steps we have migrated our data from MongoDB to Databricks.

Additional Resources for MongoDB Integrations and Migrations

• Stream data from mongoDB Atlas to BigQuery
• Move Data from MongoDB to MySQL
• Connect MongoDB to Snowflake
• Connect MongoDB to Tableau
• Sync Data from MongoDB to PostgreSQL
• Move Data from MongoDB to Redshift

Conclusion

MongoDB connector is the right path for you when your team needs data from MongoDB once in a while. However, a custom ETL solution becomes necessary for the increasing data demands of your product or marketing channel. You can free your engineering bandwidth from these repetitive & resource-intensive tasks by selecting Hevo’s 150+ plug-and-play integrations.

Saving countless hours of manual data cleaning & standardizing, Hevo’s pre-load data transformations get it done in minutes via a simple drag n drop interface or your custom python scripts. No need to go to your data warehouse for post-load transformations. You can simply run complex SQL transformations from the comfort of Hevo’s interface and get your data in the final analysis-ready form. 

Sign up for a 14-day free trial with Hevo and streamline your data integration. Also, check out Hevo’s pricing page for a better understanding of the plans.

FAQ on MongoDB to Databricks