With mobile marketing becoming a powerful source of company revenue, it’s highly likely that CleverTap is your firm’s choice for user engagement. That means you also need a data warehouse solution like BigQuery for deeper data analytics.
So the next question is how do you plan to migrate data from CleverTap to BigQuery? This article will bring you up to speed on your existing options and help you make an informed decision for your business.
CleverTap is a mobile marketing automation and customer engagement platform offering real-time analytics on customer behavior and omnichannel marketing opportunities across the entire customer life cycle. Its marketing management suite allows businesses to influence user behavior over mobile apps and encourages them to perform desired actions to achieve business outcomes. Over 8000 businesses like Vodafone, Sony, etc. trust CleverTap to expand their customer engagement.
The analytics data generated from CleverTap is often exported to a data warehouse, a hub for collecting data from disparate sources. A data warehouse such as BigQuery is the default choice of many enterprises that offer a means of giving a deeper insight into businesses.
BigQuery is the cloud-based data warehousing solution for enterprises from Google Cloud Platform (GCP). BigQuery, in simple terms, can load big data from various disparate sources, perform data analysis, and allow users to run SQL queries to offer meaningful business insights. Companies prefer BigQuery for its efficient data management, built-in machine learning capabilities, and easy integration with other GCP products.
You can learn more about Google BigQuery here.
Method 1: Custom Code Method to Migrate CleverTap Data to BigQuery
If you have access to or can recruit technical staff with the experience to work with APIs and BigQuery data warehouse, you can use this approach. At first, you have to log in to your GCP service account, allow access to CleverTap, and create a bucket. Then you can move on to creating a new data export and finally loading data from CleverTap to BigQuery.
Method 2: Using Hevo a No-code Data Pipeline
Hevo Data provides a hassle-free solution and helps you directly transfer data from CleverTap to BigQuery and numerous other Databases/Data Warehouses or destinations of your choice instantly without having to write any code. Hevo comes with a graphical interface that allows you to configure your CleverTap source and load data in real-time. Hevo is fully managed and completely automates the process of not only loading data from your desired source but also enriching the data and transforming it into an analysis-ready form without having to write a single line of code. Hevo’s pre-built with 150+ data sources (including 40+ free data sources) will take full charge of the data transfer process, allowing you to focus on key business activities.
Get Started with Hevo for Free
Methods to Migrate CleverTap Data to BigQuery
Following are the 2 popular methods to perform CleverTap to BigQuery data migration:
Download the Cheatsheet on How to Set Up High-performance ETL to BigQuery
Learn the best practices and considerations for setting up high-performance ETL to BigQuery
Method 1: Custom Code Method to Migrate CleverTap Data to BigQuery
Here are the basic building blocks of implementing a custom code to copy data from CleverTap to BigQuery.
- CleverTap account with a Corporate Pack activated (the data export feature is not available to lower plans).
- Required permissions to create and access Google Cloud Platform.
- A successful setup of BigQuery allowing admin privilege.
Step 1: Create a Service Account for your Project
The very first step is to log in to the GCP account and create a service account to authenticate and allow CleverTap to upload data exports to your GCP project. Create a new Service Account from “IAM & Admin”, give name and description, and enable storage admin permission. Now, you need to create a JSON key for the service account. Once you click on “Create Key” and select the type as JSON, the key will get auto-downloaded. You should save it for uploading to CleverTap later.
Step 2: Create A Bucket
Now, navigate to the ‘Storage’ option in the left panel and click on “Create Bucket.” Give a name and fill out various settings options to create the bucket.
Step 3: Add your GCP Service Credentials and Bucket Details to CleverTap
Log into CleverTap dashboard and click on the ‘Settings’ and follow the path: Partner > Exports > Google. Now, copy the contents of the downloaded Service Account Key JSON file and paste them into the Service Key. You need to add the name of the bucket as given in the project and click on Save Credentials. If you don’t receive the “credential saved” message, then regenerate the Service Account key again, and copy and paste it again.
Step 4: Create a New Data Export
Now, follow the path in CleverTap: Partner Data > Exports > Activity Log > Create Export. You need to select GCP as the export partner. Also, select the relevant export type, frequency, and format from the lists.
Step 5: Loading Data from GCP to BigQuery
Once the GCP setup is complete, data is available there. The next task is to set up BigQuery to load that data.
Limitations of using Custom Code
- This method involves many error-prone steps that require to be constantly monitored to ensure data is intact and safe at all times.
- While the multiple system setup needs some level of proficiency and understanding, maintenance requires a much deeper level of expertise. Hence, the entire process can become complex very quickly, asking for a substantial knowledge overhead and a lot of time.
- Cleaning, transforming, and optimizing data before loading to a data warehouse is a natural process for better efficiency. This is an added workload, developers must deal with.
- You must have CleverTap corporate pack for this method to work or be ready to build your custom ETL setup using CleverTap APIs. This means the complexity, know-how, and time needed increases manifold.
The alternative would be to let a platform like Hevo handle moving your data from CleverTap to BigQuery while you focus on your business growth.
Method 2: Using a No-code Data Pipeline
Hevo is a user-friendly no-code data pipeline solution that quickly adapts to its customer needs. It’s a simple and straightforward platform that requires minimal user engagement. It transfers data from CleverTap to BigQuery for free
- CleverTap admin right is required.
- You must have the BigQuery Admin and Storage Admin roles enabled.
Step 1: Configure and set up CleverTap as Data Source
CleverTap is an omnichannel platform used by companies to engage and retain their customers across industry domains such as travel, fin-tech, e-commerce, food-tech, and hospitality. You can load your CleverTap Events to a Destination system using Hevo Pipelines.
First, you need to configure your CleverTap account by filling up the following data fields.
Step 2: Set BigQuery as the Destination
With Hevo Pipelines, you can configure Google BigQuery as a Destination for loading data from any Source system. You now need to authenticate and connect to your BigQuery data warehouse.
That’s all! Hevo will auto-load your data from CleverTap to BigQuery in no time.
Advantages of Using Hevo
Hevo offers a lot of advantages over building your own solution to migrate data from CleverTap to Bigquery:
Sign up here for a 14-Day Free Trial!
- Minimal Setup – Hevo offers a hassle-free data loading experience, saving on the complexity and time involved in data transfers.
- Automatic Schema Detection and Mapping – Hevo monitors the schema of CleverTap data and automatically implements any change as noticed in BigQuery seamlessly.
- Fully Managed Solution – Hevo is a fully managed data solution, requiring zero maintenance or involvement from users.
- Zero Data Loss – Hevo’s fault-tolerant architecture ensures your data is 100% intact during the entire data migration process.
- Automatic Real-time Recurring Data Transfer – Hevo allows periodic data transfer in real-time with custom frequency and is only limited by CleverTap’s minimum 4 hours data export frequency and BigQuery’s 30 minutes waiting period for data loading policies.
- Added Integration Option – Hevo ensures that you get the similar power of moving data from other sources to BigQuery with ease. Hevo readily integrates with several databases, analytics platforms, sales and marketing applications, etc.
- Ability to Transform Data – Hevo allows you to transfer data both before and after moving it to the data warehouse. This ensures that you always have analysis-ready data in your BigQuery data warehouse.
- Committed Customer Support – Hevo team understands the value of helping the customers. They are 24X7 fully committed to supporting you over call, email, and chat.
The article explained to you the two methods of connecting CleverTap to BigQuery in a step-by-step manner. It also discussed the limitations of writing custom scripts to set up your ETL process. If you have no qualms about facing those limitations then you can try the manual ETL setup.
Hevo Data on the other hand can simplify your task by eliminating the need to write any code. It will automate the process of data transfer from CleverTap to BigQuery and provide you with a hassle-free experience. Hevo provides granular logs that allow you to monitor the health and flow of your data. It helps you transfer data from CleverTap to BigQuery for free.
Visit our Website to Explore Hevo
In addition to CleverTap, Hevo can load data from a multitude of 150+ other data sources including Databases, Cloud Applications, SDKs, and more. This allows you to scale up your data infrastructure on demand and start moving data from all the applications important for your business
Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also have a look at our unbeatable pricing that will help you choose the right plan for your business needs!
We would be thrilled to know what you think of the methods to transfer data from CleverTap to BigQuery detailed in this article. Let us know your thoughts and your preferred way of moving data in the comments section below.