CleverTap to BigQuery: 2 Easy Methods

With mobile marketing becoming a powerful source of company revenue, it’s highly likely that CleverTap is your firm’s choice for user engagement. That means you also need a data warehouse solution like BigQuery for deeper data analytics. 

So the next question is how do you plan to migrate data from CleverTap to BigQuery? This article will bring you up to speed on your existing options and help you make an informed decision for your business.

Understanding CleverTap

CleverTap is a mobile marketing automation and customer engagement platform offering real-time analytics on customer behavior and omnichannel marketing opportunities across the entire customer life cycle. Its marketing management suite allows businesses to influence user behavior over mobile apps and encourages them to perform desired actions to achieve business outcomes. Over 8000 businesses like Vodafone, Sony, etc. trust CleverTap to expand their customer engagement.

The analytics data generated from CleverTap is often exported to a data warehouse, a hub for collecting data from disparate sources. A data warehouse such as BigQuery is the default choice of many enterprises that offer a means of giving a deeper insight into businesses.

Understanding BigQuery

BigQuery is the cloud-based data warehousing solution for enterprises from Google Cloud Platform (GCP). BigQuery, in simple terms, can load big data from various disparate sources, perform data analysis, and allow users to run SQL queries to offer meaningful business insights. Companies prefer BigQuery for its efficient data management, built-in machine learning capabilities, and easy integration with other GCP products.

You can learn more about Google BigQuery here.

Methods to Migrate CleverTap Data to BigQuery

Following are the 2 popular methods to perform CleverTap to BigQuery data migration:

• Method 1: Custom Code Method to Migrate CleverTap Data to BigQuery
• Method 2: Using Hevo a No-code Data Pipeline

Method 1: Custom Code Method to Migrate CleverTap Data to BigQuery

Here are the basic building blocks of implementing a custom code to copy data from CleverTap to BigQuery.

• Prerequisites
• Step 1: Create a Service Account for your Project
• Step 2: Create a Bucket
• Step 3: Add your GCP Service Credentials and Bucket Details to CleverTap
• Step 4: Create a New Data Export
• Step 5: Loading Data from GCP to BigQuery

Prerequisites

1. CleverTap account with a Corporate Pack activated (the data export feature is not available to lower plans).
2. Required permissions to create and access Google Cloud Platform.
3. A successful setup of BigQuery allowing admin privilege.

Step 1: Create a Service Account for your Project

The very first step is to log in to the GCP account and create a service account to authenticate and allow CleverTap to upload data exports to your GCP project. Create a new Service Account from “IAM & Admin”, give name and description, and enable storage admin permission. Now, you need to create a JSON key for the service account. Once you click on “Create Key” and select the type as JSON, the key will get auto-downloaded. You should save it for uploading to CleverTap later.

Step 2: Create A Bucket

Now, navigate to the ‘Storage’ option in the left panel and click on “Create Bucket.” Give a name and fill out various settings options to create the bucket.

Step 3: Add your GCP Service Credentials and Bucket Details to CleverTap

Log into CleverTap dashboard and click on the ‘Settings’ and follow the path: Partner > Exports > Google. Now, copy the contents of the downloaded Service Account Key JSON file and paste them into the Service Key. You need to add the name of the bucket as given in the project and click on Save Credentials. If you don’t receive the “credential saved” message, then regenerate the Service Account key again, and copy and paste it again. 

Step 4: Create a New Data Export

Now, follow the path in CleverTap: Partner Data > Exports > Activity Log > Create Export. You need to select GCP as the export partner. Also, select the relevant export type, frequency, and format from the lists.

Step 5: Loading Data from GCP to BigQuery

Once the GCP setup is complete, data is available there. The next task is to set up BigQuery to load that data.

Limitations of using Custom Code

• This method involves many error-prone steps that require to be constantly monitored to ensure data is intact and safe at all times.
• While the multiple system setup needs some level of proficiency and understanding, maintenance requires a much deeper level of expertise. Hence, the entire process can become complex very quickly, asking for a substantial knowledge overhead and a lot of time. 
• Cleaning, transforming, and optimizing data before loading to a data warehouse is a natural process for better efficiency. This is an added workload, developers must deal with.
• You must have CleverTap corporate pack for this method to work or be ready to build your custom ETL setup using CleverTap APIs. This means the complexity, know-how, and time needed increases manifold.

The alternative would be to let a platform like Hevo handle moving your data from CleverTap to BigQuery while you focus on your business growth. 

Method 2: Using a No-code Data Pipeline

Hevo is a user-friendly no-code data pipeline solution that quickly adapts to its customer needs.  It’s a simple and straightforward platform that requires minimal user engagement. It transfers data from CleverTap to BigQuery for free

Prerequisites

1. CleverTap admin right is required.
2. You must have the BigQuery Admin and Storage Admin roles enabled.

Step 1: Configure and set up CleverTap as Data Source

CleverTap is an omnichannel platform used by companies to engage and retain their customers across industry domains such as travel, fin-tech, e-commerce, food-tech, and hospitality. You can load your CleverTap Events to a Destination system using Hevo Pipelines.

First, you need to configure your CleverTap account by filling up the following data fields.

Step 2: Set BigQuery as the Destination

With Hevo Pipelines, you can configure Google BigQuery as a Destination for loading data from any Source system. You now need to authenticate and connect to your BigQuery data warehouse.

That’s all! Hevo will auto-load your data from CleverTap to BigQuery in no time.

Advantages of Using Hevo

Hevo offers a lot of advantages over building your own solution to migrate data from CleverTap to Bigquery:

1. Minimal Setup – Hevo offers a hassle-free data loading experience, saving on the complexity and time involved in data transfers.
2. Automatic Schema Detection and Mapping – Hevo monitors the schema of CleverTap data and automatically implements any change as noticed in BigQuery seamlessly. 
3. Fully Managed Solution – Hevo is a fully managed data solution, requiring zero maintenance or involvement from users.
4. Zero Data Loss – Hevo’s fault-tolerant architecture ensures your data is 100% intact during the entire data migration process.
5. Automatic Real-time Recurring Data Transfer – Hevo allows periodic data transfer in real-time with custom frequency and is only limited by CleverTap’s minimum 4 hours data export frequency and BigQuery’s 30 minutes waiting period for data loading policies. 
6. Added Integration Option – Hevo ensures that you get the similar power of moving data from other sources to BigQuery with ease. Hevo readily integrates with several databases, analytics platforms, sales and marketing applications, etc.
7. Ability to Transform Data – Hevo allows you to transfer data both before and after moving it to the data warehouse. This ensures that you always have analysis-ready data in your BigQuery data warehouse.
8. Committed Customer Support – Hevo team understands the value of helping the customers. They are 24X7 fully committed to supporting you over call, email, and chat.

Sign up here for a 14-Day Free Trial!

Conclusion

The article explained to you the two methods of connecting CleverTap to BigQuery in a step-by-step manner. It also discussed the limitations of writing custom scripts to set up your ETL process. If you have no qualms about facing those limitations then you can try the manual ETL setup.

Hevo Data on the other hand can simplify your task by eliminating the need to write any code. It will automate the process of data transfer from CleverTap to BigQuery and provide you with a hassle-free experience. Hevo provides granular logs that allow you to monitor the health and flow of your data. It helps you transfer data from CleverTap to BigQuery for free.

Visit our Website to Explore Hevo

In addition to CleverTap, Hevo can load data from a multitude of 150+ other data sources including Databases, Cloud Applications, SDKs, and more. This allows you to scale up your data infrastructure on demand and start moving data from all the applications important for your business

Want to take Hevo for a spin? Sign Up here for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also have a look at our unbeatable pricing that will help you choose the right plan for your business needs!

We would be thrilled to know what you think of the methods to transfer data from CleverTap to BigQuery detailed in this article. Let us know your thoughts and your preferred way of moving data in the comments section below.