CleverTap to Redshift: 2 Easy Methods

Does your company use CleverTap to manage its Mobile Marketing Automation and User Engagement? Are you trying to move the data generated by CleverTap to Redshift for deeper analytics? If any of this applies to you, then you will find this blog helpful. This post will outline 2 methods of moving your data from CleverTap to Redshift and explain their benefits and limitations. 

This blog discusses both approaches, allowing you to critically evaluate the advantages and disadvantages of these approaches. It will provide a brief introduction of CleverTap and Redshift for the uninitiated. Read along to learn more about these 2 methods.

Prerequisites

• CleverTap account with a Corporate Pack activated (The data export feature is not available in lower plans).
• Required permissions to create and access Amazon S3.
• A successfully set up Amazon Redshift Data Warehouse.

What is CleverTap?

CleverTap is a Mobile Marketing Automation customer engagement platform offering real-time analytics on customer Behavioral and Omnichannel marketing opportunities across the entire customer lifecycle. Its marketing management suite allows businesses to influence user behavior over mobile apps and encourages them to perform desired actions to achieve business outcomes. Over 8000 businesses Vodafone, Sony, etc. trust CleverTap to expand their customer engagement.

While CleverTap provides detailed data points and analytics within their product, businesses would often want to export this data to a single source of truth – a Data Warehouse such as Amazon Redshift to derive deeper actionable insights.

To learn more about CleverTap, visit here.

What is Amazon Redshift?

Amazon Redshift is a Cloud-based Data Warehouse service offered by the Amazon Web Services (AWS) platform as part of its Cloud Services. Built on flexible Massive Parallel Processing (MPP) architecture, it can perform advanced analytics on Big Data through SQL tools. Redshift commands the largest Cloud-based Data Warehouse deployments share thanks to its easy integration with various AWS services, fast performance, and cost-effectiveness.

You can read more about Amazon Redshift here.

Methods to Set up CleverTap to Redshift Integration

You can implement the following two methods to set up your CleverTap to Redshift Integration:

• Method 1: Manual ETL Process to Set up CleverTap to Redshift Integration
• Method 2:Using Hevo Data to Set up CleverTap to Redshift

Method 1: Manual ETL Process to Set up CleverTap to Redshift Integration

Following are the steps of implementing a custom code to copy data from CleverTap to Redshift:

• Step 1: Create an AWS S3 Bucket
• Step 2: Create an API Key for the Bucket
• Step 3: Add the Bucket Details to CleverTap
• Step 4: Load Data from CleverTap to S3
• Step 5: Load Data from S3 to Redshift

Step 1: Create an AWS S3 Bucket

Search for S3 in the AWS service box in your AWS account and click the Bucket icon to create a Bucket. Next, configure the Bucket by providing the Bucket name and region (should be the same as where the CleverTap data is hosted). On the “Set Properties” tab choose your logging, versioning, and encryption preferences as per your internal organization’s policies. Make changes as per your need to the “Set Permission” tab and finally create the S3 bucket. This Bucket will appear in your S3 console.

Step 2: Create an API Key for the Bucket 

CleverTap uses this API key with write access for exporting data. Enter the Security Credentials option to add a username and select Programmatic Access as your access type. Next, add the user to a group, give a Group name. Then click on Create policy, go to the JSON tab and paste the following code – 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::name of the S3 bucket you created"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::name of the S3 bucket you created/*"
            ]
        }
    ]
}

This script allows CleverTap to get information about the Bucket you created and upload files to it. Now, assign the new policy to the newly created group and finally create the policy. Now you get your Access key ID and the Secret Access Key needed for the next step.

Step 3: Add the Bucket Details to CleverTap

Now in the CleverTap login page, go to Settings and give Bucket name, Access key ID, and the Secret Access Key and save it. 

Step 4: Create a New Data Export and Export Format

Next, follow the path – Partner Data -> Exports -> Activity Log -> Create Export to create an export path. Choose AWS as an Export Partner, Export Type, Export Frequency, and Export Format.

Step 5: Loading Data from S3 to Redshift

Once the data is available on S3, you can then choose to move this data to Amazon Redshift. You can get the detailed instructions here. 

Challenges of Building a Custom Setup

• This process needs you to put together multiple systems to build a seamless Data Pipeline. You would need to invest both time and expensive developer bandwidth to achieve this.
• Since the process involves multiple steps, it is quite brittle. This means you would need to constantly monitor the infrastructure to ensure that no data is lost due to any failure points.
• Many times, you would need to Clean, Transform and Enrich data before loading it to the Data Warehouse. Building this layer adds additional overhead on the developers and increases the overall timeline of the project. 
• The above approach is possible only if you have a corporate pack with CleverTap. If you are on a lower subscription plan, you will need to build a custom ETL set up by using CleverTap APIs. This adds more complexity to the process.
  

A much easier and hassle-free approach would be to load data from CleverTap to Redshift using a simple code-free platform such as Hevo. Read on to learn more about this approach. 

Method 2: Using Hevo Data to Set up CleverTap to Redshift Integration

Hevo Data, a No-code Data Pipeline, helps you directly transfer data from CleverTap (a Free Source) and 150+ other data sources to Data Warehouses such as Redshift, Databases, BI tools, or a destination of your choice in a completely hassle-free & automated manner. Hevo is fully managed and completely automates the process of not only loading data from your desired source but also enriching the data and transforming it into an analysis-ready form without having to write a single line of code. Its fault-tolerant architecture ensures that the data is handled in a secure, consistent manner with zero data loss.

Hevo Data takes care of all your Data Preprocessing needs and lets you focus on key business activities and draw a much more powerful insight on how to generate more leads, retain customers, and take your business to new heights of profitability. It provides a consistent & reliable solution to manage data in real-time and always have analysis-ready data in your desired destination. 

Setting up CleverTap to Redshift data export using Hevo is a straightforward 2 step process:

• Configure CleverTap as Data Source:
  First, you need to configure your CleverTap account by providing the necessary inputs. 

• Configure Redshift as the Destination:
  The next step is to authenticate and connect to your Redshift data warehouse as below.

Yes! That is all. You can now sit back and watch your data load from CleverTap to Redshift. 

Advantages of Using Hevo

There are a couple of reasons why you should opt for Hevo over building your own solution to migrate data from CleverTap to Redshift.

• Minimal Setup – The complexity, as well as the time spent on setup on Hevo, is much less compared to building the setup yourself, giving you a hassle-free experience.
• Automatic Schema Detection and Mapping: Hevo scans the schema of incoming CleverTap automatically. In case of any change, Hevo seamlessly incorporates the change in Redshift. 
• Fully Managed Solution: Hevo is a completely managed data solution, relieving users from any maintenance aspect. 
• Automatic Real-time Recurring Data Transfer: CleverTap allows you to set up automatic recurring data exports, with a minimum frequency of every 4 hours (max up to every 24 hours). Whereas with Hevo, data is recurrently transferred to Redshift in real-time as well as daily or weekly, just the way you need.
• No Data Loss: The fault-tolerant architecture of Hevo makes sure that there is zero loss of your precious data while moving from CleverTap to Redshift.
• Added Integration Option – Hevo ensures that you get the similar power of moving data from other sources to Redshift with ease. Hevo natively integrates with several Databases, Sales and Marketing Applications, Analytics platforms, etc. 
• Strong Customer Support – Hevo team guarantees you have 24×7 support over Call, E-Mail, and Chat.
• Ability to Transform Data – Hevo allows you to transfer data both before and after moving it to the Data Warehouse. This ensures that you always have analysis-ready data in your Redshift Data Warehouse.

Simplify your Data Analysis with Hevo today! 

SIGN UP HERE FOR A 14-DAY FREE TRIAL!

Conclusion

The article provided you an introduction to CleverTap and Amazon Redshift, explaining their individual importance in the current business context. It also explained the step-by-step process of setting up CleverTap to Redshift integration using 2 methods. The first method involving custom ETL codes has certain challenges that are mentioned in the article.

VISIT OUR WEBSITE TO EXPLORE HEVO

Hevo Data can simplify your task by eliminating the need to write any code. It will automate the process of data transfer from CleverTap to Redshift for Free and provide you a hassle-free experience.

The ease of use, speed, reliability and excellent service – Hevo offers all-around performance.

SIGN UP and experience the power of Hevo first hand. 

Share your experience with moving data from CleverTap to Redshift in the comments section below!