CleverTap to Redshift: 2 Easy Methods

Does your company use CleverTap to manage its Mobile Marketing Automation and User Engagement? Are you trying to move the data generated by CleverTap to Redshift for deeper analytics? If any of this applies to you, then you will find this blog helpful. This post will outline 2 methods of moving your data from CleverTap to Redshift and explain their benefits and limitations. 

This blog discusses both approaches, allowing you to critically evaluate the advantages and disadvantages of these approaches. It will provide a brief introduction of CleverTap and Redshift for the uninitiated. Read along to learn more about these 2 methods.

Prerequisites

• CleverTap account with a Corporate Pack activated (The data export feature is not available in lower plans).
• Required permissions to create and access Amazon S3.
• A successfully set up Amazon Redshift Data Warehouse.

What is CleverTap?

CleverTap is a Mobile Marketing Automation customer engagement platform offering real-time analytics on customer Behavioral and Omnichannel marketing opportunities across the entire customer lifecycle. Its marketing management suite allows businesses to influence user behavior over mobile apps and encourages them to perform desired actions to achieve business outcomes. Over 8000 businesses Vodafone, Sony, etc. trust CleverTap to expand their customer engagement.

While CleverTap provides detailed data points and analytics within their product, businesses would often want to export this data to a single source of truth – a Data Warehouse such as Amazon Redshift to derive deeper actionable insights. To learn more about CleverTap, visit their official documentation.

What is Amazon Redshift?

Amazon Redshift is a Cloud-based Data Warehouse service offered by the Amazon Web Services (AWS) platform as part of its Cloud Services. Built on flexible Massive Parallel Processing (MPP) architecture, it can perform advanced analytics on Big Data through SQL tools. Redshift commands the largest Cloud-based Data Warehouse deployments share thanks to its easy integration with various AWS services, fast performance, and cost-effectiveness. You can read more about Amazon Redshift in their official documentation.

Methods to Set up CleverTap to Redshift Integration

Method 1: Manual ETL Process to Set up CleverTap to Redshift Integration

Following are the steps of implementing a custom code to copy data from CleverTap to Redshift:

• Step 1: Create an AWS S3 Bucket
• Step 2: Create an API Key for the Bucket
• Step 3: Add the Bucket Details to CleverTap
• Step 4: Load Data from CleverTap to S3
• Step 5: Load Data from S3 to Redshift

Step 1: Create an AWS S3 Bucket

Search for S3 in the AWS service box in your AWS account and click the Bucket icon to create a Bucket. Next, configure the Bucket by providing the Bucket name and region (should be the same as where the CleverTap data is hosted). On the “Set Properties” tab choose your logging, versioning, and encryption preferences as per your internal organization’s policies. Make changes as per your need to the “Set Permission” tab and finally create the S3 bucket. This Bucket will appear in your S3 console.

Step 2: Create an API Key for the Bucket 

CleverTap uses this API key with write access for exporting data. Enter the Security Credentials option to add a username and select Programmatic Access as your access type. Next, add the user to a group, give a Group name. Then click on Create policy, go to the JSON tab and paste the following code – 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::name of the S3 bucket you created"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::name of the S3 bucket you created/*"
            ]
        }
    ]
}

This script allows CleverTap to get information about the Bucket you created and upload files to it. Now, assign the new policy to the newly created group and finally create the policy. Now you get your Access key ID and the Secret Access Key needed for the next step.

Step 3: Add the Bucket Details to CleverTap

Now in the CleverTap login page, go to Settings and give Bucket name, Access key ID, and the Secret Access Key and save it. 

Step 4: Create a New Data Export and Export Format

Next, follow the path – Partner Data -> Exports -> Activity Log -> Create Export to create an export path. Choose AWS as an Export Partner, Export Type, Export Frequency, and Export Format.

Step 5: Loading Data from S3 to Redshift

Once the data is available on S3, you can then choose to move this data to Amazon Redshift. You can get the detailed instructions here. 

Challenges of Building a Custom Setup

• This process needs you to put together multiple systems to build a seamless Data Pipeline. You would need to invest both time and expensive developer bandwidth to achieve this.
• Since the process involves multiple steps, it is quite brittle. This means you would need to constantly monitor the infrastructure to ensure that no data is lost due to any failure points.
• Many times, you would need to Clean, Transform and Enrich data before loading it to the Data Warehouse. Building this layer adds additional overhead on the developers and increases the overall timeline of the project. 
• The above approach is possible only if you have a corporate pack with CleverTap. If you are on a lower subscription plan, you will need to build a custom ETL set up by using CleverTap APIs. This adds more complexity to the process.

A much easier and hassle-free approach would be to load data from CleverTap to Redshift using a simple code-free platform such as Hevo. Read on to learn more about this approach. 

Method 2: Using Hevo Data to Set up CleverTap to Redshift Integration

You can connect from CleverTap to Redshift with Hevo in 2 following steps:

• Step 1: After logging into your Hevo account, click on +Create Pipelines, select CleverTab, configure your details, and click on Test & Continue.

• Step 2: Once your source is configured, you can select your Destination as Redshift. Fill out the required details and click on Save & Continue.

And Done! Your Pipeline is set.

Streamline your data operations with our resource on moving data from CleverTap to MySQL. Access straightforward instructions for optimizing your data management process.

Learn More About:

• Migrate Data from CleverTap to Snowflake

Conclusion

The article provided you an introduction to CleverTap and Amazon Redshift, explaining their individual importance in the current business context. It also explained the step-by-step process of setting up CleverTap to Redshift integration using 2 methods. The first method involving custom ETL codes has certain challenges that are mentioned in the article.

Hevo Data can simplify your task by eliminating the need to write any code. It will automate the data transfer process from CleverTap to Redshift for Free and provide a hassle-free experience.

FAQ on CleverTap to Redshift