AppsFlyer to Redshift: Move Data in Minutes

on Tutorial • February 20th, 2020 • Write for Hevo

Whether you are looking to load data from AppsFlyer to Redshift for in-depth analysis or you are looking to simply backup Appsflyer data to Redshift, this post can help you out. This blog highlights the steps and broad approaches required to load data from Appsflyer to Redshift. Before we dive in, let us understand these applications in brief.

Introduction to Appsflyer 

Appsflyer is an attribution platform for mobile app marketers. It helps businesses understand the source of traffic and measure advertising. It provides a dashboard that analyses the users’ engagement with the app. That is, which users engage with the app, how they engage and the revenue they generate.

Introduction to Redshift 

AWS Redshift is a data warehouse built using MPP (massively parallel processing) architecture. It forms part of the AWS cloud computing platform and its owned and maintained by AWS. It has the ability to handle large volumes of data sets and huge analytical workloads. The data is stored in a column-oriented DBMS principle which makes it different from other databases offered by Amazon.  

Using Redshift SQL, you can query megabytes of structured or unstructured data and save the results in your S3 data lake using Apache Parquet format. This helps you to do further analysis using Amazon SageMaker, Amazon EMR, and Amazon Athena. Read more about Redshift 

Methods to Load Data from Appsflyer to Redshift

While there are many approaches to move data from AppsFlyer to Redshift, this blog talks about two popular methods listed below:

Method 1: Building Custom ETL Scripts

This approach would be a good way to go if you have decent engineering bandwidth allocated to the project. The broad steps would involve – Understanding the AppsFlyer data export APIs, building code to bring data out of Appsflyer, and loading data into Redshift. Once set up, this infrastructure would also need to be monitored and maintained for accurate data to be available all the time.

Method 2: Use Hevo Data, a Fully-Managed Data Integration Platform

Hevo comes with out-of-the-box integration with AppsFlyer and loads data to Redshift without having to write any code. Hevo’s ability to reliably load data in real-time combined with its ease of use makes it a great alternative to Method 1.

This blog outlines both of the above approaches. Thus, you will be able to analyze the pros and cons of each when deciding on a direction as per your use case.  

Appsflyer to Redshift: Loading Data Using Custom Code 

Step 1: Getting data from Appsflyer

AppsFlyer supports a wide array of APIs that allow you to pull different data points both in raw (impressions, clicks, installs, etc.) and aggregated (aggregated impressions, clicks, or filtering by Media source, country, etc.) format. You can read more about them here. Before jumping on to implementing an API call, you would first need to understand the exact use case that you are catering to. Basis that, you will need to choose the API to implement. 

Note that certain APIs would only be available to you based on your current plan with AppsFlyer.

For the scope of this blog, let us bring in data from PULL APIs. PULL APIs essentially allow the customers of AppsFlyer to get a CSV download of raw and aggregate data. You can read more about the PULL APIs here.

In order to bring data, you would need to make an API call describing the data points you need to be returned. The API call must include the authorization key of the user, as well as the date range for which the data needs to be extracted. More parameters might be added to request information like currency, source, and other specific fields. 

A sample PULL API call would look like this: 

https://hq.appsflyer.com/export/master_report/v4?api_token=[api_token] 

&app_id=[app_id]&from=[from_date]&to=[to_date]&groupings=[list]&kpis=[list] 

As a response, a CSV file is returned from each successful API query. Next, you would need to import this data into Redshift. 

Step 2: Loading Data into Redshift 

As a first step, identify the columns you want to insert and use the CREATE TABLE Redshift command to create a table. All the CSV data will be stored in this table.

Loading data with INSERT command is not the right choice because it inserts data row by row. Therefore, you would need to load data to Amazon S3 and use copy command to load it into Redshift.

In case you need this process to be done on a regular basis, cron job should be set up to run with reasonable frequency, ensuring that the AppsFlyer data in Redshift data warehouse stays up to date.

Appsflyer to Redshift Using Custom Code: Limitations and challenges 

  1. Accessing Appsflyer Data in Real-time:  After you’ve successfully created a program that loads data to your warehouse, you will need to deal with the challenge of loading new or updated data. Replicating the data in real-time when a new or updated record is created slows the operation because it’s resource-intensive. To get new and updated data as it appears in the Appsflyer, you will need to write additional code and build cron jobs to run this in a continuous loop.
  2. Infrastructure Maintenance: When moving data from AppsFlyer to Redshift, many things can go wrong. For example, AppsFlyer may update the APIs or sometimes the Redshift data warehouse might be unavailable. These issues can cause the data flow to stop, resulting in severe data loss. Hence, we would need to have a team that can continuously monitor and maintain the infrastructure. 

Easier Alternative to Load Data from Appsflyer to Redshift:

Using a Data Integration platform like Hevo (14-day free trial) to load data from Appsflyer to Redshift is easier, elegant, reliable, and fast. 

Hevo overcomes all the limitations mentioned. You move data in just two steps, no coding required.  

  • Authenticate and Connect Appsflyer Data Source

AppsFlyer to Redshift - Connecting AppsFlyer Source

  • Configure the Redshift Data warehouse where you want to load the data

AppsFlyer to Redshift - Connecting Destination

The Hevo Advantage:

Hevo platform allows you to seamlessly move data from AppsFlyer to Redshift. Here are some more advantages:

  1. Minimal Setup – You will need minimal effort and bandwidth to set up the platform because Hevo is fully managed. 
  2. No Data Loss – The data is moved from Appsflyer to Redshift by the Hevo’s fault-tolerant architecture without data loss. 
  3. 100’s of Out of the Box Integrations – In addition to Appsflyer, Hevo can bring data from Cloud Applications, Databases, SDKs, and so on into Redshift in just a few clicks. So, you will always have a reliable partner to cater for your growing data needs.  
  4. Automatic schema detection and mapping – Hevo scans the schema of incoming Appsflyer data automatically. When changes are detected, it handles this seamlessly by incorporating this change on Redshift. 
  5. Exceptional Support – The 24×7 support provided by Hevo ensures that you always have Technical support. Hevo provides 24/7 support over email and Slack. 

What are your thoughts about moving data from AppsFlyer to Redshift? Let us know in the comments. 

No-code Data Pipeline for Redshift