Google Analytics to BigQuery ETL: Steps to Move Data in Minutes

on Tutorial • June 25th, 2019 • Write for Hevo

Are you trying to move your data from Google Analytics to BigQuery? Are you confused how to do this easily? If yes, then you are in the right place. This blog covers various methods to move data from Google Analytics to BigQuery in a few simple steps.

Let’s see how this blog is structured for you:

  1. What is Google Analytics?
  2. What is Google BigQuery?
  3. Pricing and Billing
  4. Method 1: Export Google Analytics to BigQuery Using BigQuery Data Transfer Service
  5. Potential Issues with Google Analytics ETL to Bigquery
  6. Method 2: Migrating Data from Google Analytics to BigQuery Using Hevo

What is Google Analytics?

Google Analytics is one of the most widely used services to track website traffic, individual customer session data, purchasing information, along with other metrics. The vast amounts of data produced by Google Analytics present an excellent opportunity for extracting useful and actionable business insights. However, as is, Google Analytics does not present an ideal format for analyzing all the data that it produces, it becomes necessary to move data from the application to a Data warehouse, i.e move data from Google Analytics to BigQuery. This, in turn, enables deeper analysis. 

What is Google BigQuery?

BigQuery is a data warehouse solution provided by Google that provides for very fast SQL-like queries of those extremely large datasets. The speed and efficiency of queries are due to the use of Google’s vast infrastructure and proprietary technology, such as the Dremel query engine. In this post, you will learn how to export Google Analytics data to BigQuery. 

Pricing and Billing

BigQuery charges for data storage, streaming inserts, and querying data but loading and exporting are free of charge. Know more about pricing here.

You need to a proper form of payment on file in Cloud for export to commence. If your exported is interrupted due to an invalid payment, you will not be able re-export at that time.

How to Export Data from Google Analytics to BigQuery?

  • Method 1: Export Google Analytics to BigQuery Using BigQuery Data Transfer Service
    The BigQuery Data Transfer Service is Google’s native intra-product data pipeline service. It automates the loading of data into BigQuery. The service works exclusively for migrating data from a number of Google services such as Google Analytics 360 and Google Ad Manager to BigQuery.
  • Method 2: Migrating Data from Google Analytics to BigQuery Using Hevo
    Hevo Data comes with pre-built integrations for both Google Analytics and BigQuery. This means you get easy and seamless integration between Google Analytics and BigQuery right out of the box. With a few simple clicks, a sturdy data export setup can be created between the applications. Hevo is a fully managed platform so that means that no coding or maintenance will be needed from your end. Hevo will handle the groundwork while your analysts can work with BigQuery to answer the big questions.

Let’s go through these options.

Method 1: Export Google Analytics to BigQuery Using BigQuery Data Transfer Service

To begin with, Google Analytics 360 and BigQuery are both Google products. That being the case, it is not surprising that there is also a pre-existing data migration solution for getting information from one product to the other. This is known as the BigQuery Data Transfer Service. However, this solution is not the simplest, nor the easiest to implement. 

Steps to Export Google Analytics Data to BigQuery

  1. Step 1: Create Google APIs Console Project
  2. Step 2: Enable BigQuery within the Project
  3. Step 3: Setup Billing for the Project
  4. Step 4: Add the Service Account
  5. Step 5: Link BigQuery to Google Analytics 360

Step 1: Create a Google APIs Console Project

In order to use the BigQuery Data Transfer Service for migrating data, you must first set up BigQuery Export. This requires logging into the Google APIs Console and either creating a new project or selecting an existing one.

Step 2: Enable BigQuery within the Project

Edit the API Library settings of the chosen project to enable the BigQuery API.

Step 3: Setup Billing for the Project

Ensure that a billing account has been set up. This is required to start using the service for data replication. Billing may be validated by creating a data set in your BigQuery project. If the data set is created, with no errors, then billing has been set up correctly. 

In case you want to try this set up without having to input your billing details, you could also try to set up a BigQuery Sandbox.

Step 4: Add the Service Account to the Project

The Google Analytics service account, analytics-processing-dev@system.gserviceaccount.com, must be added as a member of the project, with project-level permission set to Editor. This is required to allow Google Analytics to export data to BigQuery.

Step 5: Link BigQuery to Google Analytics 360

  • You would need to have an email address that has:
    (a) EDIT access to Google Analytics Property.
    (b) OWNER access to the BigQuery project. 
  • In the Admin panel → Property, link BigQuery by entering your project Id.

Once completed, data transfer will commence within 24 hours. 

Potential Issues with Google Analytics ETL to Bigquery

Despite the native status of the BigQuery Data Transfer Service and the BigQuery export, this solution has its drawbacks.

1. Limited Updates Set by Google

When using Google’s data streaming option the number of updates to the data warehouse is determined and set by Google. If you are looking to get data streaming in real-time, this could be a limitation.

2. Pipeline Maintenance

Exporting via the BigQuery Data Transfer Service requires that several key components be completed and maintained.

  1. The Google Analytics service account must always have EDIT access to the project. If for whatever reason this is changed then all proceeding exports will fail until permission has been restored.
  2. If the BigQuery API was somehow disabled in the project settings then the exports will fail.

Failure in any of these instances would result in the most serious consequences:

  • Irretrievable Data Loss: The greatest drawback of using the BigQuery Data Transfer Service is the risk of data loss. If any of the previously mentioned issues were to occur, or anything else that may cause an interruption in the data export, then that data would be lost, permanently. The BigQuery Data Transfer Service has no facilities to mitigate data loss as a result of a failed export.
  • Limited Scope and Usability: Another major issue with using the BigQuery Data Transfer Service is its limitation regarding integration. It cannot be used to integrate platforms and services outside of Google. In fact, there are even some Google services, such as Google Drive, that BigQuery does not integrate with.
  • Limited Scope for Transforming Data: Let’s say you want to change the time from PST to UTC while moving the Google Analytics data to BigQuery, or you do not want to move data of specific campaigns. With BigQuery Data Transfer Service you will not be able to achieve these simple modifications too.

Method 2: Migrating Data from Google Analytics to BigQuery Using Hevo

Hevo, a No-code Data Pipeline can be set up to export Google Analytics data to BigQuery without using BigQuery Export in 2 simple steps:

  • Connect the data source by authenticating Google Analytics:
Google Analytics to BigQuery: A screenshot of configuring source for google analytics data export in Hevo..
  • Configure the BigQuery warehouse where you want to move your Google Analytics data to:
Google Analytics to BigQuery: A screenshot of configuring Bigquery as destination in Hevo.

The resulting Hevo data pipeline will now reliably move Google Analytics data to the BigQuery warehouse for further analysis. Sign up for a 14-day free trial to try this first hand. Meanwhile, you can get a complete product overview by watching the following video:

Advantages of Using Hevo 

The Hevo data integration platform lets you move data from Google Analytics to BigQuery seamlessly. Here are some other advantages:

  • Zero Data Loss – Hevo’s unique fault-tolerant architecture is built to ensure the completeness of data. Hevo ensures that data is reliably moved without data loss. 
  • Unlimited Integrations – Hevo can connect to any source and any destination. All while providing a common interface.
  • Low time to Implementation – Once the simple setup procedure is complete, Hevo can migrate data from in no time.
  • Automatic Schema Detection, Mapping, and Evolution – Hevo analyses the schema of the data it receives for replication and automatically maps said data seamlessly onto the BigQuery table structure.
  • Fully Managed – The Hevo platform is fully managed and works out of the box. This will let you focus on extracting insights from your data and not worry about data availability.
  • Alerts and Notification – If any issues occur in the data replication Hevo automatically notifies the relevant stakeholders of your team with real-time alerts via email or Slack, allowing them to take timely action.
  • Scalability – Hevo is built to handle data of any scale. With Hevo, your business can grow without any data hiccups. 
  • Exceptional Support – Technical support for Hevo is provided on a 24/7 basis over both email and Slack.

Effective business intelligence requires accurate and up-to-date data. Hevo ensures that your access to this data is never compromised. Hevo ensures that the data is accurately moved in real-time. 

What’s more? Hevo integrates with a wide array of data sources such as Cloud Applications,  Cloud Storage, Databases, and more (www.hevodata.com/integrations), opening the door for a wide range of future possibilities that your business may need.

Sign up for a 14-day free trial here and experience efficient and effective data export from Google Analytics to BigQuery.

No-code Data Pipeline for BigQuery