Quickstart to Using BigQuery Python Client Library

Analyzing the data stored in On-Premise Data Warehouses can be a time-consuming task, especially if your data is being generated at an exponential rate and requires continuous scaling to ensure optimal performance.

This is one of the biggest reasons why companies today are adopting Cloud-based Data Storage solutions. One such solution is Google BigQuery which is one of the most used and widely accepted Data Warehousing solutions.

Organizations that use Google BigQuery to store large amounts of data often need to access the data through programming languages.

The typical requirement for this arises from the need to transform data into various forms suitable for consumption and to move data to other Databases for certain use cases.

Python is a very popular language choice for such requirements because of its data manipulation capabilities and ease of integration with data processing frameworks like Spark.

There is a BigQuery Python Client Library available for the users to query datasets in Google BigQuery in a seamless manner.

Upon a complete walkthrough of this article, you will gain a decent understanding of Google BigQuery along with the unique features that it offers.

You will also learn about the steps required to leverage Google BigQuery Python Client Library to query BigQuery Public Datasets using Python.

Read along to learn more about Google BigQuery Python Client Library!

Prerequisites

• Basic Hands-on experience with Google Cloud Console.
• Basic Knowledge of Python.

Table of Contents

• Introduction to Google BigQuery
• Introduction to Python
• Introduction to Cloud Client Libraries
• Steps to Follow Before using BigQuery Python Client Library
  • Step 1: Create a Cloud Platform Project
  • Step 2: Enable Billing for your Cloud Platform Project
  • Step 3: Enable the Google Cloud BigQuery API
  • Step 4: Set up Authentication
• Steps to Query Datasets using BigQuery Python Client Library
• Frequently Answered Questions
• Conclusion

Introduction to Google BigQuery

Google BigQuery is a fully managed Cloud-based Data Warehouse that enables you to manage and analyze your data with built-in capabilities such as Machine learning, Geospatial Analysis, and Business Intelligence.

Google BigQuery’s Serverless architecture allows you to use SQL queries to answer your company’s most critical questions without worrying about Infrastructure administration.

With Google BigQuery’s Scalable and Distributed Analytics Engine, you can query Terabytes and Petabytes of data in a span of just a few minutes.

Google BigQuery Interfaces include the Google Cloud Console and the Google BigQuery Command-Line tool.

Developers and Data Scientists can use Client libraries in Programming Languages such as Python, Java, JavaScript, and Go, as well as the REST APIs to transform and manage data.

You will see one such example(BigQuery Python Client Library) later in this article.

Introduction to Python

Python is a high-level, general-purpose programming language designed for Web Development, Software Development, Machine Learning, and more.

It emphasizes code readability and its Object-oriented approach helps programmers in writing logical codes for small and large-scale projects with ease.

It is one of the world’s fastest-growing programming languages used by Software Developers, Data Analysts, Scientists, Students, and Accountants. Python finds its use in various industries and companies from Artificial Intelligence to Data Science.

Introduction to Cloud Client Libraries

Cloud Client Libraries are the recommended way to programmatically access Google Cloud APIs. Cloud Client Libraries support access to Google Cloud Services in a way that greatly reduces the repetitive code you have to write.

These Libraries provide high-level abstraction which is why they are easy to understand. Cloud Client Libraries also work well with the standard libraries and integrate better with your codebase.

In the later section of this article, you will learn about the steps required to use BigQuery Python Client Library to query datasets in Google BigQuery.

Steps to Follow Before using BigQuery Python Client Library

There is a certain sequence of steps that you need to follow before using the Google BigQuery Python Client Library. They are as follows:

Step 1: Create a Cloud Platform Project

If you don’t have a pre-existing Cloud Platform Project in your Google Cloud Console then you have to create one before using Google BigQuery Python Client Library.

Follow the steps given below to create a new Cloud Platform Project.

• Sign In to the Google Cloud Console and click on the option Select a Project.
• Click on the option New Project to create a new project.
• Enter the name for your project and then click on Create.

Step 2: Enable Billing for your Cloud Platform Project

When you create a new Cloud Platform Project, you will be asked to choose which of your Cloud Billing accounts you would like to link to the project that you have created.

If you only have one Cloud Billing account, then that account is automatically linked to your project.

If you don’t have a Cloud Billing account, you’ll need to create one and turn on billing for your project before you can leverage the features of the BigQuery Python Client Library.

Follow the steps given below to create a new Cloud Billing account:

• Sign In to the Manage Billing Accounts page within the Google Cloud Console platform.

• Click on Create Account and enter the name for your Cloud Billing Account.
• Depending on the configuration, you might be asked to fill either of the following two fields- Organization or Country Name.
• Click on Continue.

Step 3: Enable the Google Cloud BigQuery API

For all Cloud Platform Projects, the Google BigQuery API is enabled by default. To see whether Google Cloud BigQuery API is enabled for your project, run the following command:

gcloud services list

If the API is not enabled, you can paste the following command in the Command Shell to enable it:

gcloud services enable bigquery.googleapis.com

Step 4: Set up Authentication

Google Cloud APIs support multiple authentication flows for different runtime environments. For the best developer experience, you can use the Google Cloud Client Libraries with Google Cloud APIs.

They use the authentication libraries that are provided by Google to support a variety of authentication flows and runtime environments.

You can follow the general steps given below to build an application using Google Cloud APIs and before using BigQuery Python Client Library:

• Select the Google Cloud Client Library for the language you are most comfortable using.
• Determine the correct authentication flow for the application you want to build.
• Now you need to create the application credentials needed for your application.
• Pass the application credentials to the Client Libraries when the application starts, ideally through Application Default Credentials (ADC).

Steps to Query Datasets using BigQuery Python Client Library

You can follow the steps given below to query datasets using Google BigQuery Python Client Library:

• Step 1: The first step in connecting Google BigQuery to any Programming Language is to configure the required dependencies. The first step is to install the BigQuery Python Client in a virtual environment using pip.virtualenv. pip.virtualenv is a tool that is used to create virtual Python environments.
  
  With a virtual environment, it’s possible to install the BigQuery Python Client Library without needing System Install Permissions, and without clashing with the installed system dependencies. Use the following code lines to install the BigQuery Python Client library:

Note: Google BigQuery Python Library supports Python Versions 3.5 and later.

For Windows Operating System

pip install virtualenv
virtualenv <your-env>
<your-env>Scriptsactivate
<your-env>Scriptspip.exe install google-cloud-bigquery

For Mac/Linux

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-cloud-bigquery

• Step 2: Once you have successfully installed the BigQuery Python Client Library, you will now use it to create a simple script to access the data from one of the Public Datasets available in Google BigQuery.
  
  To do so, you need to import Google BigQuery by using the following command.

from google.cloud import BigQuery

• Step 3: Initialize the client by using the following command.

Sample_client = BigQuery.Client()

• Step 4: Form the query as follows. The query will access a Public Dataset in BigQuery that has data about names in the USA and will return the data associated with the state ‘TX’.

QUERY = """
    SELECT 
    FROM `BigQuery-public-data.usa_names.usa_1910_2013`
    WHERE STATE=’TX’ LIMIT 100
"""

• Step 5: Once you have written the Query, it is time to execute the same. Query Execution in Google BigQuery is called a Job. Since Query Executions sometimes take longer than expected to execute, they are referred to with the term job. Initiate the job as follows:

results = Sample_client.query(QUERY)

• Step 6: The final step requires you to print the result using a loop.

for row in results:
    print(row.name)

Once you follow all the steps in the correct sequence, you will be able to query datasets using BigQuery Python Client Library.

Frequently Answered Questions

Here are a few Frequently Answered Questions about the BigQuery Python Client.

Do I need to know how to program to query datasets in Google BigQuery using Python Client?
Yes, you’ll need to have basic Python knowledge to query BigQuery Public Datasets using the BigQuery Python Client. Also, BigQuery uses Structured Query Language (SQL) to interact with Relational Databases. It is necessary to have a cursory knowledge of SQL when working with BigQuery.

Can I use my own Client Code?
Given that you’re an experienced Developer and Google’s Client Libraries don’t meet your specific requirements, you can always write your own custom code to access the service’s lower-level service APIs directly.

I don’t want to programmatically access Google Cloud APIs, are there any other ways to access Cloud APIs?
You can access some of the functionality of Google APIs using the tools in Google Cloud CLI or using the Google Cloud console.

Why am I experiencing a degraded performance (or an issue) while working with the BigQuery Python Client Library?
It is possible that you are using an older version of the Python Client Library, you can consider upgrading it to take advantage of all improvements in newer versions. To get the best possible support experience, it is recommended to stay up to date with the releases.

Apart from Python Client Library, what other languages are supported in Google Cloud Client Libraries?
Cloud Client Libraries support various programming languages such as Go, C#, PHP, Ruby, Python, C++, Java, etc.

Conclusion

This article introduced us to BigQuery, Python, and Cloud Client Libraries. It later took us through a step-by-step guide to leveraging BigQuery Python Client Library to query Google BigQuery datasets using Python.

We also learned about the points that should be considered before using BigQuery Python Client Library.

With your Data Warehouse, Google BigQuery, live and running, you’ll need to extract data from multiple platforms to carry out your analysis.

However, integrating and analyzing your data from a diverse set of data sources using custom ETL Scripts can be challenging and this is where Hevo Data comes into the picture.

It is quite clear that Google BigQuery allows users to input data for querying and analysis. BigQuery Java API Client also facilitates this process and allows users to work with complex datasets. 

We hope your experience of learning about the Google BigQuery Python Client Library was fruitful and would love to know your thoughts in the comments. Drop a note 😊