What is AWS Glue?: 4 Comprehensive Aspects

With the vast sea of information that is growing every day, most organizations are looking toward Cloud-based solutions to collect, store and work on their data. Furthermore, the ETL process is necessary to convert the data collected from numerous sources into a common format that is accepted by the Data Warehouse. AWS Glue is one such ETL tool.

Are you looking to learn more about Amazon Glue and its fitment for your ETL needs? If yes, you have landed on the right post. This blog helps you decode some of the key aspects to consider while evaluating this tool (or any ETL solution for that matter): Glue’s Features, Pricing, Use Cases, and Limitations. Before we dive in, let us try to understand a little about the ETL process itself.

AWS Glue Overview

AWS Glue is a fully managed service provided by Amazon for deploying ETL jobs. It reduces the cost, lowers the complexity, and decreases the time spent creating AWS ETL jobs. For companies that are price-sensitive but need a tool that can work with different ETL use cases, Amazon Glue might be a decent choice to consider.

Launched sometime around August 2017, Glue AWS has come a long way in adding value to its users. Here are some of the noteworthy pointers about Glue:

• Glue is a “Serverless” service. Hence, you do not need to provision or manage any resources or services.
• You only pay for resources when Glue is actively running. 
• Amazon Glue ETL comes with “crawlers” that can create metadata to view the data stored in S3. This metadata comes very handy while authoring ETL jobs.
• With the use of Python scripts, Glue can translate one source format to another source format.
• Glue allows you to create a development endpoint by yourself. This gives you the power to construct your ETL scripts swiftly and easily.

What are the Features of AWS Glue?

Since Amazon Glue is completely managed by AWS, deployment, and maintenance are super simple. Below are some important features of Glue: 

• Integrated Data Catalog
• Automatic Schema Discovery
• Code Generation
• Developer Endpoints
• Flexible Job Scheduler

1) Integrated Data Catalog

The Data Catalog is a persistent metadata store for all kinds of data assets in your AWS account. Your AWS account will have one Glue Data Catalog. This is the place where multiple disjoint systems can store their metadata. In turn, they can also use this metadata to query and transform the data. The catalog can store table definitions, job definitions, and other control information that help manage the ETL environment inside Glue.

2) Automatic Schema Discovery

It allows you to set up crawlers that connect to different data sources. It classifies the data, obtains the scheme-related info, and automatically stores it in the data catalog. ETL jobs can then use this information to manage ETL operations.

3) Code Generation

Amazon Glue comes with an exceptional feature that can automatically generate code to extract, transform and load your data. The only input Glue would need is the path/location where the data is stored. From there, Glue creates ETL scripts by itself to transform, flatten and enrich data. Normally, Scala and Python code is generated for Apache Spark.

4) Developer Endpoints

This is one of the best features of Amazon Glue and helps interactively develop ETL code. When Glue automatically generates a code for you, you will need to debug, edit and test the same. The Developer Endpoints provide you with this service. Using this, custom readers, writers or transformations can be created. These can further be imported into Glue ETL jobs as custom libraries.

5) Flexible Job Scheduler

One of the most important features of Glue is that it can be invoked as per schedule, on-demand, or on an event-trigger basis. Also, you can simply start multiple jobs in parallel. Using the scheduler, you can also build complex ETL pipelines by specifying dependencies across jobs. Amazon Glue ETL always retries the jobs in case they fail. They also automatically initiate filtering for infected or bad data. All kinds of inter-job dependencies will be handled by Glue.

What is AWS Glue Architecture?

The different parts of AWS Glue Architecture are as follows:

• AWS Management Console
• AWS Glue Data Catalog
• AWS Crawlers
• AWS ETL operations
• Job Scheduling System

1) AWS Glue Console

The AWS Management Console is a browser-based web application for managing AWS resources. It has the following functionalities:

• Defines Glue objects such as crawlers, jobs, tables, and connections.
• Creates a layout for crawlers to work in.
• Creates job trigger events and timetables.
• Filters and searches Glue objects on AWS.
• Edits transformation scenario scripts.

2) AWS Glue Data Catalog

Glue Data Catalog offers centralized, uniform metadata storage for data tracking, querying, and transformation using saved metadata.

3) AWS Crawlers and Classifiers

Crawlers and classifiers automatically scan data from various sources, classify data, detect schema information, and store metadata in the Glue Data Catalog.

4) AWS Glue ETL Operations

The core of the ETL program generates Python or Scala code for data cleaning, enrichment, duplicate removal, and other complex data transformation tasks.

5) Job Scheduling System

A versatile scheduling system has the responsibility of starting jobs based on various events or a timetable.

What are the Benefits of AWS Glue?

By performing Glue ETL, you can leverage the following benefits:

• Various teams within your organization can employ Glue to collaborate on data consolidation tasks such as extracting, cleansing, normalizing, joining, loading, and executing scalable ETL workflows. This reduces the time it takes to analyze and use your data from months to just a matter of minutes.
• You can also automate many of the repetitive tasks associated with data integration. It provides recommendations for a schema for searching data sources, identifying data formats, and storing data. Code is automatically generated to perform the data conversion and loading process. With Glue, you can execute and manage thousands of ETL jobs and use Standard SQL commands to integrate and replicate data across several data stores.
• Since it works in a serverless ecosystem, you aren’t required to manage any infrastructure. Glue AWS manages, configures, and scales the resources needed to perform the data consolidation tasks. You only have to pay for the resources that the job consumes while it is running.

What is the Pricing of AWS Glue? 

Amazon Glue always charges an hourly rate, billed by the second. The pricing will depend on crawlers – who discover the data and ETL Jobs – who process and load your data. In addition to this, a simple monthly fee is involved in storing and accessing metadata from the data catalog. As per the free tier rules of Glue, Amazon does not charge for the first million objects stored and the first million objects accessed/requested. In addition to this, if you create a developer endpoint to generate and develop your ETL code, you will have to pay an hourly rate, billed per second. Yes, the pricing structure can be slightly overwhelming. Let us look at some scenarios to understand this better:

• ETL Jobs – For this example, consider Apache Spark as a Glue job type that runs for 10 minutes and consumes 6 DPUs. The Price of 1 Data Processing Unit (DPU) – Hour is $0.44. Since your job ran for 10 minutes an hour and consumed 6 DPUs, you will be billed 6 DPUs X 10 minutes at $0.44 per DPU-hour or $0.44.

• Development Endpoint – Now, let’s assume that you will provide a development endpoint to directly connect to your computer to interactively develop your ETL code. If 5 DPUs have been provisioned for your endpoint, then you need to run this development endpoint for 24 minutes. Hence you will be charged for 5 DPUs X 24 Minutes at $0.44 per DPU-Hour or $0.88. 

• AWS Glue Data Catalog billing Example – As per AWS Glue Data Catalog, the first 1 million objects stored and access requests are free. If you store more than 1 million objects and place more than 1 million access requests, then you will be charged.  Let’s assume that you will use 330 minutes of crawlers, and they hardly use 2 data processing units (DPU). The number of objects stored is < 1 million. Hence, the storage cost is 0. Assume that your access requests exceed 1 million requests. In this case, you will be charged $1. Crawlers will be charged at $0.44 per Data Processing Unit (DPU)-Hour, and you will pay for 2 Data Processing Unit (DPU) X 30 minutes at $0.44 or $0.44. Hence, the total monthly bill will be $1.44.

Learn about minimizing AWS Glue Costs and practical strategies to optimize the expenses.

What are the use cases for AWS Glue?

This section highlights the most common use cases of Glue. You can use Glue with some of the famous tools and applications listed below: 

• AWS Glue with Athena
• AWS Glue for Non-native JDBC Data Sources
• AWS Glue integrated with AWS Data Lake
• Snowflake with AWS Glue

1) AWS Glue with Athena

In Athena, you can easily use AWS Glue Catalog to create databases and tables, which can later be queried. Alternatively, you can use Athena in AWS Glue ETL to create the schema and related services in Glue.

2) AWS Glue for Non-native JDBC Data Sources

AWS Glue by default has native connectors to data stores that will be connected via JDBC. This can be used in AWS or anywhere else on the cloud as long as they are reachable via an IP. It natively supports the following data stores- Amazon Redshift, and Amazon RDS ( Amazon Aurora, MariaDB, MSSQL Server, MySQL, Oracle, PL/pgSQL).

To learn more about Amazon Redshift.

3) AWS Glue Integrated with AWS Data Lake

It can be integrated with AWS Data Lake. Further, ETL processes can be run to ingest, clean, transform and structure data that is important to you.

4) Snowflake with AWS Glue

Snowflake has great plugins that seamlessly gel with Amazon Glue. Snowflake data warehouse customers can manage their programmatic data integration process without worrying about physically maintaining it or maintaining any kind of servers and spark clusters. This allows you to get the benefits of Snowflake’s query pushdown, SQL translation into Snowflake, and Spark workloads. 

To know more about Snowflake Data Warehouse.

What are the Limitations and Challenges of AWS Glue?

While there are many noteworthy features of Glue, there are some serious limitations as well. 

• In comparison to the other ETL tools available today, Glue has only a few pre-built components. Also, given that it was developed by and for the AWS Console, it is not open to match all kinds of environments.
• Glue works well only with ETL from JDBC and S3 (CSV) data sources. If you are looking to load data from other cloud applications, such as File Storage Base, Glue will not be able to support it.
• Using Glue, all data is first staged on S3. This sync has no option for incremental sync from your data source. This can be limiting if you are looking at ETL data in real time.
• Glue is a managed AWS Service for Apache Spark and not a full-fledged ETL solution. Tons of new work is required to optimize PySpark and Scala for Glue. 
• Glue does not give any control over individual table jobs. ETL is applicable to the complete database.
• While Glue provides support for writing transformations in Scala and Python, it does not provide an environment to test the transformation. You are forced to deploy your transformation on parts of real data, thereby making the process slow and painful.
• Glue does not support traditional relational database types of queries well. Only SQL types of queries are supported, too, through some complicated virtual tables.
• The learning curve for Glue is steep. If you are looking to use Glue for your ETL needs, then you would have to ensure that your team is comprised of engineering resources with a strong knowledge of spark concepts.
• The soft limit of handling concurrent jobs is three only, though it can be increased by building a queue for handling limits. You will have to write a script to handle a smart auto-DPU to adjust the input data size.

As discussed above, you will encounter quite a few challenges while working with Amazon Glue, especially outside the AWS environment. Also, it requires your engineering team to spend a lot of time learning, building, monitoring & maintaining all the AWS pipelines. To remedy this, you can use a more effortless Cloud-based ETL Tool like Hevo Data. To find out how Hevo can be an effective & economical choice for you, check out the detailed comparison between Amazon Glue vs Hevo Data.

AWS Glue vs. Hevo

You can get a better understanding of Hevo’s Data Pipeline as compared to AWS Glue using the following table:

Parameter	AWS Glue	Hevo Data
Specialization	ETL, Data catalog	ETL, Data Replication, Data Ingestion
Pricing	AWS Data Catalog charges monthly for storage while AWS glue ETL charges on per hourly basis.	Hevo Pricing follows a flexible & transparent model where you pay as you grow. Hevo offers 3 tiers of pricing, Free, Starter & Business.
Data Replication	Full table; Incremental via Change Data Capture (CDC) through AWS Database Migration Service (DMS).	Full table; Incremental via SELECT/Replication key, Timestamp & Change Data Capture (CDC).
Connector Availability	Glue caters to Amazon platforms such as Redshift, S3, RDS, and DynamoDB — and AWS destinations, and other databases via JDBC	Hevo has native connectors with 150+ data sources and integrates with Redshift, BigQuery, Snowflake, and other Data Warehouses & BI tools.

Conclusion

The article discussed AWS Glue in great detail. It introduced you to the concept of the ETL process. Furthermore, it explained the four essential aspects of this ETL tool: Features, Pricing, Use Cases, and Limitations. However, Amazon Glue, although it is an efficient tool, still faces certain limitations, which were discussed above.Hevo is an all-in-one cloud-based ETL pipeline that will not only help you transfer data but also transform it into an analysis-ready form. Hevo’s native integration with 150+ data sources (including 60+ free sources) ensures you can move your data without the need to write complex ETL scripts.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Check out the Hevo pricing details to understand which plan fulfills all your business needs.

Share your experience with this blog and your understanding of Amazon Glue in the comments section!

FAQ on AWS Glue