Apache Atlas vs Google Data Catalog Simplified 101: Key Differences

In the world today, everyone is producing more data than ever before and with this daily production comes a lot of challenges that give rise to rules and regulations thereby making it harder to find the right type of data at the right time.

Data Access and Data Governance has become a major challenge to businesses as they try to secure ways in which these challenges can be overcome using data cataloging tools. With a Data Catalog in place, you can gain access to your data, tell the kind of data that you have now, know who is moving it, what the data is being used for, and how the data needs to be protected from harmful practices. 

This article is to differentiates the key components between two of the industry-leading data catalog platforms, Apache Atlas vs Google Data Catalog, using highlights of their features to show how they differ from one another.

Another part of the metadata is the operational data and can be seen as the operations that have been carried out on data such as when was it last refreshed, what sort of ETL jobs created it, the number of times the tables has been accessed by users, and which particular user accessed these data.

What is Google Data Catalog?

Google Data Catalog is a fully managed and scalable metadata management service provided by Google with which you can organize, discover, and manage your data stored in Google Cloud. In other words, Data Catalog is a centralized service that is fully managed by Google Cloud with privileges that enables you to build an optimized search index for your data assets such as datasets, tables, views, spreadsheets, data streams, text/CSV files, etc. This index is built on Data Catalog using the assets’ metadata like name, description, column definition, etc.

The metadata on Google Data Catalog comes with pre-defined structures and allows users to add more attributes using templates to their assets. It also stores metadata for assets managed by other Google Cloud Platform (GCP) services and details about them can be gotten using Data Catalog’s UI or API.

Google Data Catalog has specific components that will be discussed next to show how it is distinct from Apache Atlas stating its usefulness in the management of your metadata and data governance in general.

What is Apache Atlas?

Apache Atlas is a platform that offers solutions to data governance and metadata management. It enables the gathering, processing, and maintenance of metadata by monitoring the data processes, data stores, files, and any updates in the metadata repository.

Apache Atlas is conventionally used within the Hadoop environment though, it can be used with other environments as it allows integration with other enterprise data ecosystems. Atlas has a scalable and extensive set of core foundational governance services thereby enabling you to effectively and efficiently use its metadata and governance capabilities to organize and build catalogs for your data assets, classify, and manage them for your team of data professionals.

Apache Atlas has unique capabilities such as lineage, entities, etc. which will be discussed next to show its key differences vis-a-vis Google Data Catalog.

• Metadata Types and Instances
• Entities
• Classifications
• Lineage
• Search/Discovery
• Security and Data Masking
• Search Catalog
• Entry Group
• Get Entry
• Lookup Entry
• Templates and Tags
• On-Prem Connectors

1. Metadata Types and Instances

Apache Atlas allows users to define a model for metadata objects to be managed and these models are composed of definitions known as types. A type represents one or a collection of attributes that defines the properties for the metadata objects. You can define new types for the metadata to be managed and the metadata types can be pre-defined for various Hadoop and non-Hadoop metadata.

The metadata types in Apache Atlas can have primitive and enum attributes, complex attributes, object references, and can also inherit other metadata types. 

2. Entities

Entities on Apache Atlas can be defined as instances of types that capture metadata object details and their relationships. They represent an asset’s technical metadata, though they do not come with pre-defined structures, they have pre-defined entity types for various Hadoop and non-Hadoop metadata. REST Application Programming Interface (APIs) can also be used to work with types and instances to allow for easier integration.

3. Classifications

Entities are associated with Classifications to enable easier discoveries of assets and promote effective data management. Apache Atlas does not create a different object but uses Classifications objects to apply them to entities and have superTypes attributes such as expiry_data attribute in EXPIRES_ON.

The capabilities of Classification allow you to create dynamic classifications such as PII, EXPIRES_ON, DATA_QUALITY, SENSITIVE, etc. Classification on Apache Atlas can be propagated through lineage, therefore, ensuring that classifications follow the data as it goes through different processing.

4. Lineage

Apache Atlas has a unique feature called Lineage. Lineage allows you to view the history/trajectory of your data as it moves from one process to another using an intuitive User Interface (UI) and REST APIs to either access or update it. Lineage is carried out as Atlas reads the content of your metadata to build relationships among the data assets by noting the input and output of the query information it receives and ultimately generates a lineage map that shows all usage and transformations done on the assets over time.

It is very helpful to data professionals as you can quickly identify the sources of data and understand the impact of data and schema changes.

5. Search/Discovery

This is used to search entities by type, classification, attribute value, or free text through an intuitive UI. It also has SQL-like query language to search entities, Domain Specific Language (DSL), and uses REST APIs to search with complex criteria.

6. Security and Data Masking

Apache Atlas is integrated with Apache Ranger to enable authorization/ data-masking on data access based on classifications associated with entities and offers security for metadata access controls on access to entity instances and operations such as add/update/remove classifications.

7. Search Catalog

The search catalog is simply the search feature found in Data Catalog where you can search the catalog for results. A result set is returned when you search the catalog summarizing the details about the indexed assets where each SearchResult may have a small set of fields such as SearchResultType, SearchResultSubtype, relativeResourceName, and LinkedResource.

The search result is usually divided into two, ENTRY and TAG_TEMPLATE. ENTRY refers to data assets managed by other Google Cloud Services. Services such as BigQuery and Pub/Sub are automatically indexed by Data Catalog. TAG_TEMPLATE on the other hand refers to data assets that are native to Data Catalog.

8. Entry Group

An entry group is used to keep related entries together and it can be also used to determine users who can create, edit, and view entries within the group. The related data is kept in the group using Cloud Identity and Access Management (IAM). As stated in the previous section, services such as BigQuery and Pub/Sub are automatically indexed by Data Catalog to form entry groups.

9. Get Entry

The Get Entry operation is used to retrieve information regarding a given data asset, it receives a name parameter and returns one catalog Entry for each Search Catalog operation carried out. The Entry represents a native Data Catalog entity that describes the asset’s technical metadata and contains fieldsets that change according to its type.

10. Lookup Entry

Lookup Entry is used to find a catalog Entry associated with a data asset without having to perform a catalog search as you can go from an asset’s name to its catalog entry in one step. With Lookup Entry in Data Catalog, users can know the type of data stored in  Google Cloud and also know where it is stored.

11. Templates and Tags

Tags on Google Data Catalog are used to improve data governance. A Tag is a native Data Catalog entity that is used to automate processes and attach additional metadata to any data asset indexed by the catalog. A Tag can be attached to an Entry, its flexibility enables you to attach multiple fields as required to get a classification job done properly, and it must be created according to a user-defined Tag Template. For example, a Tag Template can be used to classify assets to search and troubleshoot all tables which have a failed status.

12. On-Prem Connectors

With Data Catalog, you can ingest technical metadata from non-Google Cloud data assets to Data Catalog for a unified view of all your data assets.

Comparison Table

Conclusion

This article has shows the key differences between Apache Atlas and Google Data Catalog. It has postulated that if you want a platform where you will install and manage your data catalog yourself then Apache Atlas is it, whereas, Google Data Catalog is a fully managed and server-less product, where you do not need to manage anything as it is hosted on the Google Cloud Platform (GCP).

It further showed that with Apache Atlas, you have features such as entities, classification, lineages, etc. while entries, tags, and templates are components synonymous with Google Data Catalog.

Hevo Data, a No-code Data Pipeline provides you with a consistent and reliable solution to manage data transfer between a variety of sources and a wide variety of Desired Destinations with a few clicks. Try a 14-day free trial and experience the feature-rich Hevo suite firsthand. Also, check out our unbeatable pricing to choose the best plan for your organization.

Frequently Asked Questions