Simplifying Schema Creation and Management with Hevo’s Automated Schema Mapper

If you are a data engineer or work with data engineers, you know how tiresome it’s to manage Schema creation and evolution for data pipelines. You need to hard-code, monitor, and map every object and field. It’s a laborious process that kills your pipeline productivity and workflow.

With Hevo, 1000+ organizations have automated their data pipelines and significantly increased their data team’s productivity. One of the key anchors for highly effective pipelines on our platform is a fully automated and managed Schema Mapper.

Hevo’s Automated Schema Mapper (Automapper) automates the complete Schema creation and evolution process and makes Schema Management super easy.

This article will cover how our Automapper eases all the hassle of creating and updating Schema.

First, let’s understand why Manual Schema Creation and Changes are a Hassle

While creating Schema for a pipeline, your engineers have to create every object and field and define field types. Imagine if you have 100s of fields, it will take hours to create each field and activate your pipeline.

Once the Schema is created and the pipeline is activated, the job gets more challenging for your engineers. Data is mostly under construction and is evolving. Schema at source keeps changing. Your team has to allocate daily hours to monitor the Schema evolution at the source and update it at the destination.

Hard-coding these Schema changes also lead to your data pipeline downtime as your engineers have to pause the pipeline to make the changes in the code. If your data team cannot detect and update a Schema change at the destination, your pipeline breaks and disrupts the data flow.

Manually creating and updating Schema takes all your engineering bandwidth, and it causes huge delays in delivering data to analysts and frequent data downtime.

Eliminate all these Schema Hassles with our Automated Schema Mapper

Our Automapper automatically analyses the source schema for new data pipelines and creates the destination Schema regardless of complexity.

It’s equipped with a centralized inference engine that identifies data types for each field from the source and replicates it to the appropriate data types in the destination.

This enables your engineers to instantly deploy a new data pipeline without any efforts in creating Schema. All they need to do is select the source and configure the destination.

For all your data pipelines on Hevo, our Automapper automatically detects any change at the Source Schema and mirrors it on the Destination Schema without any input or monitoring from your engineers.

Destination Schema is always in sync with the Source Schema without any hours or efforts from your data engineers or team.

These changes are performed without pausing the pipelines or disrupting the data flow. Your analysts get consistent access to fresh data in real-time while our Automapper updates the Schema at the backend.

Fully Comprehensive Schema Management and Evolution

Hevo’s Automapper detects and updates all kinds of Schema evolution and changes at the destination. Here’s how it reacts to various types of source Schema changes:

1. If the data type is changed for a field at your source then our Automapper automatically updates the respective destination column’s data type.

2. If a new field is added to your existing object then our Automapper automatically creates a new column in the destination table.

3. If a field is deleted from your existing object then our Automapper doesn’t load any further data in the respective column.

4. If a new object is added to your source data then our Automapper creates and maps the respective Objects in the Destination and initiates the data replication process.

You have control over Schema updates by Automapper

We have found that there are cases where your team wants control over the Schema created by our Automapper. They want the ability to manage how data is stored at the Destination.

Hevo’s Automapper has an option where your data engineers or data team can edit the Schema created or updated by the Automapper.

It’s a simple interface where they can make custom changes to the Schema by selecting the type of change and defining the change; there is no coding involved.

With this option, you can:

• Drop specific fields in Objects
• Change field data type
• Drop certain Objects
• Change Object names

Here is how our Automated Schema Mapper has benefited our customers

Automatic Schema Management with Hevo

Experience how you can set up your data pipelines using Hevo in minutes without creating and managing Schema. Sign up for our 14-day free trial.