The ELT process has modernized data pipelines, fastened your data loading speed, and facilitated efficient data analysis.
However, there is still a delay in the analysis and the time to obtain reports and insights, as your analysts have to run a few additional data transformation jobs at the warehouse to clean and format the data.
This is because the data loaded into the warehouse may not be consistent, organized, and per the data warehouse tables format.
If you eliminate these additional data transformation jobs at the data warehouse, you will accelerate your analytics. All you need is a method to format and clean the data before loading it to the warehouse.
Thus, we innovated and launched an Inflight Transformation on Hevo’s ELT Pipelines, where data teams can perform lightweight data transformation inflight.
What is Inflight Transformation?
Inflight transformation is an ELT data pipeline feature innovated by Hevo, i.e., implementing lightweight data transformation within the pipeline before loading the data to the warehouse.
Hevo’s Inflight Transformation cleans, enriches, and standardizes data on the fly without impacting the load performance of the pipeline.
We believe inflight transformation should be essential in your modern ELT data pipelines. It enables your data teams to effortlessly maintain a consistent data format at the warehouse.
Why should you perform Inflight Transformation on Hevo?
Setting up inflight transformation on Hevo powers up your modern data analytics stack and offers a number of benefits to your analytics process. Following are the key benefits:
Data Consistency at Warehouse
Data from different sources can be formatted in a single format for your data warehouse. Your data teams can easily manage data-type mismatches and ensure the data is always compatible with the data warehouse tables.
Faster Analytics
It speeds up your analysis by eliminating the need to run additional transformations at the warehouse. Analysts get real-time access to consistent and organized data.
Setting up an ELT data pipeline with inflight transformation is the fastest way to move data for analytics.
Legacy Data Management
Usually, data from your legacy data sources will not be compatible with your modern data warehouse. But with Inflight Transformation, your team can easily load legacy data to a warehouse by formatting the legacy data inflight as per the data warehouse structure.
Key functions you can set up on Hevo’s Inflight Transformation
Your data team can perform several lightweight data formatting or transformation functions on Hevo. They can enrich, split, merge, normalize data, and handle data-type mismatch.
Following are the most commonly used inflight transformation jobs that your team can set up on Hevo,
1. Data and Time Formatting
Multiple data sources record dates and times in different formats. However, it’s efficient to maintain a single format in your data warehouse. You can format your date and time from source to any format inflight.
2. Mask or Hash Data
Your data may include a few sensitive fields like your customers’ personal information. This should not be accessible by your business users. You can mask or hash the sensitive fields like email address, contact number, etc., on the fly.
3. Data Enrichment
You can enrich your data by adding new fields inflight. For example, adding the product name for the respective dataset from the source while loading it to the warehouse.
4. JSON Normalization
You can convert or flatten JSON objects from NoSQL databases into a relational database. Functions like parsing JSON packages, formatting JSON objects into rows and columns, loading nested data into tables, and setting up relationships between those tables can be performed on the fly using inflight transformation.
5. Clean and Filter Events
There are a few events or fields that you would not want to load to your warehouse. You can filter out such fields or events in the pipeline itself. For example, filtering out all the canceled orders from your sales dataset.
Here is an example of how Inflight Transformation boost ELT data pipelines
Consider a global e-commerce company with multiple versions of their website like .us, .au for each location. They want to build a dashboard to analyze accumulated and region-wise traffic on each product page.
They would need to load the data for each region from Google Analytics to their data warehouse while adding the respective location name for each record. This can be easily achieved by creating a pipeline with inflight transformation each property on Google Analytics.
Using Inflight Transformation on Hevo, they can add a field ‘Country’ and populate it with the respective country name for each region. For a pipeline loading data from .au, they can set the rule to add a new field ‘Country’ with populated value ‘Australia’.
With this, analysts will have access to complete data of website traffic for each region with their respective location names in the data warehouse. Thus, they can directly run queries to build a dashboard for product pages while adding country as a dimension.
Powerful and Flexible Inflight Transformation Console
Hevo provides a flexible and powerful console to set up inflight transformations. There are 2 ways to set up inflight transformation on Hevo, one is by using our drag-and-drop interface, and the other is by using the Python console.
Using a drag-and-drop interface, you can effortlessly set up required formatting in minutes without writing any code.
You can set up and write your own rule or logic for data formatting using the Python console.
Here is what our customers say about our Inflight Transformation
Try out Hevo’s Inflight Transformation
Discover how inflight transformation further optimizes your data pipelines and data stack by signing up for a 14-day free trial.
If you are a Hevo customer and you want to set up inflight transformation then you can simply go to your respective pipeline’s transformation section and set up your respective rule or function.