Data Challenges
Data plays a significant role at StockGro as it is used to make decisions on upcoming features, monitor KPIs, track growth, and power some of the application's main features. As StockGro’s platform matured with the addition of more features and a growing user base, the volume of data generated started increasing exponentially.
For instance, there are millions of transactions happening on StockGro’s application every day. All this data is stored in multiple MongoDB instances and is synced with StockGro’s BigQuery data warehouse.
Before Hevo, StockGro had built an on-premise data pipeline solution but this solution required constant maintenance, frequent manual intervention, and was unable to keep up with the exponential growth in data volume and number of data sources.
Earlier, Our pipelines were built using self-hosted Airflow, which required constant maintenance and couldn't handle schema updates. In addition, it was slow, and as a result, it wasn't real-time and often missed syncing some data. We spent between 5-10 hours per week to maintain these pipelines, plus additional hours to add new collections for syncing. And despite this, we had to run the scripts manually to generate some critical reports.
After 2-3 months of using Airflow, Raman and his team realized that it wouldn’t be a long-term solution as the current data challenges will only be aggravated with the anticipated increase in data volume. To resolve current data challenges and future-proof StockGro’s data stack, Raman started looking for a 3rd party data pipeline tool.
The Solution
After evaluating Hevo and some of the popular data pipeline tools in the market, Raman found that most were very expensive. Even then, he experienced data losses and faced problems connecting to data sources. Raman came across Hevo and decided to try it out.
We stumbled upon Hevo and decided to use it for some of the collections. It was a great first experience, as we were able to set up our pipelines within a few hours. We decided to move all the pipelines to Hevo, and it took only a few days to complete this process.
After finishing the initial setup process and getting used to the platform, here are some of the features and qualities that Raman and his team like the most about Hevo -
- Auto Schema Mapping: StockGro doesn’t have to worry about adding a new collection or table to the sync as Hevo will automatically map all the attributes.
- Zero Maintenance: Hevo is built on fault-tolerant architecture so Raman can schedule all data pipelines with predefined transformations using Hevo’s pipeline scheduling, workflows and models features enabling a zero maintenance experience.
- Wide range of sources and destinations: Raman is confident about having Hevo as StockGro’s data pipeline tool in the long run because of its existing and growing list of pre-built connectors to sources and destinations. Plus, he also appreciates how easy it is to connect to sources and build data pipelines.
Raman and his team use Hevo to load the transactional data generated by StockGro’s 4 million users from multiple MongoDB instances and sync it with their Google BigQuery data warehouse, which works as a single source of truth. StockGro’s team uses Redash and custom tools to analyze the data in BigQuery and build dashboards.
Key Results
Before Hevo, it used to take Raman and his team 5-10 hours every week to maintain their on-premise data pipelines. Now, the whole process is maintenance-free and the team can redirect their focus to analyzing the data without worrying about its availability.
Everyone in the team now has access to real-time, accurate data, making it easy to move fast. In addition, Hevo has made it effortless for us to maintain the pipelines, and the whole process is very reliable and scalable. In the last few months alone, our data has grown 5X, and Hevo maintained the same data accuracy and syncing speed.
Recently, StockGro raised $5 million in a pre-series A funding round to grow its user base, hire talent and build new features. Hevo is proud to be StockGro’s growth partner in its mission to democratize investments for the next generation
Excited to see Hevo in action and understand how a modern data stack can help your business grow? Sign up for our 14-day free trial or register for a personalized demo with our product expert.
