As Snowflake calls it, the native application framework is a doorway into the future of data warehousing. Currently in private preview, this new data management solution brings seamless data unification where developers can build and monetize data-intensive applications through Snowflake Marketplace, and users can install and run those applications directly on Snowflake. Some are even calling it the replacement for ETL/reverse ETL. Is that the case? ...We don’t think so.
Snowflake recently announced the new “Native Application Framework” at its annual user conference, Snowflake Summit 2022.
The release stirred up tweets and online takes from multiple data practitioners.
Snowflake, the global, unified network of data, is expanding the scope of its data cloud platform to the application cloud.
“We’re sort of evolving from a data warehouse to a data platform to a data cloud. And now we want to evolve that from a data cloud to really an application cloud”
Chris Child, senior director of product management at Snowflake.
Native application framework is a big step toward Snowflake’s philosophy of breaking down data silos. It aims to bring all of the work into Snowflake’s customer account. Using this deployment model, developers can build, distribute, and monetize applications through Snowflake Marketplace to a massive crowd of 6,300 Snowflake customers (as of April 30, 2022). Snowflake users can then install and use those applications directly on their Snowflake instances without having to move data from application servers to their Snowflake warehouses.
The new era of data collaboration is promising enormous benefits to application developers since they can directly leverage Snowflake’s independent compute and storage layer to package code and share it with other accounts directly or through the Snowflake Marketplace. LiveRamp has already built two solutions using the native application framework, and the future seems bright.
For application consumers, they no longer have to worry about sending sensitive data outside of their data cloud. Snowflake users can now enjoy greater trust in terms of data security and privacy by having applications operate on their data in the cloud warehouse itself.
But despite all the hype around native apps, the bigger question hanging over the industry is: “Do Snowflake native apps provide useful and enduring value, or is it just another empty promise?”
Snowflake’s native application framework claims to free its users from countless pipeline and data integration complexities. We expect this initiative to carve a path similar to the iPhone’s App Store wherein people can easily install and use any app on top of Snowflake. Maybe it will become a revolutionary shift or “the” replacement for ETL/reverse ETL, as some are claiming, or maybe it will evolve into yet another defunct catalog of so-called useful applications, just like SharePoint’s and Slack’s marketplace. What could happen next? Let’s find out.
Understanding The Native Apps Architecture
Snowflake Native Apps Architecture
Snowflake native application framework allows developers to package code or data using stored procedures, tasks, streams, UDFs, Snowpark, etc. on Streamlit (low code UX framework). Application developers can easily deploy code or provide data for applications and leave the operational complexity and data security to Snowflake. All of it is automated with native apps.
The application developer (provider) can build their own application-related objects (tables, views, etc.) and code (UDF, stored procedures, tasks, etc.) with the inclusion of “installer”. The installer is a stored procedure running DDL commands to create application objects. When a consumer imports a database from Share through App Listing, they also receive the installer’s stored procedure, using which they can create different objects in another schema (like App Objects, as shown in the diagram).
Snowflake customers can customize the installer process so that it can create different objects based on the parameters it receives. This lets developers charge customers based on the features they use. If a customer doesn’t have a Snowflake account, an application developer can give them a read-write account and charge them.
It’s interesting to note that application developers or providers can’t access customer accounts or view the data contained therein. The user has full control over the application database and can decide how it works with other information in their account. The developer can also write code, using logic and install methods, to hide it from the customer and protect their own intellectual property.
Snowflake Marketplace: App Store for Your Data Apps
Snowflake’s native apps feature Streamlit, an open-source app framework like Swift.
To extend the comparison, Snowflake is more like the iPhone as a whole than just an App Store where widgets can be bought. App developers don’t have to play in a limited sandbox. Instead, they can make custom apps that make full use of the hardware and native software.
Like an iPhone, Snowflake can also automatically control things like app permissions. An app for reverse ETL may be permitted to transmit data outside of Snowflake, an app for machine learning (ML) may retain data inside Snowflake but be permitted to send metadata back to a SaaS service, and an app for logging might be prohibited from talking with anybody or anything outside of Snowflake.
This would drastically alter the economics of developing data apps and give customers more power. Small teams might profitably create and distribute applications if users could find them in the store, and Snowflake users’ security teams could simply determine what access they feel comfortable providing without conducting formal due diligence. Comparable to downloading a smartphone app, installing a data app entails: inserting your credit card info, checking a few boxes, and you’re ready to go.
Is Native Application Framework a “Real” Alternative to ETL or Reverse ETL?
“Is data sharing and native data apps announced at snowflake summit real alternatives to etl and reverse etl?” asks one Twitter user.
Our answer – No.
At least in the foreseeable future, it is not going to happen. Let’s understand why:
Since native applications are hosted on the Snowflake cloud and data does not need to be loaded onto these apps, many people have incorrectly formed the impression that the emergence of this framework will eradicate the need for an ETL or reverse ETL tool.
Though it appears so, people are ignoring the fact that ETL tools are still the ones responsible for consolidating application data into the Snowflake repository. Your application runs on top of data stored in Snowflake, but the first data transfer from the application to Snowflake will always require an ETL.
Another key aspect to consider is that no matter how successful the Snowflake marketplace becomes, it can never onboard all the applications that organizations use. For instance, consider the case of the popular Apple App Store. As successful as it is, for some reason, it was unable to keep Fortnite onboard & we are sure that there are many such examples. In the case of Snowflake, the provision of native application connectors for Google Analytics, Google Ads, & Looker will undoubtedly cause conflicts for Google. They’d rather be more interested in selling their own warehouse, BigQuery, over Snowflake.
No matter how large the Snowflake Marketplace gets, there will always be cases when application providers and Snowflake disagree. Users will still use applications that Snowflake doesn’t support, so the market will still need ETL and reverse ETL solutions.
Benefits to Application Developers and Consumers
Application Developers
Snowflake native apps framework provides developers with several advantages across the whole lifespan of an application, from development to deployment to distribution.
Applications can make use of built-in Snowflake capabilities such as always-on availability, global collaboration, in-platform monetization, and native governance and security because they are built on the Snowflake infrastructure. With the help of these tools, developers can concentrate their efforts on creating robust apps quickly rather than creating and maintaining the necessary infrastructure.
In addition, because Snowflake Data Cloud supports different geographies, application developers can create their apps once and run them flawlessly almost anywhere. Companies that include their apps in Snowflake Marketplace become members of the Snowflake Marketplace Partner Program, thus allowing thousands of Snowflake users from all over the world to quickly find, download, and pay for their apps. This brings more visibility to their applications, and ultimately more sales.
For such companies, sales and deployment cycles are shortened. They no longer have to worry about vendor risk or difficulties with data exfiltration since Snowflake customers don’t need to relocate or share access to data. Since customers allow data access directly to the application without having to share it with the developers, developers can also streamline their operations by not having to maintain sensitive data. Because apps use compute in their customers’ accounts, application developers can increase their margins by lowering compute expenses.
Application Consumers
Using the native application framework, Snowflake customers can enhance data governance and minimize data silos because there is minimal data movement across applications and centralized, universal access.
Customers get to conceal data from the app provider. Using role-based access controls (RBAC), they can restrict what an application can do in their accounts by giving it certain rights like external access or log sharing.
Furthermore, Snowflake customers can drastically simplify and cut the length of the application procurement process with Snowflake Marketplace from days to just a few clicks. They can eliminate the time and effort needed to discover the application, since having a marketplace of trusted applications can help companies onboard a new application faster. Customers can use their data faster and receive more value from it by finding pre-built apps in Snowflake Marketplace as opposed to developing their own applications.
…Back to Our Question: Should You Hop On The Native Apps Bandwagon?
Case 1: You Own a Database Instead of a Data Warehouse
If your business uses cloud-based CRM, marketing, support, or analytical tools and stores data in a database management system like MySQL or PostgreSQL, the idea of investing in the Snowflake warehouse to take advantage of the Snowflake Native Apps may excite you.
But think again. The number one reason why you have invested in consolidating your data into a database is the cost. If you foresee your business data growing at an exponential rate, you might anticipate that native apps will give you the highest level of security, seamless operability, and enhanced transparency from the Snowflake data cloud.
It is not wise to hop on this wagon just because of the native application framework, as there are tons of other qualities to a data warehouse that can’t be neglected. You need to evaluate whether Snowflake is the right fit for your data needs, your business use cases, and your existing set of applications. Because once you buy into the Snowflake data warehouse service, it is really hard to migrate to another data warehouse.
To help you evaluate the cost of implementing a Snowflake data warehouse vs. a database, here we share some rough estimates:
| Feature | Snowflake Warehouse | Databases |
| Storage & compute cost | $23000/yr | $5000/yr |
| Personnel Cost | $500,000/yr (Data Warehouse Architect – $120k, Data Engineer – $140k, Analyst – $80k, Manager – $150k) | $70,000/yr(Database Administrator) |
There are other costs involved as well, but they depend more on the size of your data than the storage technology you choose to implement. You need to keep in mind factors like ROI when deciding to invest in it.
The primary question you should ask in such a situation is, “Will business benefits over the long term generate enough capital to handle the added expense and improve profits?”
If the answer is yes, and your business belongs to an industry that is thriving, great! Go ahead and discuss it with your management. If it’s the opposite, we advise you to concentrate on thoughtfully managing and organizing your data and not over-exerting your budget by paying for services you don’t actually need.
Case 2: You Own a Snowflake Warehouse
If your business is already a Snowflake customer, you do have the prerogative to use the native application framework. Alternatively, if you are a traditionalist who does not wish to dismantle the existing process, you can opt to stick with the current ETL methods.
The main advantage of native apps is that they provide complete transparency into what is happening with your data and eliminate the need to transfer or manage your data into another cloud-based tool. Snowflake native apps protect your client-sensitive data by adding an extra layer of security and preventing it from being siloed into a third-party application. Snowflake’s powerful engines can also be used by application developers to scale application storage and compute without the hassles of infrastructure management.
The downside is that you might be using an application that is not available on the native app framework. As a result, you miss out on the benefits of using the Snowflake-native application, which may give your competitors an advantage in terms of increased client trust and seamless data integration between applications and the Snowflake warehouse.
It’s also important to note that if you are somewhat of a data-mature company, the Snowflake Marketplace isn’t going to provide support for every app that you use, at least in the expected beginnings.
The Snowflake application marketplace will evolve over time, and the key aspect here is that the need for an ETL solution will not phase out. You’d still be using ETL to ingest a somewhat reduced number of data events from Snowflake’s unsupported applications. If your application armory comprises a miscellany of both Snowflake’s supported and unsupported applications, you should consider whether the change is worthwhile (fewer events = lower ETL costs, at the expense of less transparency and less ability to control and monitor data movements).
If there is an application split (some applications on native cloud and some apps standalone), how will you merge cross-departmental data? What will that look like?
There isn’t complete clarity on this since the framework is still in the beta phase. We believe that dealing with this situation will necessitate continuous data movement between the warehouse and the applications, still so in the form of ETL. You need to figure out how easy or difficult it will be to switch from existing tech to Snowflake native apps.
Another major issue is that Snowflake hasn’t made it clear what happens to your application data once you switch to the “native app kingdom”. Will that data be replicated from the application cloud to your Snowflake in its entirety? Will that data be completely accurate? There is still some uncertainty here.
Case 3: You Own Amazon Redshift/Google BigQuery Warehouse
If yours is a company that uses other data warehouse services like Redshift or Bigquery, Snowflake native apps might and should be of the least concern to you. We assume that since you’ve already invested in and researched existing data warehouses’ costs, migrating to a whole new platform is going to be incredibly hard.
Your decision to change the existing data warehouse should never be motivated by an “embryonic feature” that has yet to prove its worth. You should critically evaluate all the factors you are thinking of when migrating to the Snowflake warehouse.
The key factors to consider here are the migration strategy and the migration cost. You need to ponder over the question, “Does the native application framework have advantages that will help my business grow enough to cover your migration costs as well as yield significant business benefits in the long run?”
Another fact that we would like to underscore is that BigQuery and Redshift are provided by Google and Amazon, which are “the most” technically sound organizations in the world. Even if the Snowflake native application framework becomes a huge success, Google and Amazon have the technical strength to quickly release their own equivalent technologies (Google already has its own Workspace Marketplace). We believe that waiting patiently is a preferable and wise decision than chasing and paying the costs of migrating to a new warehouse.
And for the skeptics, we have compiled a rough estimate of the storage and compute costs associated with 100 TB of data.
| Feature | Snowflake | Redshift | Google BigQuery |
| Storage Cost | $27,600/yr | $29,496/yr | $24,576/yr |
| Compute Cost | $21,774/yr | $18,573/yr | $20,000/yr |
| Total | $49,374/yr | $48,069/yr | $44,546/yr |
Potential of The New Developments: Native Apps
With recent announcements, a lot of data connoisseurs have been enthused about the future of data warehouses. Whether on purpose or not, they’ve all portrayed the same idea:
Bucky Moore, partner at Kleiner Perkins, shares his thoughts on the future of cloud data services: “The logical next step is to use this foundation to build full-featured applications that both read and write to the warehouse.”
The future of the data industry, according to Martin Casado, general partner at Andreessen Horowitz, “All apps are just going to be reimplemented on top of the data layer.”
Thoughts of Patrick Chase, Principal at Redpoint Ventures, “It makes sense for apps to read and write to their customer’s data warehouse instead of their own database.”
You can envision the technical diagram underlying each of these predictions: The data warehouse is located in the backend, the data apps are located in the front, and an arrow pointing to a few decent APIs is located in the middle.
After Snowflake purchased Streamlit, a program to support the development of data apps, earlier this year, we got the exact same image in our heads. We reasoned that the acquisition made sense because, with a few adjustments, Streamlit could be used to create data apps by functioning as “a framework for integrating with Snowflake.” It might potentially be “a marketplace for apps that Snowflake stands behind” with a few minor adjustments. And this is exactly what happened.
Additionally, Snowflake extensively promoted Streamlit during the keynote speech, unveiling the app architecture and marketplace and implying that data applications are synonymous with Streamlit apps.
In our opinion, Snowflake had already envisaged creating something far beyond just the data applications. They planned on creating and deploying data applications on top of Snowflake warehouse, providing users with a seamless data integration utility. Snowflake Summit 2022 provided us a glimpse of the progression, and there’s a lot more that’s expected to come.
Snowflake native apps have now captured the buzz, and developers have already started building apps on the Snowflake native framework. If you are curious to understand how you can build one, you can check out this extensive piece.
See How Snowflake Native Apps Are Bringing Another Layer of Security to Your Data
Having to expose sensitive client information to other 3rd party applications in order to obtain reliable analytics is a challenge.
There are many advantages to these native apps, but we believe that their ability to do calculations directly in a customer’s data warehouse is by far their largest advantage. Access can be rigorously controlled, and a complete audit trail of all calculations and inquiries is made available to the businesses.
This feature was unavailable in the previous generation of analytics tools. Before Redshift, Snowflake, and BigQuery, it was difficult to find publicly accessible, indefinitely elastic data warehousing technologies. The only way to provide extensive analytics was to ingest every click, input field, and geo-location from different applications. People may be aware that the programs they use track every move they make, but they may be unprepared for the fact that a network of third-party suppliers is also privy to every aspect of their online behavior.
The analytics tools of this age, however, have the option to change. Rather than consuming NSA-level details about each individual’s IP address, device tag, and behavioral preference, Snowflake warehouse-native tools can function directly on raw data within a customer’s environment, ensuring that data never leaves the premises. However, a lot of newly developed tools are not moving in this direction; some even state directly that they view possessing the private customer data of other businesses as a “strategic asset.”
This tweet outlines our point:
We strongly believe that when you’re ready to buy analytics or experimental tools as a Snowflake customer, you should ask the analytics tool provider: Why are you unable to do calculations on my cloud? Why do I need to send you this much information?
And you need to insist on a serious response.
Final Thoughts
Snowflake empowers thousands of businesses to centralize their data and eliminate data silos. The native application framework is an audacious step towards making data collaboration easier. Using native apps on the Snowflake cloud, Snowflake is bringing wider app accessibility for Snowflake customers while at the same time eliminating the hassles of infrastructure management for application developers.
In our opinion, the native application framework has the potential to redefine what data warehouses can do. Snowflake has a strong customer base and a thriving community of data professionals vouching for their warehouse services, and they won’t have problems hosting a good number of applications on their marketplace.
Change can always seem exciting at the beginning.
Microsoft’s SharePoint and Salesforce’s Slack also had similar beginnings with their marketplaces. Ask most professionals now, and most will admit that these have dissolved into a graveyard of somewhat useful widgets. And Snowflake’s native application marketplace isn’t exempt from this.
We believe so because there are many popular applications like Google Analytics, Google Ads, and BI tools like Looker offered by competitors that are hard to imagine on Snowflake’s native cloud. Google would rather have users invest in BigQuery over Snowflake.
For Snowflake, the risk may be enormous, but we think the return will be even greater. Running applications inside a data warehouse can reinvent what a data warehouse looks like in the future. Other database providers will be forced to follow Snowflake if the ecosystem gains traction. Snowflake will be the industry’s iOS—the platform that every developer wants to create for first—as the first mover on a premium product.
Akshaan is a data science enthusiast who loves to embrace challenges associated with maintaining and exploiting growing data stores. He has a flair for writing in-depth articles on data science where he incorporates his experience in hands-on training and guided participation in effective data management tasks.