Quick Guide to the Snowflake Semantic Layer in 2024

Snowflake is a cloud data warehouse that has taken the world by storm, establishing itself as one of the core technologies in the cloud era. Snowflake is a cross-cloud platform; you can run it on AWS, Azure, or GCP. Central to its power is the Snowflake Semantic Layer, which helps organizations transform raw data into business-friendly terms, ensuring consistency across departments. The “Data-as-a-Service” platform allows users to ingest data in multiple formats and run different workloads against it, such as data analysis, machine learning, and reporting. 

With such capabilities, organizations normally rely on Snowflake as their main data warehouse. Snowflake decouples computing from storage, allowing the creation of different warehouses—i.e., compute clusters—each for a different department or team in the organization that needs to query and analyze the data. This gives organizations finer control over their bills.

Organizations that have such large deployments often face other kinds of challenges. How can the different departments define their business definitions and agree on them with other departments? The definition of a business term can differ between departments. Data in the data warehouse is in its rawest form and lowest granularity, and the column and table names are too technical.

Semantic Layer’s Definition and Types

The semantic layer solved the problem of having too many definitions of the same entities. It is placed between the data warehouse and the consumption layer—i.e., the tools consuming data from the data warehouse. The semantic layer defines different dimensions and metrics according to the business. 

Organizations tried first to implement the semantic layer at different levels of data warehouses. These implementations were :

• Semantic Layer in Data Warehouse. Some organizations tried to implement the business definitions and metrics directly in the data warehouse. This was soon proven ineffective as it was contrary to data warehouses’ generic and granular nature.
• Semantic Layer in Data Mart. Implementing business logic as separate data marts was a popular choice – and still somehow -. Data marts are built based on the department that needs them and are refreshed with data from the data warehouse. The problem with this implementation is the duplication of effort and data. Data engineers had to build extra data pipelines to feed data marts, and managing the scheduling of those pipelines could be complicated if different data marts have different refresh rates.
• Semantic Layer in the BI Tools. Another implementation relied on the BI tools to define and implement the business logic while they extract data from the data warehouse. This was problematic as these definitions could not be shared across different BI tools, which meant duplication of definitions.

Nowadays, a semantic layer is often implemented as a dedicated metadata layer independent of the data warehouse.

Benefits of Semantic Layer

Implementing the semantic layer maps business entities to physical data. This helps to democratize the data and facilitates access to it. Semantic layers have given us some additional unique features and benefits that include :

• Single Source of Truth. The semantic layer offers a single source of truth for all business definitions. Implementing the same business logic in multiple locations is unnecessary, which increases the risk of discrepancies between definitions.
• Enhanced Security and Governance. Having all your business logic in one location allows you to implement better security policies. It also enables finer governance and auditing, such as knowing who made what changes and when.
• Easy Creation and Sharing of New Logic. Business users can easily create new metrics without the need for complex SQL and share their work with others to avoid duplication. 

Honeydew: A New Approach

If you want to build a snowflake semantic layer, Honeydew is your perfect solution. Firstly created under the Snowflake startup challenge, Honeydew is a SaaS that provides a semantic layer on top of Snowflake. Its entities, datasets, and metrics data model is simple yet effective. Additionally, SQL is at the heart of Honeydew. This makes it possible to quickly onboard teams of data analysts, data scientists, and data engineers. Honeydew is SOC2 Type II compliant and controls data access through the Snowflake security model.

How to Build a Snowflake Semantic Layer Using Honeydew

But how does it work? You start by defining your business logic on Honeydew. Honeydew is built on entities. You can create your entities based on one or more Snowflake tables, views, or complex queries. Honeydew allows you to physically store your entities on tables or in views in Snowflake. After that, you define relations between those entities. Relations are defined by their direction (n-ary) and columns to join. Honeydew supports different joins, such as left, right, full, and inner.

You can then define datasets. The dataset consists of attributes chosen from entities you built earlier, filters to apply to those entities, and metrics defined based on the attributes of the used entities. Like entities, datasets can also be stored in Snowflake.

Honeydew uses metrics and attributes to define business logic. Attributes can be derived from multiple entities if there is a derived relation between those entities. They can be constant values, calculated values, or even use aggregations. Metrics, on the other hand, are aggregations without the “group by” clause. They can be calculated using any attribute in the dataset if it has the same or lower granularity. 

You connect your tools directly to Honeydew to interact with the snowflake semantic layer. Honeydew supports an SQL interface as well as JDBC. Your tools do not need to know the underlying physical model of how your data is stored on Snowflake. They can use the entities and metrics Honeydew exposed to them. Behind the scenes, Honeydew translates the query issued by your tools into one referencing the physical tables and views stored on Snowflake. 

Recently, Honeydew announced that it has a Snowflake Native App. This enables secure access to Honeydew directly from the Snowflake IDE or a Snowflake connection. Honeydew relies on the external services capabilities of the Snowflake native apps to have its app communicate securely with its SaaS backend.

Snowflake provides user management, access control, encryption, and network separation for every query and connection. Now, with a Native App, you can give the snowflake semantic layer the same protection when accessing it through a Snowflake connection.

Hevo: Your Data Migration and Transformation Powerhouse

Having a semantic layer is great, but how would you get your data in Snowflake in the first place? Enters Hevo, your new favorite data integration solution. Hevo is a no-code data movement platform that supports more than 150 sources and destinations. Its drag-and-drop user interface and out-of-the-box data transformations allow more people on your team to build data pipelines. 

Some of the benefits that Hevo provides are:

• No Infrastructure to Manage. Hevo is a SaaS, meaning you do not need to manage servers.
• Multiple Data Transformation Options. With Hevo, you can transform your data before loading it into your data warehouse using transformations or after loading it using models – you can even leverage your DBT’s models.
• Flexible Data Replication Options. Hevo offers many data replication options, such as replicating entire databases, specific tables, or even individual columns. It can work in batching mode or real-time.
• Auto Schema Management. Hevo provides a “Schema Mapper,” which helps map events to specific destinations. It can also map any changes in the source events to the destinations without human interactions, such as adding or removing fields from events. Furthermore, you can load new tables created in your databases using the same pipeline you created by enabling the “Include New Tables” option.
• On-Demand Credit. In Hevo, you can have an on-demand credit reaching up to 60% of your subscribed plan. The on-demand credit ensures that your pipelines are not disrupted even when you exhaust your plan’s quota. 
• Observability and Monitoring. Hevo offers you different levels of observability, including:
  • Latency and speed of data ingestion and loading
  • Billable and historical usage details through graphs and counts
  • Success and failures at each replication stage
  • Event failures and resolution assistance
  • Filtered views
  • Imminent Events quota exhaustion and available actions

Hevo offers multiple plans based on your needs, including a free plan with few connectors and a limited events quota. You can try the product for free. 

Conclusion

Semantic layers were born from the need for a unified business logic for all data consumers. Before then, business users struggled to duplicate metrics’ exact dimensions and definitions across multiple tools. For Snowflake, Honeydew presents a robust implementation of the semantic layer. It supports SQL, connections from many tools using JDBC, and a Snowflake native app to access its features from within Snowflake. Combine this with Hevo, which has integrations with 150+ sources, a drag-and-drop interface, and great observability over your data pipelines, and you will have a cloud data warehouse ecosystem that is easy to use and can provide you with great insights.

FAQ on Snowflake Semantic Layer