Datadog Snowflake Integration: 3 Easy Steps

Snowflake is a Software-as-a-Service (SaaS) platform that helps businesses to create data warehouses. It runs in the cloud, providing its users with an option for storing their data in the cloud. Snowflake has a very elastic infrastructure and its compute and storage resources scale well to cater to your changing storage needs. However, you’ll need to monitor various factors of your Snowflake data warehouse. This is why Datadog Snowflake Integration is needed.

For example, you will need to monitor the performance of your queries, billing, credit usage, and more. It is difficult to monitor all these factors manually. Thus, you will need a tool for doing this. Datadog is a cloud monitoring platform that can help you to monitor your Snowflake data warehouse. In this article, we will be discussing Datadog Snowflake integration.

Prerequisites:

This is what you need for this article:

• A Snowflake Account.

What is Datadog?

Datadog is a monitoring and Analytics platform that can be used to determine the performance Metrics and monitoring events for infrastructure and Cloud services. It can be used for monitoring services such as databases, servers, and tools.

You can deploy the Datadog monitoring service on-premise or as Software-as-a-Service (SaaS). It supports Windows, Linux, and Mac operating systems. It also supports a wide variety of cloud providers including Snowflake, Microsoft Azure, Google Cloud platform, and more.

Datadog’s user interface is made up of customizable dashboards that can show graphs generated from multiple data sources in real time. It can also notify users about performance issues on set metrics like compute rates. It notifies users through channels like Slack, PagerDuty, and Email.

What is Snowflake?

Snowflake is a data warehouse-as-a-service platform developed for the Cloud. It has a data architecture that uses the scalable, elastic Azure Blobs Storage as the Internal Storage Engine and Azure Data Lake for storage of Unstructured, Structured, and on-premise data ingested via the Azure data factory. 

Snowflake platform is an analytical, flexible, intuitive data warehouse service provided as SaaS. It provides a data warehouse that is quicker, simpler to use, and far more flexible than normal data warehouse offerings.

Snowflake provides security and protection of data using Amazon S3 policy controls, SSO, Azure SAS tokens, and Google Cloud Storage access permissions. You can also scale your storage depending on your storage needs. Thus, when you use a Snowflake data warehouse to store your data, you will enjoy the scalability, and security of your data, among many other benefits. 

What are the Steps for Datadog Snowflake Integration?

You can use the Datadog Agent to monitor your Snowflake data warehouse. After performing the Datadog Snowflake Integration, you will be able to monitor Metrics such as Billing, Credit Usage, Query Metrics, and more.

Note that the Metrics are collected through queries to Snowflake. The queries created by the Datadog Snowflake integration are all billable by Snowflake. You can use the following command to upgrade the existing Integration:

datadog-agent integration install datadog-snowflake==2.0.1

You can then perform the configuration by following the steps given below:

• Step 1: Creating Custom Role
• Step 2: Collecting Data 
• Step 3: Monitoring the Integration

Step 1: Creating Custom Role

The first step in Datadog Snowflake Integration is to create custom roles. Create a user-specific role for Datadog to monitor Snowflake. Run the following commands on Snowflake and give it access to the ACCOUNT_USAGE schema:

use role ACCOUNTADMIN;
grant imported privileges on database snowflake to role SYSADMIN;
use role SYSADMIN;

Note that by default, the integration will monitor the SNOWFLAKE database and the ACCOUNT_USAGE schema. The SNOWFLAKE database is available by default, and can only be viewed by users in the role ACCOUNTADMIN or users granted any role by the ACCOUNTADMIN. 

You can also use an alternative approach by creating a DATADOG custom role and grant it access to the ACCOUNT_USAGE role. This is shown below:

-- Create a new role for monitoring Snowflake usage.
create role DATADOG;
  
-- Grant the new role privileges on the SNOWFLAKE database.
grant imported privileges on database SNOWFLAKE to role DATADOG;

-- Create a new user, skip if you’re using an existing user.
create user DATADOG_USER
LOGIN_NAME = DATADOG_USER
password = '<PASSWORD>'
default_warehouse = <WAREHOUSE>
default_role = DATADOG
default_namespace = SNOWFLAKE.ACCOUNT_USAGE;
  
-- Grant the monitor role to the new user.
grant role DATADOG to user <USER>;

Step 2: Collecting Data 

Next, navigate to the conf.d/ folder located at the root of your Agent’s configuration directory and edit the snowflake.d/conf.yaml file to start collecting data about your Snowflake performance. 

  ## @param account - string - required
    ## Name of your account (provided by Snowflake), including the platform and region if applicable.
    ## For more information on Snowflake account names,
    ## see https://docs.snowflake.com/en/user-guide/connecting.html#your-snowflake-account-name
    #
  - account: <ACCOUNT>
   
    ## @param user - string - required
    ## Login name for the user.
    #
    user: <USER>
   
    ## @param password - string - required
    ## Password for the user
    #
    password: <PASSWORD>
  
    ## @param role - string - required
    ## Name of the role to use.
    ##
    ## By default, the SNOWFLAKE database is only accessible by the ACCOUNTADMIN role. Snowflake recommends
    ## configuring a role specific for monitoring:
    ## https://docs.snowflake.com/en/sql-reference/account-usage.html#enabling-account-usage-for-other-roles
    #
    role: <ROLE>
  
    ## @param min_collection_interval - number - optional - default: 3600
    ## This changes the collection interval of the check. For more information, see:
    ## https://docs.datadoghq.com/developers/write_agent_check/#collection-interval
    ##
    ## NOTE: Most Snowflake ACCOUNT_USAGE views are populated on an hourly basis,
    ## so to minimize unnecessary queries the `min_collection_interval` defaults to 1 hour.
    #
    min_collection_interval: 3600
  
    # @param disable_generic_tags - boolean - optional - default: false
    # Generic tags such as `cluster` will be replaced by <integration_name>_cluster to avoid
    # getting mixed with other integraton tags.
    # disable_generic_tags: true

The default value for the min_collection_interval parameter is 1 hour. The Snowflake metrics are normally aggregated by day, but you can reduce the number of queries by increasing the interval. 

Step 3: Monitoring the Integration 

Once done, restart the agent. Congratulations! You have performed Datadog Snowflake integration. You can now monitor your Snowflake data warehouse in Datadog.

What Can You Achieve by Setting up Datadog Snowflake Integration?

By connecting Datadog with Snowflake, you can monitor and visualize the performance and usage of your Snowflake data warehouse in real-time, as well as correlate it with other infrastructure and application metrics. This can help you identify and troubleshoot issues, optimize resource utilization, and understand the impact of changes on your data warehouse’s performance.

Some specific benefits of connecting Datadog and Snowflake include the following:

• Monitor key Snowflake metrics: You can track important Snowflake metrics such as CPU usage, data warehouse size, and query performance and set alerts to notify you of any issues.
• Visualize Snowflake data: You can use Datadog’s dashboards and charts to visualize your Snowflake data and see how it changes over time. This can help you identify trends and patterns and make informed decisions about your data warehouse.
• Correlate Snowflake data with other metrics: By connecting Datadog to other tools and services you use, you can see how your Snowflake data warehouse is performing in the context of your overall system. For example, you can see how Snowflake query performance is affected by the load on your application servers or how data warehouse size changes as you ingest more data from external sources.
• Automate incident response: You can use Datadog’s integration with Snowflake to automate incident response and remediation. For example, you can set up alerts to automatically trigger a script that scales up your data warehouse when it is under heavy load or to automatically notify the relevant team when there are issues with the data warehouse.

Overall, Datadog Snowflake monitoring can help you gain a better understanding of your data warehouse’s performance and usage and make it easier to identify and resolve issues as they arise.

Conclusion

As organizations expand their businesses, managing the work process becomes crucial for the efficient monitoring of data. Datadog provides data monitoring solutions. Datadog Snowflake Integration powers stakeholders and management to collaborate on their workflow and build a quality product meeting the requirements with ease. In case you want to export Data from Datadog into your desired Database/destination like Snowflake, then Hevo Data is the right choice for you! 

Visit our Website to Explore Hevo

Hevo Data provides its users with a simpler platform for integrating data from 150+ sources such as Datadog for Analysis. It is a No-code Data Pipeline that can help you combine data from multiple sources. You can use it to transfer data from multiple data sources into your Data Warehouse, Database, or a destination of your choice. It provides you with a consistent and reliable solution to managing data in real-time, ensuring that you always have Analysis-ready data in your desired destination. It helps you carry out processes like loading data from Datadog to Snowflake easily.

Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also have a look at our unbeatable pricing that will help you choose the right plan for your business needs!

Share your experience of learning about Datadog Snowflake Integration! Let us know in the comments section below!