Sharing information between departments or business units ("domains") of a company is critical for success. Sharing and consuming data assets is more successful if data is shared as a product. A data product is a collection of related data objects plus metadata, such as a business description, ownership and contact information, service level objectives, data dictionary, and more. In a Data Mesh, data products are typically also subject to various data management and organizational principles.

Snowflake Internal Marketplace enables companies to publish documented and governed data products, so they are discoverable and understandable for data consumers. Optionally, data quality metrics and SLOs can be included to make the product more trustworthy. The marketplace also offers rich capabilities to manage access to data products and wrap detailed governance around them to control which consumers can use which data products or which parts of a data product.

Snowflake Horizon Diagram

What You'll Learn

What You'll Need

What You'll Build

The setup instructions for this lab describe all the steps for you to create the 3 accounts, domain profiles, and roles shown in the diagram below.

The internal marketplace exists by default. It does not need to be created. But, you will configure it with provider profiles for the different business units via the organization account. The organization account is a recent Snowflake capability to optionally monitor and manage a set of regular accounts.

LabScenario

The setup follows these steps:

For Steps 2 through 7 you can download scripts here and execute them in different accounts as per the instructions below. In the Snowflake UI you can easily import these scripts like this:

Import

Step 1: Create a Snowflake trial account

Sign up for a trial account here

Step 2: Configure the first account and create two more accounts in the same org

-- Run this code in your PRIMARY Account
-- Make sure you update the four variables below (email_var, firstname_var, lastname_var, and pwd_var)

USE ROLE accountadmin;

-- Use the same name and email for all accounts
set email_var = 'FILL_IN_YOUR_EMAIL';
set firstname_var  = 'FILL_IN_YOUR_FIRST_NAME';
set lastname_var  = 'FILL_IN_YOUR_LAST_NAME';

-- Use the same password for users in all accounts
set pwd_var = 'FILL_IN_YOUR_PASSWORD';

CREATE OR REPLACE WAREHOUSE compute_wh WAREHOUSE_SIZE=xsmall INITIALLY_SUSPENDED=TRUE;
GRANT ALL ON WAREHOUSE compute_wh TO ROLE public;

--  Create a user and role for the sales domain:
USE ROLE accountadmin;
CREATE OR REPLACE ROLE sales_data_scientist_role;

SET my_user_var = CURRENT_USER();
ALTER USER identifier($my_user_var) SET DEFAULT_ROLE = sales_data_scientist_role;

CREATE OR REPLACE USER sales_admin
  PASSWORD = $pwd_var
  LOGIN_NAME = sales_admin
  DISPLAY_NAME = sales_admin
  FIRST_NAME = $firstname_var
  LAST_NAME = $lastname_var
  EMAIL = $email_var
  MUST_CHANGE_PASSWORD = FALSE
  DEFAULT_WAREHOUSE = compute_wh
  DEFAULT_ROLE = sales_data_scientist_role
  COMMENT = 'Sales domain admin';

GRANT ROLE sales_data_scientist_role TO USER sales_admin;
GRANT ROLE accountadmin              TO USER sales_admin;  -- for simplicity in this lab
GRANT CREATE SHARE ON ACCOUNT                    TO ROLE sales_data_scientist_role;
GRANT CREATE ORGANIZATION LISTING ON ACCOUNT     TO ROLE sales_data_scientist_role;


-- Next, create a user and role for the marketing domain:
USE ROLE accountadmin;
CREATE OR REPLACE ROLE marketing_analyst_role;

CREATE OR REPLACE USER marketing_admin
  PASSWORD = $pwd_var
  LOGIN_NAME = marketing_admin
  DISPLAY_NAME = marketing_admin
  FIRST_NAME = $firstname_var
  LAST_NAME = $lastname_var
  EMAIL = $email_var
  MUST_CHANGE_PASSWORD = FALSE
  DEFAULT_WAREHOUSE = compute_wh
  DEFAULT_ROLE = marketing_analyst_role
  COMMENT = 'Marketing domain admin';

GRANT ROLE marketing_analyst_role TO USER marketing_admin;
GRANT CREATE SHARE ON ACCOUNT                    TO ROLE marketing_analyst_role;
GRANT CREATE ORGANIZATION LISTING ON ACCOUNT     TO ROLE marketing_analyst_role;

USE ROLE orgadmin;
GRANT MANAGE LISTING AUTO FULFILLMENT ON ACCOUNT TO ROLE sales_data_scientist_role;
GRANT MANAGE LISTING AUTO FULFILLMENT ON ACCOUNT TO ROLE marketing_analyst_role;

While waiting for the email, you can go ahead and run the following parts.

Now, run the following commands to create the next two accounts that you need.

-- Run this code in your PRIMARY account
-- Create a secondary account in the same region (default!):
USE ROLE orgadmin;

CREATE ACCOUNT hol_account2
  admin_name = supply_chain_admin
  admin_password = $pwd_var
  first_name = $firstname_var
  last_name = $lastname_var 
  email = $email_var
  must_change_password = false
  edition = enterprise;

-- Create an organization account for admin purposes:
CREATE ORGANIZATION ACCOUNT hol_org_account
  admin_name = org_admin
  admin_password = $pwd_var
  first_name = $firstname_var
  last_name = $lastname_var 
  email = $email_var
  must_change_password = false
  edition = enterprise; 

-- Get an overview of all the accounts in the organization.
-- This SHOW command should return 3 rows:

SHOW ACCOUNTS;

Step 3: Configure the second account HOL_ACCOUNT2

In a separate browser tab, log in to the account you created in step 1 (HOL_ACCOUNT2) and set up this account.

-- Run this in hol_account2, logged in as supply_chain_admin user
-- Make sure you run this as ACCOUNTADMIN

USE ROLE accountadmin;

CREATE OR REPLACE WAREHOUSE compute_wh WAREHOUSE_SIZE=xsmall INITIALLY_SUSPENDED=TRUE;
GRANT ALL ON WAREHOUSE compute_wh TO ROLE public;

CREATE ROLE supply_chain_admin_role;
GRANT ROLE accountadmin TO ROLE supply_chain_admin_role; -- for simplicity in this lab
GRANT ROLE supply_chain_admin_role TO USER supply_chain_admin;

ALTER USER supply_chain_admin 
  SET DEFAULT_ROLE = supply_chain_admin_role;

USE ROLE supply_chain_admin_role;
CREATE DATABASE supply_chain_db;

Step 4: Configure the organization account and rename your primary account

Login to the Organization Account HOL_ORG_ACCOUNT as the org_admin user and execute the following commands in a worksheet.

You can also download code below from the file STEP4(HOL_ORG_ACCOUNT)_configure_org_account.sql from the repository:

-- Login to the Organization Account HOL_ORG_ACCOUNT and execute the following commands in a worksheet.

USE ROLE accountadmin;

CREATE OR REPLACE WAREHOUSE compute_wh WAREHOUSE_SIZE=xsmall INITIALLY_SUSPENDED=TRUE;
GRANT ALL ON WAREHOUSE compute_wh TO ROLE public;

-- Rename the Primary Account:
USE ROLE globalorgadmin;


-- execute the following two commands together, 
-- no other commands in between:

  show accounts;
  SET my_curr_account = (SELECT "account_name" FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())) order by "created_on" ASC LIMIT 1);

-- View and rename the account:
SELECT $my_curr_account;

ALTER ACCOUNT identifier($my_curr_account) 
  RENAME TO hol_account1 SAVE_OLD_URL = true;

-- Enable users with the ACCOUNTADMIN role to set up Cross-Cloud Auto-Fulfillment
SELECT SYSTEM$ENABLE_GLOBAL_DATA_SHARING_FOR_ACCOUNT('hol_account1');
SELECT SYSTEM$ENABLE_GLOBAL_DATA_SHARING_FOR_ACCOUNT('hol_account2');

SHOW ACCOUNTS;

-- You should see 3 rows similar to the image below.
-- Make a note of your account names, URLs, and passwords!

Import

Step 5: Create profiles for the Sales, Marketing, and Supply Chain domains

Continue working as the org_admin user in your Organization Account HOL_ORG_ACCOUNT to create data provider profiles. You will set up profiles for 3 business domains: Sales, Marketing, and Supply chain.

Import

Step 6: Setup of a TPC-H sample database

Step 7: Pre-populate the Internal Marketplace with Sample Listings

Setup is now complete!

In this section you will work in HOL_ACCOUNT1 to create and publish an organizational listing.

The publishing flow consists of 5 steps:

  1. Listing Title and Ownership
  2. Selecting Data Objects to Share
  3. Configure Access Control and the Approval Process
  4. Add Optional Metadata and SLOs
  5. Publish your listing to the internal marketplace

Login in to HOL_ACCOUNT1 as user sales_admin.

Publishing Flow (Step 1 of 5): Listing Title and Ownership

  1. Navigate to the Provider Studio and click the blue +Create Listing button in the top right.
  2. Select "Internal Marketplace".

IM

  1. Click on "Untitled Listing" and give your data product a meaningful title. Let's use Order Insights in this lab. Click "Save".

  1. Click on the +Profile button and select the Sales profile as the owner of this data product.

Publishing Flow Step (2 of 5): Selecting Data Objects to Share

Now let's select the data objects that we want to share in this data product.

Publishing Flow (Step 3 of 5): Configure Access Control and the Approval Process

Next you set the access control for the data product. Click on the gray +Access Control button.

For this first data product we keep it simple and stick with the defaults:

As a result, every data consumer will need to request access to obtain approval to use the data product. Click on Set up request approval flow to proceed.

You could configure an external workflow engine for the request approval process. But for this lab we choose to Manage requests in Snowflake. The email address for notifications defaults to the one from the Sales profile but could be changed.

After you confirm the approval flow settings, Snowflake prompts you for one more configuration. Here is why: this listing is configured to be discoverable by the entire organization. What if you add another account to the organization but in a different cloud region? Then Snowflake would transparently perform incremental replication to that region to minimize egress cost. As the data provider you can choose the frequency of this replication.

So lets (1) Review the settings, (2) Change the replication interval to daily (1 Days), and then (3) Save the settings for this listing:

Publishing Flow (Step 4 of 5): Add Optional Metadata and SLOs

Data products should be understandable and trustworthy for data consumers so let's add additional metadata to describe the product (see screenshot below).

Publishing Flow (Step 5 of 5): Publish your listing to the internal marketplace

Click the blue Publish button in the top right corner.

Your data product is now live! You can see it when you navigate to the Internal Marketplace.

In this section you will request access to the new data product for the Marketing domain and the Supply chain domain.

Request Access

Now let's also request access for the Supply chain team.

Review and Grant Access

Let's switch back to the perspective of the data product owner to review and grant the access requests.

Switch from Needs Review to Resolved Requests to see the history of requests.

Now that access has been granted, let's go back to the consumer roles:

What happens when the data owner decides to update the data product?

Let's examine some simple techniques for row- and column-level access control across domains.

Row-level Access Control across Domains

The data steward of the Sales domain has requested the following access restrictions:

Implement the following policy to make your data product compliant:

use schema tpch.sf1;
use role sales_data_scientist_role;

CREATE OR REPLACE ROW ACCESS POLICY country_filter AS (country INTEGER) 
RETURNS boolean ->
  CASE
    WHEN current_account_name() = 'HOL_ACCOUNT1' 
     AND current_role()         = 'SALES_DATA_SCIENTIST_ROLE'
     THEN true
    WHEN current_account_name() = 'HOL_ACCOUNT1' 
     AND current_role()         = 'MARKETING_ANALYST_ROLE'
     AND country                = 3 /* Canada */
     THEN true
    WHEN current_account_name() = 'HOL_ACCOUNT2' 
     AND country                = 24 /* USA */
     THEN true
   ELSE false
  END;

 ALTER TABLE nation ADD ROW ACCESS POLICY country_filter ON (n_nationkey);

Before we review the impact of this policy on the data consumers, let's look at a different governance requirement that requires column masking.

Data Masking across Domains

The data steward of the Sales domain has requested the following data masking to be enforced:

Implement the following policy to make your data product compliant:

 CREATE OR REPLACE MASKING POLICY order_mask AS (value INT, cutoff_date DATE) RETURNS INT ->
  CASE
    WHEN current_account_name() = 'HOL_ACCOUNT1' 
     AND current_role()         = 'SALES_DATA_SCIENTIST_ROLE'
     THEN value
    WHEN current_account_name() = 'HOL_ACCOUNT1' 
     AND current_role()         = 'MARKETING_ANALYST_ROLE'
     AND cutoff_date            >= '1996-01-01'
     THEN value
    WHEN current_account_name() = 'HOL_ACCOUNT2' 
     AND cutoff_date            >= '1996-01-01'
     THEN value
   ELSE null
  END;

ALTER VIEW order_summary ALTER COLUMN order_amount 
  SET MASKING POLICY order_mask USING (order_amount, o_orderdate); 

ALTER TABLE orders       ALTER COLUMN o_totalprice 
  SET MASKING POLICY order_mask USING (o_totalprice, o_orderdate);

ALTER TABLE lineitem     ALTER COLUMN l_extendedprice 
  SET MASKING POLICY order_mask USING (l_extendedprice, l_commitdate);

Effect of the Policies on Data Consumers

Let's see how the Supply Chain and Marketing teams are affected by the new policies.

As soon as you remove roles or accounts, the listing is no longer accessible for this role and/or account.

In this section we will review further capabilities for managing and monitoring listings as a data product owner or as an organization data steward:

Change/Revoke Access or Discoverability for a Listing

How to Grant Listing Management Privileges

How to Audit Access to Organizational Listing

As an organization admin you can query the organization_usage.access_history view to audit the access to all data products on the internal marketplace.

use role globalorgadmin;
use warehouse compute_wh;

select q.account_name, 
    q.user_name, 
    q.role_name, 
    q.query_text, 
    q.start_time, 
    q.end_time, 
    a.direct_objects_accessed, 
    a.provider_base_objects_accessed,
    a.provider_policies_referenced
from SNOWFLAKE.ORGANIZATION_USAGE.ACCESS_HISTORY a, 
    SNOWFLAKE.ORGANIZATION_USAGE.QUERY_HISTORY q
where a.query_id = q.query_id
and q.query_text ilike '%ORDER_INSIGHTS%'
order by query_start_time desc;

So far this lab has managed listings mainly through the Snowflake UI. But, data owners and data consumers can also work with listings programmatically through the Listing API.

In this section we point you to some of the most commonly used commands. These are not end-to-end exercises to create or alter listings, but we encourage you to experiment with some of these commands.

As a Listing Owner

As a Listing Consumer

Congratulations, you completed this Snowflake Internal Marketplace journey! You have seen how data products can be authored, published, requested, consumed, and governed. These are key capabilities for sharing documented and understandable data products across business units with governance and compliance control controls.

What you Learned

Related Resources