Through this quickstart guide, you will learn how to use the Snowpark Pandas API to create a customer profile based on the using the Snowflake Sample TPC-H dataset, save it into a Snowflake table and to create a serverless task to schedule the feature engineering.
What Is Snowpark?
Snowpark is the set of libraries and code execution environments that run Python and other programming languages next to your data in Snowflake.
- Client Side Libraries - Snowpark libraries can be installed and downloaded from any client-side notebook or IDE and are used for code development and deployment. Libraries include the Snowpark Pandas API, which provides Python APIs to run your pandas code directly on your data in Snowflake.
- Code Execution Environments - Snowpark provides elastic compute environments for secure execution of your code in Snowflake. Runtime options include Python, Java, and Scala in warehouses, container runtimes for out-of-the-box distributed processing with CPUs or GPUs using any Python framework, or custom runtimes brought in from Snowpark Container Services to execute any language of choice with CPU or GPU compute.
Learn more about Snowpark.
What Is Snowpark Pandas API?
The Snowpark pandas API is a module in the Snowpark library that lets you run your pandas code directly on your data in Snowflake. Just by changing the import statement and a few lines of code, you can get the same pandas-native experience you know and love with the scalability and security benefits of Snowflake. With this API, you can work with much larger datasets so you can avoid the time and expense of porting your pandas pipelines to other big data frameworks or using larger and more expensive machines. It runs workloads natively in Snowflake through translation to SQL, enabling it to take advantage of parallelization and the data governance and security benefits of Snowflake.
Benefits Of Using The Snowpark Pandas API
- Meeting Python developers where they are – This API offers a familiar interface to Python developers by providing a pandas-compatible layer that can run natively in Snowflake.
- Scalable distributed pandas – This API bridges the convenience of pandas with the scalability of mature data infrastructure. pandas can now run at Snowflake speed and scale by leveraging pre-existing query optimization techniques within Snowflake. No code rewrites or complex tuning are required, so you can move from prototype to production seamlessly.
- Security and governance – Data does not leave Snowflake's secure platform. The Snowpark pandas API allows uniformity within data organizations on how data is accessed and allows for easier auditing and governance.
- No additional compute infrastructure to manage and tune – This feature leverages the Snowflake engine, and you do not need to set up or manage any additional compute infrastructure.
Learn more about Snowpark Pandas API.
What You'll Learn
- how to create Snowpark Pandas dataframe from a Snowflake table
- how to aggregate data to create new features
- how to Join Snowpark Pandas dataframes
- how to save a Snowpark Pandas dataframe into a Snowflake table
- how to create a serverless task to schedule the feature engineering
Prerequisites
- Git installed
- A Snowflake account. If you do not have a Snowflake account, you can register for a free trial account.
- A Snowflake account login with a role that has the ability to create database, schema and tables. If not, you will need to register for a free trial or use a different role.
What You'll Build
A Customer profile table using the Snowpark Pandas API and a serverless task that will run the feature engineering on a schedule.
Overview
This section covers cloning of the GitHub repository and creating the needed Snowflake objects (i.e role, warehouse, database, schema, etc..)
Clone The Git Repository
The very first step is to clone the GitHub repository. This repository contains all the code you will need to successfully complete this QuickStart Guide.
Using HTTPS:
git clone https://github.com/Snowflake-Labs/sfguide-data-engineering-pipelines-with-snowpark-pandas.git
OR, using SSH:
git clone git@github.com:Snowflake-Labs/sfguide-data-engineering-pipelines-with-snowpark-pandas.git
You can also use the Git integration feature of Snowflake Notebooks, in order to do that you need to fork the GitHub repository to be allowed to commit changes. For instructions how to set up Git integration for your Snowflake account see here and for using it with Snowflake Notebooks see this page.
During this step you will verify that the Snowflake Sample TPC-H dataset is available in your account, and if not add the share.
Verify That The Snowflake Sample TPC-H Dataset Is Available
- Log into Snowsight for your account
- Navigate to Databases
- Verify that you can see the SNOWFLAKE_SAMPLE_DATA database, if it is missing then you can add it folling the instructions in https://docs.snowflake.com/en/user-guide/sample-data-using
Create Database, Schema And Warehouse To Be Used
USE ROLE ACCOUNTADMIN;
CREATE DATABASE SNOW_PANDAS_DE_QS;
CREATE SCHEMA SNOW_PANDAS_DE_QS.NOTEBOOKS;
CREATE SCHEMA SNOW_PANDAS_DE_QS.DATA;
CREATE WAREHOUSE SNOW_PANDAS_DE_QS_WH;
Create Snowflake Notebook
Navigate To Snowflake Notebooks
- Navigate to the Notebooks section by clicking Projects and then Notebooks
![Navigate to Notebooks](img/665466c6401b64db.png)
- Click on the *down arrow next to + Notebook
![New notebook drop down](img/bf689d454571002d.png)
- If you have set up git integration choose Create from repository if not, then choose import .ipynb file.
![New notebook from menu](img/c2af89124ced6055.png)
Import .ipynb File
- Navigate to where you have cloned the GitHub repository and select Customer Profile Creation Pipeline.ipynb and click Open
![Select Notebook File](img/5267e4d1791ff2a.png)
- Keep the name, select SNOW_PANDAS_DE_QS and NOTEBOOKS for Notebook location, SNOW_PANDAS_DE_QS_WH for Notebook warehouse and click Create
![Select Notebook File](img/5512805e86d111b7.png)
Create From Repository
If you have forked the GitHub repository and create a integration to it in Snowflake you can open the notebook directly from the git repository.
- In the Create Notebook from Repository dialog click on Select .ipynb file
![Create Notebook from Repository Dialog](img/53146fad9125ceda.png)
2 Click on the repository integration you are using and select Customer Profile Creation Pipeline.ipynb and click Select File, if you do not see the file press Fetch to refresh with the latest changes from the repository
![Select Notebook File from Repository](img/5057bc3ebc274372.png)
- Name it Customer Profile Creation Pipeline, select SNOW_PANDAS_DE_QS, NOTEBOOKS for Notebook location and SNOW_PANDAS_DE_QS_WH for Notebook warehouse and click Create
![Create Notebook from Repository Dialog](img/124f82459def125.png)
Add Required Python Libraries
Before you run the notebook you need to add the following Python libraries:
- modin
- snowflake
- matplotlib
- seaborn
- In the Notebook click on Packages
- Search for modin and select modin in the list
![Modin search result](img/75e676b7a0a8f88d.png)
- Do the same for snowflake, matplotlib and seaborn. When done you should have the same packages as the list below (the versions might differ)
![Added packages](img/e2dff2ca6620c43d.png)
During this step you will learn how to use the Snowpark Pandas API to:
- Create datframe from a Snowflake table
- Aggregate data to create new features
- Join dataframes
- Save the result into a Snowflake table
- Create a serverless task to schedule the feature engineering pipeline (Currently in Private Preview)
Follow along and run each of the cells in the Notebook.
Within this Notebook, we will use Snowpark Pandas API to create DataFrames, join them, create new features and create a serverless task to schedule the feature engineering.
Congratulations, you have successfully completed this quickstart! Through this quickstart, we were able to showcase how you can use the Snowpark Pandas API to create DataFrames, join them, create new features, save the result to a Snowflake table, and create a serverless task to schedule the data transformation pipeline.
What You Learned
- Create datframe from a Snowflake table
- Aggregate data to create new features
- Join dataframes
- Save the result into a Snowflake table
- Create a serverless task to schedule the feature engineering
Related Resources