ForCustomer/SFNotebook_End-to-End Data Science using Snowpark_Easy Path.ipynb

{
 "metadata": {
  "hex_info": {
   "author": "Diana Shaw",
   "exported_date": "Thu Feb 29 2024 20:20:21 GMT+0000 (Coordinated Universal Time)",
   "project_id": "b4861126-fb2c-4f2e-a1db-f725fca1baba",
   "version": "draft"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat_minor": 4,
 "nbformat": 4,
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "HOL_Overview",
    "collapsed": false
   },
   "source": "# Snowflake Notebook End-to-End Data Science - Easy Path",
   "id": "276544ab-e258-4602-a809-f3db660dba95"
  },
  {
   "cell_type": "code",
   "id": "9be5a4ba-56fd-4b09-911c-615917b7c48c",
   "metadata": {
    "language": "python",
    "name": "EndToEnd",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Import necessary functions\nimport streamlit as st\nfrom snowflake.snowpark.context import get_active_session\nsession = get_active_session()\n\n# Define image in a stage and read the file\nimage=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/End-to-end_demo.png\" , decompress=False).read() \n\n# Display the image\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "HOL_Background",
    "collapsed": false
   },
   "source": [
    "## Background Information\n",
    "\n",
    "Tasty Bytes is one of the largest food truck networks in the world with localized menu options spread across 30 major cities in 15 countries. **Tasty Bytes is aiming to achieve 25% YoY sales growth over 5 years.**\n",
    "\n",
    "As Tasty Bytes Data Scientists, we have been asked to support this goal by helping our food truck drivers more intelligently pick where to park for shifts. \n",
    "\n",
    "**We want to direct our trucks to locations that are expected to have the highest sales on a given shift.\n",
    "This will maximize our daily revenue across our fleet of trucks.**\n",
    "\n",
    "To provide this insight, we will use historical shift sales at each location to build a model. This data has been made available to us in Snowflake.Our model will provide the predicted sales at each location for the upcoming shift.\n",
    "\n"
   ],
   "id": "3fa8db12-24c0-4fd5-8387-9241b55371c7"
  },
  {
   "cell_type": "code",
   "id": "a3513b76-4e4d-4d61-afa4-a6e7cda02632",
   "metadata": {
    "language": "python",
    "name": "ProblemOverview",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/problem_overview.png\" , decompress=False).read()\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Libraries",
    "collapsed": false
   },
   "source": "## Import Packages\n\nJust like the Python packages we are importing, we will import the Snowpark modules that we need.\n\n**Value**: Snowflake modules provide efficient ways to work with data and functions in Snowflake.\n\n",
   "id": "63a66d83-6552-4390-8f4a-abecaffcd076"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Libraries_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Import Packages\nimport pandas as pd\nimport plotly.express as px\nimport plotly.io as pio\nimport sys\nimport cachetools\n\n# Streamlit\nimport streamlit as st\n\n# Snowpark\nfrom snowflake.snowpark.context import get_active_session\nimport snowflake.snowpark.functions as F\nsession = get_active_session()\n\n# Import Snowflake modules\nfrom snowflake.snowpark import Session\nimport snowflake.snowpark.functions as F\nimport snowflake.snowpark.types as T\nfrom snowflake.snowpark import Window",
   "id": "78669da6-cf98-4905-8ce7-4d107f566da2"
  },
  {
   "cell_type": "code",
   "id": "eb8dd6d6-7146-413c-9313-181f4bc6eccb",
   "metadata": {
    "language": "python",
    "name": "HOLAssignedVariables",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Set your user value\nuser = \"USER0\"",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "20667e33-c854-4f7b-a51d-86a33a0c0a41",
   "metadata": {
    "language": "python",
    "name": "SetHOL_UserVariables",
    "collapsed": false
   },
   "outputs": [],
   "source": "number = user[4:]\n\n# Concatenating the number with the required connection parameters\nrole = \"ROLE\" + number\nwarehouse = \"WH\" + number\nschema = \"SCHEMA\" + number",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "32055275-83cc-46a7-8027-701a189a162d",
   "metadata": {
    "language": "python",
    "name": "HOLEnvironment",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Checking environment setting and versions associated with Snowpark connection\nfrom snowflake.snowpark.version import VERSION\nprint(\"----------------------------------------\")\nsnowflake_environment = session.sql('select current_warehouse(), current_database(), current_schema(), current_version()').collect()\nsnowpark_version = VERSION\nprint('Warehouse                   : {}'.format(snowflake_environment[0][0]))\nprint('Database                    : {}'.format(snowflake_environment[0][1]))\nprint('Schema                      : {}'.format(snowflake_environment[0][2]))\nprint('Snowflake version           : {}'.format(snowflake_environment[0][3]))\nprint('Snowpark for Python version : {}.{}.{}'.format(snowpark_version[0],snowpark_version[1],snowpark_version[2]))\nprint(\"----------------------------------------\")",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Part1_PrepareData",
    "collapsed": false
   },
   "source": "# Part 1 - Use Snowpark to access and prepare data for modeling",
   "id": "1d0c0e73-1049-4096-b72e-800a41185179"
  },
  {
   "cell_type": "code",
   "id": "b5d0268d-41dd-4e76-a4b2-8aa027ace3f5",
   "metadata": {
    "language": "python",
    "name": "Part1",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Part1.png\" , decompress=False).read() \nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "1fd3b3f4-358e-40bd-8c44-2598cb992bdc",
   "metadata": {
    "name": "SnowparkMLOverview",
    "collapsed": false
   },
   "source": "\nSnowpark ML is the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake, including components for model development and operations. With Snowpark ML, you can use familiar Python frameworks for preprocessing, feature engineering, and training. You can deploy and manage models entirely in Snowflake without any data movement, silos, or governance tradeoffs."
  },
  {
   "cell_type": "code",
   "id": "4ab96dcc-ca3d-44bd-8e0b-8ab436365ddf",
   "metadata": {
    "language": "python",
    "name": "SnowparkML_Overview",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/snowpark_ML.png\" , decompress=False).read() \nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkDataFrame",
    "collapsed": false
   },
   "source": "## Snowpark DataFrame\n\nLet's create a Snowpark DataFrame containing our shift sales data from the **shift_sales_v** view in our Snowflake account using the Snowpark session.table function. A DataFrame is a data structure that contains rows and columns, similar to a SQL table.\n\n**Value:** Familiar representation of data for Python users.\n\n",
   "id": "2c0f76db-e5c7-4d4a-92dc-f389f3e41c22"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkDataFrame_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "snowpark_df = session.table(\"SHIFT_SALES_V\")",
   "id": "449b12ef-e1a4-4d2e-8925-b62628fbcf4e"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkDataFrame_View",
    "collapsed": false
   },
   "source": "## Preview the Data\n\nWith our Snowpark DataFrame defined, let’s use the .show() function to take a look at the first 10 rows.\n\n**Value:** Instant access to data.\n\n",
   "id": "37430d96-c797-4afb-b6e9-1280d222415f"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkDataFrame_View_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Preview the data using the .show() function to look at the first 10 rows.\nsnowpark_df.show()",
   "id": "56bb0880-3518-484e-a77d-cfbf1744da17"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkDataFrame_Transformations",
    "collapsed": false
   },
   "source": [
    "## Select, Filter, Sort\n",
    "\n",
    "Notice the Null values for \"shift_sales\". Let's look at a single location.To do this, we will make another Snowpark DataFrame, location_df, from the above DataFrame and we will:\n",
    "\n",
    "1. Select columns\n",
    "2. Filter to a single location ID\n",
    "3. Sort by date\n",
    "\n",
    "**Value**: Efficient transformation pipelines using Python syntax and chained logic.\n",
    "\n"
   ],
   "id": "abdd91e2-64ff-4ba0-bbca-18c236b0dbfc"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkDataFrame_Transformations_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Select\n",
    "location_df = snowpark_df.select(\"date\", \"shift\", \"shift_sales\", \"location_id\", \"city\")\n",
    "\n",
    "# Filter\n",
    "location_df = location_df.filter(F.col(\"location_id\") == 1135)\n",
    "\n",
    "# Sort\n",
    "location_df = location_df.order_by([\"date\", \"shift\"], ascending=[0, 0])\n",
    "\n",
    "# Display\n",
    "location_df.show(n=20)"
   ],
   "id": "2ee49ad3-0979-4363-8a43-0dc33e4c234c"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_How",
    "collapsed": false
   },
   "source": "We can see that shift sales are populated 8 days prior to the latest date in the data. The **missing values** represent future dates that do not have shift sales yet.\n\n## Snowpark works in two main ways:\n\n1. Snowpark code translated and executed as SQL on Snowflake\n2. Python functions deployed in a secure sandbox in Snowflake",
   "id": "cb6d2ad4-4c91-43a5-83ac-05c2800ffdce"
  },
  {
   "cell_type": "code",
   "id": "f1472b6e-2120-45c2-8216-3322876c0f28",
   "metadata": {
    "language": "python",
    "name": "Snowpark",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image1=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Snowpark_ML_API.png\" , decompress=False).read() \nimage2=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/SnowparkValue.png\" , decompress=False).read() \n\n# Display the image\nst.image(image1, width=1000)\nst.subheader(\"Here's the value of using Snowpark:\")\nst.image(image2, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_ExplainQuery",
    "collapsed": false
   },
   "source": "## Explain the Query\n\nLet's look at what was executed in Snowflake to create our location_df DataFrame.\n\nThe translated SQL query can be seen in the Snowsight interface under _Activity_ in the _Query History_ or directly in our notebook by using the explain() function. \n\n**Value:** Transparent execution and compute usage.",
   "id": "08f1b00b-3560-49a4-a7ea-523fd16687c8"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_Explain",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "location_df.explain()"
   ],
   "id": "8db7ed9c-b597-4737-8b09-43540b9d17b4"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "CompareSFDataFramevsPandasDataFrame",
    "collapsed": false
   },
   "source": [
    "## Compare DataFrame Size\n",
    "\n",
    "Let's bring a sample of our Snowflake dataset to our Python environment in a pandas DataFrame using the to_pandas() function. We will compare how much memory is used for the pandas DataFrame compared to the Snowpark DataFrame. As we will see, no memory is used for the Snowpark DataFrame in our Python environment. All data in the Snowpark DataFrame remains on Snowflake.\n",
    "**Value:** No copies or movement of data when working with Snowpark DataFrames.\n",
    "\n"
   ],
   "id": "0ce2b480-b880-4fdd-aa3c-3c94d7d0c0cf"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "CompareSFDataFramevsPandasDataFrame_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Bring 10,000 rows from Snowflake to pandas\n",
    "pandas_df = snowpark_df.limit(10000).to_pandas()\n",
    "\n",
    "# Get Snowpark DataFrame size\n",
    "snowpark_size = sys.getsizeof(snowpark_df) / (1024*1024)\n",
    "print(f\"Snowpark DataFrame Size (snowpark_df): {snowpark_size:.2f} MB\")\n",
    "\n",
    "# Get pandas DataFrame size\n",
    "pandas_size = sys.getsizeof(pandas_df) / (1024*1024)\n",
    "print(f\"Pandas DataFrame Size (pandas_df): {pandas_size:.2f} MB\")"
   ],
   "id": "3b78bc1f-4154-4288-b2b2-85c496a7829d"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_DataExploration",
    "collapsed": false
   },
   "source": "## Data Exploration\n\nHere, we will use Snowpark to explore our data. A common pattern for exploration is to use Snowpark to manipulate our data and then bring an aggregate table to our Python environment for visualization.\n\n**Value:** - Native Snowflake performance and scale for aggregating large datasets. - Easy transfer of aggregate data to the client-side environment for visualization.\nAs we explore our data, we will highlight what is being done in Snowflake and what we are transferring to our client-side environment (Python notebook environment) for visualization.\n\n",
   "id": "eb800752-b83e-4340-9e1d-d3da732c7fb9"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkDF_Count",
    "collapsed": false
   },
   "source": "## How many rows are in our data?\n\nThis will give us an idea of how we might need to approach working with this data. Do we have enough data to build a meaningful model? What compute might be required? Will we need to sample the data?\n\n**What's happening where?:** Rows counted in Snowflake. No data transfer.\n\n",
   "id": "276e6283-133c-4531-bf10-b1314521b1e2"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkDF_Count_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "#Use the .count() function\n",
    "snowpark_df.count()"
   ],
   "id": "d57909f6-d03b-4f12-9ca0-e831ed44107f"
  },
  {
   "cell_type": "code",
   "id": "5dc81ecc-d844-45c7-aa54-fb8b931b0e92",
   "metadata": {
    "language": "python",
    "name": "SnowparkDF_Count_SiS",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "df = session.table('SHIFT_SALES_V').group_by(F.col('CITY')).agg(F.count('LOCATION_ID').alias('total_locations'))\nst.bar_chart(data=df,x='CITY',y='TOTAL_LOCATIONS')",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_Describe",
    "collapsed": false
   },
   "source": "## Let's calculate some descriptive statistics.\n\nWe use the Snowpark describe() function to calculate summary statistics and then bring the aggregate results into a pandas DataFrame to visualize in a formatted table.\n\n**What's happening where?:** Summary statistics calculated in Snowflake. Transfer aggregate summary statistics for client-side visualization.\n\n",
   "id": "271fb4e5-0907-4d0d-bb63-85a79b702f44"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_Describe_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "#Use the Snowpark DataFrame .describe function. You need to need to visualize from a pandas DataFrame\n",
    "snowpark_df.describe().to_pandas()"
   ],
   "id": "8a73b888-fb61-46c2-8571-98b7e9932b43"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_PreForModeling_Numeric",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# What are the numeric columns?\n",
    "# Define Snowflake numeric types\n",
    "numeric_types = [T.DecimalType, T.DoubleType, T.FloatType, T.IntegerType, T.LongType]\n",
    "\n",
    "# Get numeric columns\n",
    "numeric_columns = [col.name for col in snowpark_df.schema.fields if type(col.datatype) in numeric_types]\n",
    "numeric_columns"
   ],
   "id": "10b3b4af-2182-411f-bbc3-c4f356c701ed"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_PreForModeling_Categorical",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# What are the categorical columns?\n",
    "# Define Snowflake categorical types\n",
    "categorical_types = [T.StringType]\n",
    "\n",
    "# Get categorical columns\n",
    "categorical_columns = [col.name for col in snowpark_df.schema.fields if type(col.datatype) in categorical_types]\n",
    "categorical_columns"
   ],
   "id": "003897a5-2f35-468d-bf7e-8777a37d19b5"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_Explore_SiS",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# What are the average shift sales (USD) by city?\n# Group by city and average shift sales\ndf = snowpark_df.group_by(\"city\").agg(F.mean(\"shift_sales\").alias(\"avg_shift_sales\"))\n\n# Pull to pandas and plot\nst.bar_chart(data=df,x='CITY',y='AVG_SHIFT_SALES')",
   "id": "8665c3bf-1b18-402f-9601-76e7bd2968ed"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_FeatureEngineering",
    "collapsed": false
   },
   "source": "## Feature Engineering\n\nNow let's keep revelant columns and transform columns to create features needed for our prediction model.To make some of our features more useful, we will normalize them using standard preprocessing techniques, such as One-Hot Encoding and MinMaxScaling. With SnowparkML, you can use a standard sklearn-style API to execute fully distributed feature engineering preprocessing tasks on Snowflake compute, with zero data movement. Let's fit a scaler and encoder to our data, then use it to transform the data, producing new feature columns.\n\n\n**Value:** The Snowpark syntax makes pipelines easy to implement and understand. The syntax also allows for easy migration of Spark pipelines to Snowflake.\n\n\n**All transformations for feature engineering in this notebook will be executed on Snowflake compute.**\n\nNotice what we haven't had to do? No tuning, maintenance, or operational overhead. We just need a role, warehouse, and access to the data.\n\n**Value**: Near-zero maintenance. Focus on the work that brings value.\n\n",
   "id": "d2bd04e5-acfa-4f3a-9fab-fa73c285685b"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_FeatureEngineering_RollingAverage",
    "collapsed": false
   },
   "source": [
    "## Create a Rolling Average Feature\n",
    "\n",
    "We will use a Snowflake window function to get a **rolling shift average by location** over time. Window functions allow us to aggregate on a \"moving\" group of rows.\n",
    "\n",
    "#### **Step 1. Create a Window**\n",
    "\n",
    "Our window will partition the data by location and shift. It will order rows by date. It will include all rows prior to the current date of the observation it is aggregating for.\n",
    "\n"
   ],
   "id": "eebb0bf0-0026-4dce-b913-a3a8c1c94216"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_FeatureEngineering_RollingAverage_CreateWindowCode",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "window_by_location_all_days = (\n",
    "    Window.partition_by(\"location_id\", \"shift\")\n",
    "    .order_by(\"date\")\n",
    "    .rows_between(Window.UNBOUNDED_PRECEDING, Window.CURRENT_ROW - 1)\n",
    ")"
   ],
   "id": "08a80979-6c77-48b9-9ddc-8fb1e5f013bd"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_FeatureEngineering_AggregateWindows",
    "collapsed": false
   },
   "source": [
    "#### **Step 2. Aggregate across the Window**\n",
    "\n"
   ],
   "id": "2a392ecc-89d5-4c9e-8e35-2ff1047e2f44"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_FeatureEngineering_AggregateWindowsCode",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "snowpark_df = snowpark_df.with_column(\n",
    "    \"avg_location_shift_sales\", \n",
    "    F.avg(\"shift_sales\").over(window_by_location_all_days)\n",
    ")"
   ],
   "id": "06722409-6f5a-478d-ace6-47d9eef38b4b"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_ImputeMIssingValues",
    "collapsed": false
   },
   "source": [
    "## Impute Missing Values\n",
    "\n",
    "The rolling average feature we just created is missing if there are no prior shift sales at that location. We will replace those missing values with 0.\n",
    "\n"
   ],
   "id": "dc1cefbe-a91b-45d8-a22a-aa52b6ebe838"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_ImputeMIssingValues_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "snowpark_df = snowpark_df.fillna(value=0, subset=[\"avg_location_shift_sales\"])"
   ],
   "id": "523adbc7-205c-463f-acd2-d73c8cb1889c"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkMLModelingAPI",
    "collapsed": false
   },
   "source": "## Leverage Snowpark ML Modeling API to create features\n\nSnowpark ML provides APIs to support each stage of an end-to-end machine learning development and deployment process and includes two key components: [Snowpark ML Modeling](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-modeling) and [Snowpark ML Ops](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-mlops-model-registry).\n\n[Snowpark ML Modeling](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-modeling) supports data preprocessing, feature engineering, and model training in Snowflake using popular machine learning frameworks, such as scikit-learn, xgboost, and lightgbm. This API also includes a preprocessing module that can use compute resources provided by a Snowpark-optimized warehouse to provide scalable data transformations.\n\nSnowpark ML Operations (MLOps), featuring the [Snowpark ML Model Registry](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-mlops-model-registry), complements the Snowpark ML Development API. The model registry allows secure deployment and management of models in Snowflake, and supports models trained both inside and outside of Snowflake.",
   "id": "5b8d2565-0726-4f1e-a842-edff14425bac"
  },
  {
   "cell_type": "code",
   "id": "d890e654-de61-43ff-bce6-c8dce6f3b6ad",
   "metadata": {
    "language": "python",
    "name": "Snowpark_4DE_DS",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Snowpark4DE.png\" , decompress=False).read()\n\n# Display the image\nst.subheader(\"Here are a Snowpark Features commonly used for Data Engineering tasks:\")\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "AlterWarehouseSizeUp",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "session.sql(\"ALTER WAREHOUSE \" + warehouse + \" SET WAREHOUSE_SIZE = 'LARGE'\").collect()",
   "id": "a0ade1af-883d-4843-bf9d-6fa61a449fa0"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLModelingAPI_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Import Snowpark ML: Machine Learning Toolkit for Snowflake\n",
    "import snowflake.ml.modeling.preprocessing as snowmlpp\n",
    "\n",
    "# Define our scaler and ordinal encoding functions\n",
    "\n",
    "# Snowpark ML scaler (MinMaxScaler) is used to shrink data within the given range, usually of 0 to 1. \n",
    "# It transforms data by scaling features to a given range. It scales the values to a specific value range without changing the shape of the original distribution.\n",
    "# For the Tasty_Bytes data, use MinMaxScaler to normalize \"CITY_POPULATION\" into \"CITY_POPULATION_NORM\" with values between 0 and 1.\n",
    "\n",
    "def fit_scaler(session, df):\n",
    "    mm_target_columns = [\"CITY_POPULATION\"]\n",
    "    mm_target_cols_out = [\"CITY_POPULATION_NORM\"]\n",
    "    snowml_mms = snowmlpp.MinMaxScaler(input_cols=mm_target_columns, output_cols=mm_target_cols_out)\n",
    "    snowml_mms.fit(df)\n",
    "    return snowml_mms\n",
    "\n",
    "# Snowpark ML ordinal encoding (OE) is used to improve model performance by providing more information to the model about categorical variables. \n",
    "# It can help to avoid the problem of ordinality, which can occur when a categorical variable has a natural ordering (e.g. “small”, “medium”, “large”).\n",
    "# For the Tasty_Bytes data, use OE to change \"SHIFT\" which is currently AM or PM into and integer representation of \"SHIFT_OHE\" is 1.0 or 0.0. \n",
    "\n",
    "def fit_oe(session, df):\n",
    "    oe_target_cols = [\"SHIFT\"]\n",
    "    oe_output_cols = [\"SHIFT_OE\"]\n",
    "    snowml_oe = snowmlpp.OrdinalEncoder(input_cols=oe_target_cols, output_cols=oe_output_cols)\n",
    "    snowml_oe.fit(df)\n",
    "    return snowml_oe"
   ],
   "id": "f2e664e2-e37a-4599-bef3-bd47d870732a"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLModelingAPI_PreprocessingFunctionsCode",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Run Snowpark ML preprocessing functions against our feature data\n",
    "\n",
    "# For the Tasty_Bytes data, use MinMaxScaler to normalize \"CITY_POPULATION\" into \"CITY_POPULATION_NORM\" with values between 0 and 1.\n",
    "snowml_mms = fit_scaler(session, snowpark_df)\n",
    "normed_df = snowml_mms.transform(snowpark_df)\n",
    "\n",
    "# For the Tasty_Bytes data, use OneHotEncoder to change \"SHIFT\" which is currently AM or PM into and integer representation of \"SHIFT_OHE_AM\" is 1 or 0 and \"SHIFT_OHE_PM\" is 1 or 0. \n",
    "snowml_oe = fit_oe(session, normed_df)\n",
    "oe_df = snowml_oe.transform(normed_df)\n",
    "oe_df.show()"
   ],
   "id": "7bef3b90-ee88-4202-a0c5-07d04d9fe691"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_CreateHoldoutData",
    "collapsed": false
   },
   "source": [
    "## Filter to Historical Data\n",
    "\n",
    "Our data includes placeholders for future data with missing shift sales. The future data represents the next 7 days of shifts for all locations. The historical data has shift sales for all locations where a food truck parked during a shift. We will only use historical data when training our model and will filter out the dates where the shift_sales column is missing.\n",
    "\n"
   ],
   "id": "2ae7ff76-59bc-4aea-adad-90a8acbaf60e"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_CreateHoldoutData_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Data Science best practice: Always perform data quality on your training set e.g. remove nulls or invalid cells as they are the biggest problem in a training set as they output high false positives\n",
    "\n",
    "# Specifically for Tasty_Bytes data, dates where \"shift_sales\" are null values reflect future dates where sales need to be predicted.\n",
    "# Filter out these future dates so these records will not be used in model training. \n",
    "historical_df = oe_df.filter(F.col(\"shift_sales\").is_not_null())"
   ],
   "id": "411ec9ff-3751-4b71-b5ac-4be29cef27d3"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_PersistTransformations",
    "collapsed": false
   },
   "source": [
    "## Persist Transformations\n",
    "\n",
    "If we want to save the changes we can either save it as a table, meaning the SQL generated by the DataFrame is executed and the result is stored in a table or as a view where the DataFrame SQL will be the definition of the view.\n",
    "**save_as_table** saves the result in a table, if **mode='overwrite'** then it will also replace the data that is in it.\n",
    "\n"
   ],
   "id": "b77f1264-42f7-4f7f-993a-07d143e15e72"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_PersistTransformations_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Let's select \nhistorical_df.write.save_as_table(table_name='HOL.' + schema + '.INPUT_DATA', mode='overwrite')\nsession.table('HOL.'+schema+'.INPUT_DATA').show()",
   "id": "e19f47fa-281c-49c0-8f45-d9630662dafb"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Part2_TrainMLModel",
    "collapsed": false
   },
   "source": "# Part 2 - Use Snowpark to train a model",
   "id": "cd41e298-f6d3-4b8c-9206-ebb214565053"
  },
  {
   "cell_type": "code",
   "id": "b208ffd0-7755-4d04-8e7c-d2ab48e60fe6",
   "metadata": {
    "language": "python",
    "name": "Part2",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Part2.png\" , decompress=False).read() \n\n# Display the image\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_ModelPrep",
    "collapsed": false
   },
   "source": [
    "## Drop Columns\n",
    "\n",
    "Let's return to the original prepared table, with all cities listed, and drop columns that will not be used in the model.\n",
    "\n"
   ],
   "id": "80a40aff-0039-4bda-ab7b-ff98ff42ca1e"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_ModelPrep_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "prepared_df = historical_df.drop(\"location_id\", \"city_population\", \"shift\", \"city\", \"date\")\n",
    "prepared_df.show()"
   ],
   "id": "2a93dfbd-ad07-46c8-b917-defd5bdf61fa"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Snowpark_XGBoostRegressorModel",
    "collapsed": false
   },
   "source": [
    "## Build a simple XGBoost Regression Model on Snowflake\n",
    "\n",
    "We will now use our training data to train a linear regression model on Snowflake.Recall from above, the two main ways that Snowpark works:\n",
    "\n",
    "1. Snowpark code translated and executed as SQL on Snowflake\n",
    "2. Python functions deployed in a secure sandbox in Snowflake\n",
    "\n",
    "We will be leveraging the deployment of Python functions into Snowflake for training and model deployment.\n",
    "\n"
   ],
   "id": "db7f8fa5-bce9-44aa-b25f-845865d59f79"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_XGBoostRegressorModel_Columns",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Retrieve column names needed in the next code block to populate feature_column_names\n",
    "prepared_df.columns"
   ],
   "id": "e15ac942-7134-4b08-9f16-dcb1f6be32a5"
  },
  {
   "cell_type": "code",
   "id": "52d8857d-2a03-4279-a7ec-954fa51fd1e3",
   "metadata": {
    "language": "python",
    "name": "CheckColumns",
    "collapsed": false
   },
   "outputs": [],
   "source": "LABEL_COLUMNS = [\"SHIFT_SALES\"]\nOUTPUT_COLUMNS = [\"PRED_SHIFT_SALES\"]\nFEATURE_COLUMN_NAMES = [\"SHIFT_OE\", \"CITY_POPULATION_NORM\", \"MONTH\", \"DAY_OF_WEEK\",\"LATITUDE\",\"LONGITUDE\",\"AVG_LOCATION_SHIFT_SALES\"]\n",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_XGBoostRegressorModelSetup_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's define relevant features needed for the prediction model.\n",
    "LABEL_COLUMNS = [\"SHIFT_SALES\"]\n",
    "OUTPUT_COLUMNS = [\"PRED_SHIFT_SALES\"]\n",
    "FEATURE_COLUMN_NAMES = [\"SHIFT_OE\", \"CITY_POPULATION_NORM\", \"MONTH\", \"DAY_OF_WEEK\",\"LATITUDE\",\"LONGITUDE\",\"AVG_LOCATION_SHIFT_SALES\"]\n",
    "\n",
    "input_df = prepared_df.select(*LABEL_COLUMNS, *FEATURE_COLUMN_NAMES)\n",
    "input_df.show()"
   ],
   "id": "caa7e956-b69f-4437-bb15-d95c99b89944"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkML",
    "collapsed": false
   },
   "source": [
    "SnowparkML also includes metric calculations such as correlations, and more. We will use the SnowparkML correlation method on our input dataframe to identify any linearly correlated features to the output. We'll also use matplotlib to plot the resulting matrix. Notice that all of the correlation calculations are pushed down to Snowflake!\n",
    "\n"
   ],
   "id": "b18c401c-6e4c-4b45-a684-f11f8c0bb323"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkML_Correlation_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "import seaborn as sns\nimport matplotlib.pyplot as plt\nfrom snowflake.ml.modeling.metrics.correlation import correlation\ncorr_df = correlation(df=input_df)\n\nfig, ax = plt.subplots()\nsns.heatmap(corr_df.corr(), ax=ax, annot=True)",
   "id": "31a22f10-17ee-4129-a3b7-3b6f5fe8e8fe"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkML_SplitData4Modeling",
    "collapsed": false
   },
   "source": [
    "What's great about this, is that we are using a lot of Snowpark components under the hood- the dataframe API, SQL, Python stored procedures and more. But with the new SnowparkML API, data scientists can take advantage of all that Snowpark affords them, while using common, familiar APIs that match how they do their work today.\n",
    "\n",
    "Now that we have our feature data, let's actually fit an XGBoost model to our features to attempt to predict future sales. We'll fit several different models with different hyperparameters, and then show how we can use the Snowpark Model Registry to select our best-fit model.\n",
    "\n"
   ],
   "id": "8139efe5-f965-46cb-8bd7-6dfa0798ae9d"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkML_SplitData4Modeling_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Split the data into train and test sets\ntrain_df, test_df = input_df.random_split(weights=[0.9, 0.1], seed=98)",
   "id": "c82c4134-289e-4482-b24e-bfc57ef5071c"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkMLAPI_Modeling",
    "collapsed": false
   },
   "source": "## What's happening when you leverage Snowpark ML Modeling API?\n\nLet's run our training job using the SnowparkML Modeling API- this will push down our model training to run on Snowflake, and you'll notice that the type of the model object returend is a SnowparkML XGBClassifier- this has some benefits, but also is fully compatible with the standard sklearn/xgboost model objects.\n\n- The model.fit() function actually creates a temporary stored procedure in the background. This also means that the model training is a single-node operation. Be sure to use a Snowpark Optimized Warehouse if you need more memory. We are just using an XS Standard Virtual Warehouse here, which we created at the beginning of this quickstart.\n- The model.predict() function actually creates a temporary vectorized UDF in the background, which means the input DataFrame is batched as Pandas DataFrames and inference is parallelized across the batches of data. You can check the query history once you execute the following cell to check.\n\nThe Snowpark ML Modeling API enables the use of popular Python ML frameworks, such as scikit-learn and XGBoost, for feature engineering and model training without the need to move data out of Snowflake. \n\nBenefits of Snowpark ML Modeling: \n- **Feature engineering and preprocessing:** Improve performance and scalability with distributed execution for common scikit-learn preprocessing functions. \n- **Model training:** Accelerate model training for scikit-learn, XGBoost and LightGBM models without the need to manually create stored procedures or user-defined functions (UDFs), and leverage distributed hyperparameter optimization (now in public preview).\n\nBehind the scenes, Snowpark ML parallelizes data processing operations by taking advantage of Snowflake’s scalable computing platform.",
   "id": "edac01ae-6e20-41ff-8377-e9fed0014eb7"
  },
  {
   "cell_type": "code",
   "id": "5f3d7302-301c-47a4-b6bc-234cd359ff67",
   "metadata": {
    "language": "python",
    "name": "SnowparkMLAPI",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/DistributedDataProcessing.png\" , decompress=False).read() \n\n# Display the image\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_Modeling_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "from snowflake.ml.modeling.xgboost import XGBRegressor\n# Define the XGBRegressor\nregressor = XGBRegressor(\n    label_cols = LABEL_COLUMNS,\n    input_cols = FEATURE_COLUMN_NAMES,\n    output_cols = OUTPUT_COLUMNS\n)\n\n# Train\n_ = regressor.fit(train_df)\n\n# Predict\n# result = regressor.predict(test_df)",
   "id": "c3179b2b-be59-41e2-961c-449efb2def2a"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "Snowpark_MLAPI_ModelPredictOutput_Code",
    "language": "python",
    "collapsed": false,
    "codeCollapsed": false
   },
   "outputs": [],
   "source": "# Just to illustrate, we can also pass in a Pandas DataFrame to Snowpark ML's model.predict()\nregressor.predict(test_df.to_pandas())",
   "id": "e809f760-f997-44d8-9c84-ee63a29c71d3"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_ModelAccuracy_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Let's analyze the results using Snowpark ML's MAPE\n# Use Snowpark ML metrics to calculate\nfrom snowflake.ml.modeling.metrics import mean_absolute_percentage_error, mean_squared_error\n\n# Predict\nresults = regressor.predict(test_df)\n\n# Calculate MAPE\nmape = mean_absolute_percentage_error(df=results, y_true_col_names=LABEL_COLUMNS, y_pred_col_names=OUTPUT_COLUMNS)\n\n# Calculate MSE\nmse = mean_squared_error(df=results, y_true_col_names=LABEL_COLUMNS, y_pred_col_names=OUTPUT_COLUMNS)\n\nresults.select([*LABEL_COLUMNS, *OUTPUT_COLUMNS]).show()\nprint(f'''Mean absolute percentage error: {mape}''')\nprint(f'''Mean squared error: {mse}''')",
   "id": "85674393-aeed-406c-a802-db292b597454"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_ModelAccuracy_SiS",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Plot actual vs predicted \ng = sns.relplot(data=results[\"SHIFT_SALES\", \"PRED_SHIFT_SALES\"].to_pandas().astype(\"float64\"), x=\"SHIFT_SALES\", y=\"PRED_SHIFT_SALES\", kind=\"scatter\")\ng.ax.axline((0,0), slope=1, color=\"r\")",
   "id": "957b744c-12d9-4e60-9542-5a6fffd26379"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowparkMLAPI_GridSearch"
   },
   "source": [
    "## Snowpark ML's GridSearchCV()\n",
    "\n",
    "Now, let's use Snowpark ML's GridSearchCV() function to find optimal model parameters.\n",
    "\n"
   ],
   "id": "bd32172b-7aab-48bc-9852-8b2ff136ee46"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_GridSearch_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "from snowflake.ml.modeling.model_selection import GridSearchCV\n\ngrid_search = GridSearchCV(\nestimator=XGBRegressor(),\nparam_grid={\n\"n_estimators\":[25, 50],\n\"learning_rate\":[0.4, 0.5],\n},\nn_jobs = -1,\nscoring=\"neg_mean_absolute_percentage_error\",\ninput_cols=FEATURE_COLUMN_NAMES,\nlabel_cols=LABEL_COLUMNS,\noutput_cols=OUTPUT_COLUMNS\n)\n_ = grid_search.fit(train_df)",
   "id": "22005572-4302-43d8-982e-ef903a879024"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_GridSearch_ModelImprovement_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Let's analyze the grid_search results\ngs_results = grid_search.to_sklearn().cv_results_\nn_estimators_val = []\nlearning_rate_val = []\nfor param_dict in gs_results[\"params\"]:\n    n_estimators_val.append(param_dict[\"n_estimators\"])\n    learning_rate_val.append(param_dict[\"learning_rate\"])\nmape_val = gs_results[\"mean_test_score\"]*-1\n\ngs_results_df = pd.DataFrame(data={\n    \"n_estimators\":n_estimators_val,\n    \"learning_rate\":learning_rate_val,\n    \"mape\":mape_val})\n\ng2 = sns.relplot(data=gs_results_df, x=\"learning_rate\", y=\"mape\", hue=\"n_estimators\", kind=\"line\")",
   "id": "0ad4f16a-8baf-470e-b0fc-d7d13e128990"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowparkMLAPI_OptimalModel_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's save our optimal model and its metadata:\n",
    "optimal_model = grid_search.to_sklearn().best_estimator_\n",
    "optimal_n_estimators = grid_search.to_sklearn().best_estimator_.n_estimators\n",
    "optimal_learning_rate = grid_search.to_sklearn().best_estimator_.learning_rate\n",
    "\n",
    "\n",
    "optimal_mape = gs_results_df.loc[(gs_results_df['n_estimators']==optimal_n_estimators) &\n",
    "                        (gs_results_df['learning_rate']==optimal_learning_rate),'mape'].values[0]"
   ],
   "id": "bc418ad8-c865-401e-b5a3-60647ae00954"
  },
  {
   "cell_type": "code",
   "id": "a9ac738d-35d7-430a-a256-b7c0b63aeb39",
   "metadata": {
    "language": "python",
    "name": "cell1",
    "collapsed": false
   },
   "outputs": [],
   "source": "optimal_mape, optimal_n_estimators, optimal_learning_rate",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "ScaleDownWarehouse"
   },
   "source": [
    "## **Scale down your assigned Snowflake compute warehouse.**\n",
    "\n"
   ],
   "id": "8ba3f0db-5bd5-4673-8f31-97887b0a11fe"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "ScaleDownWarehouse_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Decrease size of Snowflake compute warehouse to XSMALL\nsession.sql(\"ALTER WAREHOUSE \" + warehouse + \" SET WAREHOUSE_SIZE = 'XSMALL'\").collect()",
   "id": "7afd018d-eb9f-408d-8b34-c2721cf4bbcb"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Part3_ModelRegistry",
    "collapsed": false
   },
   "source": "# Part 3 - Use Snowpark for MLOps",
   "id": "696b4aad-2ab7-4b60-ac9d-a4d3234a2895"
  },
  {
   "cell_type": "code",
   "id": "7cce1de4-5316-44a7-8993-1e9871a27a3f",
   "metadata": {
    "language": "python",
    "name": "Part3",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Part3.png\" , decompress=False).read() \n\n# Display the image\nst.image(image)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowflakeModelRegistry",
    "collapsed": false
   },
   "source": "## Now let's use Snowflake Model Registry (Snowpark ML Ops)\n\nModel Registry was created to support model management operations including model registration, versioning, metadata and audit trails. Integrated deployment infrastructure for batch inference is a critical ease-of-use feature. Users can deploy ML models for batch inference from the registry directly into a Snowflake Warehouse as a vectorized UDF, or as a service to a customer-specified Compute Pool in Snowpark Container Services.\n\nSnowflake's Model Registry supports SciKitLearn, XGBoost, Pytorch, Tensorflow and MLFlow (via the pyfunc interface) models.\n\nModel Registry allows easy deployment of pre-trained open-source models from providers such as HuggingFace. See this [blog](https://www.snowflake.com/blog/accelerate-ml-workflow-python-snowpark-ml/) for more detailsor the [Snowflake Model Registry documentation](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-mlops-model-registry)",
   "id": "5dbfe1d1-fbf4-429d-9a8f-0b9e2e5dccf3"
  },
  {
   "cell_type": "code",
   "id": "8270739e-eb7d-4a4f-9eb6-fcf1a22de4d1",
   "metadata": {
    "language": "python",
    "name": "ModelRegistry",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/ModelRegistry.png\" , decompress=False).read() \n\n# Display the image\nst.image(image, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "5cdc7c40-1171-47b9-a505-7cd781fc987c",
   "metadata": {
    "language": "python",
    "name": "CreateModelRegistry",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Create the Model Registry and register your initial model\nfrom snowflake.ml.registry import Registry\n\nnative_registry = Registry(session, database_name=\"HOL\", schema_name=schema)\nmodel_name = \"SHIFT_SALES_PREDICTION\"",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "id": "1bde2057-bc12-4554-90d0-4c3b70868f3a",
   "metadata": {
    "language": "python",
    "name": "SnowparkModelRegisty_GetModelToBeRegistered_Code",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Here is the code to get the latest version of the model named \"SHIFT_SALES_PREDICTION\" within the Snowflake Model Registry\ndef get_next_model_version_to_be_registered(model_name:str):\n    def remove_char(lst, char):\n        import re\n        from functools import reduce\n        return reduce(lambda x, y: x + [int(re.sub(char, '', y))], lst, [])\n    model_version_to_be_registered = 0\n    try:\n        m = native_registry.get_model(model_name)\n        model_version_to_be_registered = max(remove_char(m.show_versions()['name'].tolist(),'V'))+1\n    except:\n        pass\n    return model_version_to_be_registered",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_CreateRegisterModel_OriginalModel_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Get sample input data to pass into the registry logging function\nX = train_df.select(FEATURE_COLUMN_NAMES).limit(100)\n\n# Define model name\nmodel_name = model_name\n\n# Let's first log the very first model we trained\nmodel_ver = native_registry.log_model(\n    model_name=model_name,\n    version_name=f'V{get_next_model_version_to_be_registered(model_name)}',\n    model=regressor,\n    sample_input_data=X, # to provide the feature schema\n)\n# Add a description\nmodel_ver.comment = \"This is the initial model of the Shift Sales Price Prediction model.\"",
   "id": "703db0dc-670a-4d44-8459-4fd0f77d3e53"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_RegisterOptimalModel_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Now, let's log the optimal model from GridSearchCV\nmodel_ver2 = native_registry.log_model(\n    model_name=model_name,\n    version_name=f'V{get_next_model_version_to_be_registered(model_name)}',\n    model=optimal_model,\n    sample_input_data=X, # to provide the feature schema\n)\n\n# Add evaluation metric\nmodel_ver2.set_metric(metric_name=\"mean_abs_pct_err\", value=optimal_mape)\n\n# Add a description\nmodel_ver2.comment = \"This is the second iteration of the Shift Sales Price Prediction model \\\n                        where we performed hyperparameter optimization.\"",
   "id": "d6e9ef6e-4b91-4da3-ba51-af552d07dc16"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_ShowVersions_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's confirm model(s) that were added\n",
    "native_registry.get_model(model_name).show_versions()"
   ],
   "id": "a4875ffe-5d80-4bde-b3fa-2b295f3abff9"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_ShowDefaultModel_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We can see what the default model is when we have multiple versions with the same model name:\n",
    "native_registry.get_model(model_name).default.version_name"
   ],
   "id": "d511dc1c-9cb1-4572-b90d-6940eebfdf49"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_ModelInference_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now we can use the default version model to perform inference.\n",
    "model_ver = native_registry.get_model(model_name).version('V0')\n",
    "result_sdf = model_ver.run(test_df, function_name=\"predict\")\n",
    "result_sdf.show()"
   ],
   "id": "462fcbcc-479b-47b8-a99a-6e7cc518276d"
  },
  {
   "cell_type": "markdown",
   "id": "b6853926-5ee3-46c0-9856-661755ea3451",
   "metadata": {
    "name": "ViewRegisteredModels",
    "collapsed": false
   },
   "source": "## View Registered Models\n\nLet's look at the Snowflake Model Registry UI. It is a unified place in Snowsight UI where all models registered in the Model Registry can be found and explored. \n\nTo navigate to the Model Registry, click on “AI & ML” in the left-nav, and select “Models”. This will display all available models in the Model Registry across all DBs/Schemas that your role has access to.\n\nCurrently, Model Registry will display user-created models from Snowpark ML, Snowpark, or models sourced from external ML platforms that are registered in Snowflake. In the future, Model Registry will also support other model types such as Cortex ML, Doc AI, Cortex Fine Tuning etc. \n\nTo explore the details of a model, click on the corresponding row in the listing. This will open the details page for the selected model as shown below. \n\nKey model details such as description, any tags that are applied to the model, and a table of all versions of the model are displayed. \n\nModel Version metadata such as metrics can be found here. This page also shows the available model methods for invocation along with the inputs and outputs of those functions.\n"
  },
  {
   "cell_type": "code",
   "id": "5210af9f-1e35-45e2-a813-040935b2fc44",
   "metadata": {
    "language": "python",
    "name": "ModelRegistryUI",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image1=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/ModelRegistered.png\" , decompress=False).read() \nimage2=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Model.png\" , decompress=False).read() \n\n# Display the image\nst.image(image1, width=1000)\nst.image(image2, width=1000)\n",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "9def2b43-3cf2-4772-bdde-9e8484a157b8",
   "metadata": {
    "name": "TestModel",
    "collapsed": false
   },
   "source": "Now that our model is built and deployed, let's see it in action! We will find the best place to park in Vancouver for tomorrow morning's shift."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "SnowflakeModelRegistry_ModelInferenceCheck_Code",
    "language": "python",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Check model predictions for holdout data SHIFT_SALES predictions for Location_IDs in Vancouver\ndate_tomorrow_df = oe_df.filter(\n    (F.col(\"shift_sales\").isNull())\n    & (F.col(\"shift_oe\") == 1)\n    & (F.col(\"city\") == \"Vancouver\")\n)\nresult_sdf = regressor.predict(date_tomorrow_df)\nresult_sdf.show()",
   "id": "4b34593d-35ea-4a93-8c46-e8d871aa5a2d"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "SnowflakeModelRegistry_ModelInferenceCheck_SiS"
   },
   "source": [
    "## Visualize on a Map\n",
    "\n",
    "The red and yellow areas indicate higher predicted sales locations and the green zones indicate lower predicted sales. We will use this insight to ensure that our drivers are parking at the high-value locations. Value: Updated predictions readily available to drive towards our corporate goals.\n",
    "\n"
   ],
   "id": "88624ec7-88ae-4ce4-b3d7-2322b6e374f0"
  },
  {
   "cell_type": "code",
   "id": "b50ec110-4223-4d70-b46d-4e8622fcf462",
   "metadata": {
    "language": "python",
    "name": "SnowparkML_VisualizePredictions_SiS_Code",
    "collapsed": false
   },
   "outputs": [],
   "source": "# Pull location predictions into a pandas DataFrame\npredictions_df = result_sdf.to_pandas()\npredictions_df.head()\n\n# Visualize on a map\nst.map(predictions_df)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "name": "Part4_SiSApp",
    "collapsed": false
   },
   "source": "# Part 4 - Create a SiS application to use predicted outputs\n\nCreate a SiS application for local managers to identify where to place daily food trucks. \n\nSee completed [SiS App](https://app.snowflake.com/sfsenorthamerica/sfhol/#/streamlit-apps/HOL.SCHEMA0.ZXGOF2KZL26CGAA8?ref=snowsight_shared) using role=PUBLIC\n\n",
   "id": "fe5ff0fb-e2c7-4aeb-b9e7-805cfae17424"
  },
  {
   "cell_type": "code",
   "id": "de5f573c-e7a9-4ac6-87dd-9261c2f958f1",
   "metadata": {
    "language": "python",
    "name": "Part4",
    "collapsed": false,
    "codeCollapsed": true
   },
   "outputs": [],
   "source": "image1=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/Part4.png\" , decompress=False).read() \nimage2=session.file.get_stream(\"@HOL.PUBLIC.ASSETS/SiSapp.png\" , decompress=False).read() \n\n# Display the image\nst.image(image1, width=1000)\n\nst.subheader(\"Here's a picture of the running SiS app:\")\nst.image(image2, width=1000)",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "011b641a-958b-47fa-99ae-d0e83e4f2277",
   "metadata": {
    "name": "Part4_SiS",
    "collapsed": false
   },
   "source": "- Create new SiS app.\n- Replace existing code with SiS_application.py code listed below.\n- Update schema variable in line 29.\n- Include required pydeck and snowflake-ml-python package.\n- Run the SiS application."
  },
  {
   "cell_type": "markdown",
   "id": "38d24cfb-06c3-453a-8539-daeb8fac1747",
   "metadata": {
    "name": "SiSApp",
    "collapsed": false
   },
   "source": "```\n# Import Python packages\nimport streamlit as st\nimport pydeck as pdk\nimport numpy as np\n\n# Import Snowflake modules\nfrom snowflake.snowpark import Session\nimport snowflake.snowpark.functions as F\nfrom snowflake.snowpark import Window\nfrom snowflake.snowpark.context import get_active_session\nfrom snowflake.ml.registry import Registry\nimport snowflake.ml.modeling.preprocessing as snowmlpp\n\n# Set your schema variable\nschema = \"SCHEMA0\"\n\n# Set Streamlit page config\nst.set_page_config(\n    page_title=\"Streamlit App: Snowpark 101\", \n    page_icon=\":truck:\",\n    layout=\"wide\",\n)\n\n# Add header and a subheader\nst.header(\"Predicted Shift Sales by Location\")\nst.subheader(\"Data-driven recommendations for food truck drivers.\")\n\n# Connect to Snowflake\n# session = init_connection()\nsession = get_active_session()\n\n# Create input widgets for cities and shift\nwith st.container():\n    col1, col2 = st.columns(2)\n    with col1:\n        # Drop down to select city\n        city = st.selectbox(\n            \"City:\",\n            session.table(\"HOL.\"+schema+\".SHIFT_SALES_V\")\n            .select(\"city\")\n            .distinct()\n            .sort(\"city\"),\n        )\n\n    with col2:\n        # Select AM/PM Shift\n        shift = st.radio(\"Shift:\", (\"AM\", \"PM\"), horizontal=True)\n\n    n_trucks = st.selectbox('How many food trucks would you like to schedule today?', np.arange(1,10))\n\n    if n_trucks > 1:\n        range = st.slider('What is the minimum distance in kilometers between food trucks?', 0, 20, 1)\n        st.write('You are requesting a minimum distance of ', range, 'km')\n        st.write('Click **:blue[Update]** to get the ', n_trucks, ' highest predicted Shift_Sales food truck locations.')\n    else:\n        st.write('Click **:blue[Update]** to get one food truck location predicted to have the Shift_Sales')\n        \n# Get predictions for city and shift time\ndef get_predictions(city, shift):\n    # Get data and filter by city and shift\n    snowpark_df = session.table(\n        \"HOL.\"+schema+\".SHIFT_SALES_V\"\n    ).filter((F.col(\"shift\") == shift) & (F.col(\"city\") == city))\n\n    # Get rolling average\n    window_by_location_all_days = (\n        Window.partition_by(\"location_id\")\n        .order_by(\"date\")\n        .rows_between(Window.UNBOUNDED_PRECEDING, Window.CURRENT_ROW - 1)\n    )\n\n    snowpark_df = snowpark_df.with_column(\n        \"avg_location_shift_sales\",\n        F.avg(\"shift_sales\").over(window_by_location_all_days),\n    ).cache_result()\n\n    # Get tomorrow's date\n    date_tomorrow = (\n        snowpark_df.filter(F.col(\"shift_sales\").is_null())\n        .select(F.min(\"date\"))\n        .collect()[0][0]\n    )\n\n    # Filter to tomorrow's date\n    snowpark_df = snowpark_df.filter(F.col(\"date\") == date_tomorrow)\n\n    # Impute\n    snowpark_df = snowpark_df.fillna(value=0, subset=[\"avg_location_shift_sales\"])\n\n    for colname in snowpark_df.columns:\n        new_colname = str.upper(colname)\n        snowpark_df = snowpark_df.with_column_renamed(colname, new_colname)\n\n    # Encode\n    snowpark_df = snowpark_df.with_column(\"shift_oe\", F.iff(F.col(\"shift\") == \"AM\", 0, 1))\\\n    .with_column(\"shift_oe\", F.iff(F.col(\"shift\") == \"PM\", 1, 0))\n\n    # Scale\n    mm_target_columns = [\"CITY_POPULATION\"]\n    mm_target_cols_out = [\"CITY_POPULATION_NORM\"]\n    snowml_mms = snowmlpp.MinMaxScaler(input_cols=mm_target_columns, \n                                       output_cols=mm_target_cols_out)\n    snowml_mms.fit(snowpark_df)\n    snowpark_df = snowml_mms.transform(snowpark_df)\n\n    # Get all features\n    feature_cols = [\"SHIFT_OE\", \n                    \"CITY_POPULATION_NORM\", \n                    \"MONTH\", \n                    \"DAY_OF_WEEK\",\n                    \"LATITUDE\",\n                    \"LONGITUDE\",\n                    \"AVG_LOCATION_SHIFT_SALES\",\n                    \"LOCATION_ID\"]\n\n    snowpark_df = snowpark_df.select(feature_cols)\n    native_registry = Registry(session=session, database_name=\"HOL\", schema_name=schema)\n    model_ver = native_registry.get_model(\"SHIFT_SALES_PREDICTION\").version('v0')\n    result_sdf = model_ver.run(snowpark_df, function_name=\"predict\")\n    return result_sdf\n\n# Update predictions and plot when the \"Update\" button is clicked\nif st.button(\":blue[Update]\"):\n    # Get predictions\n    with st.spinner(\"Getting predictions...\"):\n        predictions_sdf = get_predictions(city, shift)\n        predictions = predictions_sdf.to_pandas()\n\n    # Plot on a map\n    st.subheader(\"Predicted Shift Sales for position\")\n    predictions[\"PRED_SHIFT_SALES\"].clip(0, inplace=True)\n    st.pydeck_chart(\n        pdk.Deck(\n            map_style=None,\n            initial_view_state=pdk.ViewState(\n                latitude=predictions[\"LATITUDE\"][0],\n                longitude=predictions[\"LONGITUDE\"][0],\n                zoom=11,\n                pitch=50,\n            ),\n            layers=[\n                pdk.Layer(\n                    \"HexagonLayer\",\n                    data=predictions,\n                    get_position=\"[LONGITUDE, LATITUDE]\",\n                    radius=200,\n                    elevation_scale=4,\n                    elevation_range=[0, 1000],\n                    pickable=True,\n                    extruded=True,\n                ),\n                pdk.Layer(\n                    \"ScatterplotLayer\",\n                    data=predictions,\n                    get_position=\"[LONGITUDE, LATITUDE]\",\n                    get_color=\"[200, 30, 0, 160]\",\n                    get_radius=200,\n                ),\n            ],\n        )\n    )\n\n    max_x = predictions.loc[predictions[\"PRED_SHIFT_SALES\"].idxmax()]\n    st.write(\"Maximum Predicted Sales are expected at the following location:\", max_x)\n\n    #st.dataframe(predictions_sdf)\n    location_id = max_x[\"LOCATION_ID\"]\n    lat = max_x[\"LATITUDE\"]\n    long = max_x[\"LONGITUDE\"]\n    st.subheader(\"The following chart is generated using the st_point and st_distance Snowflake Geospatial features\")\n\n    if n_trucks == 1:\n        st.write(\"Have your only food truck positioned at Location ID \", location_id, \" to maximize SHIFT_SALES\")\n    elif n_trucks > 1:\n        best_locations = [location_id]\n        available_locations_sdf = predictions_sdf\n        st_distance = F.function('st_distance')\n        st_point = F.function('st_point')\n        for truck_n in np.arange(0,n_trucks - 1):\n            available_locations_sdf = available_locations_sdf.with_column(\"DISTANCE_TO_TRUCK\", \n                                        st_distance(\n                                            st_point(F.lit(float(long)), F.lit(float(lat))),\n                                            st_point(F.col(\"LONGITUDE\"), F.col(\"LATITUDE\"))\n                                        )/1609\n                                       ).filter(F.col(\"DISTANCE_TO_TRUCK\") >= range/1.609).order_by(\"PRED_SHIFT_SALES\", ascending=False)\n            max_x = available_locations_sdf.limit(1).to_pandas()\n            try:\n                location_id = max_x[\"LOCATION_ID\"].iloc[0]\n                lat = max_x[\"LATITUDE\"].iloc[0]\n                long = max_x[\"LONGITUDE\"].iloc[0]\n            except:\n                break\n            best_locations.append(location_id)\n\n        selected_locations = predictions[predictions[\"LOCATION_ID\"].isin(best_locations)]\n        st.map(selected_locations)\n        st.dataframe(selected_locations)\n```"
  }
 ]
}