lectures/24_deployment-conclusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 24: Deployment and conclusion\n",
    "\n",
    "UBC 2022-23\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports, LOs, Announcements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import cross_validate, train_test_split\n",
    "from sklearn.pipeline import Pipeline, make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning objectives\n",
    "\n",
    "- Describe the goals and challenges of model deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Announcements\n",
    "\n",
    "- Last lecture today!\n",
    "- hw9 is due tomorrow 11:59pm \n",
    "- Sorry for the delay in getting back the grades for hw7 and hw8. The solutions are available in Canvas. \n",
    "- No tutorials this week.\n",
    "- Next Monday the usual morning TA office hours will be there. \n",
    "- I've scheduled extra OH. 2-hour slot in my office (ICCS 237) on Thursday between 11 to 1pm and a 1-hour Zoom slot on Monday between 4 to 5pm. \n",
    "- I'll give you some time for formal course evaluations during today's lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Final exam\n",
    "\n",
    "- Our final exam is on **Dec. 15th at 7pm**\n",
    "- We are split in two rooms: **PHRM 1101 and PHRM 1201**. \n",
    "    - See the schedule [here](https://students.ubc.ca/enrolment/exams/exam-schedule)    \n",
    "- Where are you supposed to go to write your exam?\n",
    "    - There is alphabetical split. \n",
    "    - ABR-MU: PHRM 1101: Anyone with a family name that begins with M or before would go to PHRM 1101\n",
    "    - NGU-ZHU: PHRM 1201: Anyone with a family name that begins with N or after would go to PHRM 1201\n",
    "- An update on the internet in our exam rooms: I've checked with UBC IT about the internet connection in our exam rooms. We should be fine based on the following response. \n",
    "> Each of the large lecture theatres in the Pharmacy Building have capacity for 300 simultaneous client devices. The Wireless Access Points serving these rooms are 802.11ax (WIFI 6) capable and were upgraded in Summer 2022.   \n",
    "- Cummulative\n",
    "- Open book. You can refer to the notes.\n",
    "- Please bring your computer and charger. All of it is going to be on Canvas. \n",
    "- A combination of: Multiple choice questions, short answer questions, simple coding questions\n",
    "- No communication/collaboration \n",
    "- No public Piazza posts       \n",
    "- More details are posted on Piazza post @568. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model deployment (~30 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Try out this moment predictor\n",
    "\n",
    "https://render-test-xygn.onrender.com\n",
    "\n",
    "- In this lecture I will show you how to set up/develop this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### What is deployment?\n",
    "\n",
    "- After we train a model, we want to use it!\n",
    "- The user likely does not want to install your Python stack, train your model.\n",
    "- You don't necessarily want to share your dataset.\n",
    "- So we need to do two things:\n",
    "\n",
    "1. Save/store your model for later use.\n",
    "2. Make the saved model conveniently accessible.\n",
    "\n",
    "We will use [Joblib](https://joblib.readthedocs.io/) for (1) and [Flask](https://flask.palletsprojects.com/) & [render](https://render.com/) for (2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Requirements (I already did these)\n",
    "\n",
    "- render account. Register [here](https://render.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More python installations:\n",
    "\n",
    "```\n",
    "pip install Flask\n",
    "pip install Flask-WTF\n",
    "pip install joblib\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Demo: Deploying moment classification model  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Building a model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Recall the multi-class classification problem using the [HappyDB](https://www.kaggle.com/ritresearch/happydb) corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wid</th>\n",
       "      <th>reflection_period</th>\n",
       "      <th>original_hm</th>\n",
       "      <th>moment</th>\n",
       "      <th>modified</th>\n",
       "      <th>num_sentence</th>\n",
       "      <th>target</th>\n",
       "      <th>predicted_category</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>27676</th>\n",
       "      <td>206</td>\n",
       "      <td>24h</td>\n",
       "      <td>We had a serious talk with some friends of our...</td>\n",
       "      <td>We had a serious talk with some friends of our...</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "      <td>bonding</td>\n",
       "      <td>bonding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27678</th>\n",
       "      <td>45</td>\n",
       "      <td>24h</td>\n",
       "      <td>I meditated last night.</td>\n",
       "      <td>I meditated last night.</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>leisure</td>\n",
       "      <td>leisure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27697</th>\n",
       "      <td>498</td>\n",
       "      <td>24h</td>\n",
       "      <td>My grandmother start to walk from the bed afte...</td>\n",
       "      <td>My grandmother start to walk from the bed afte...</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>affection</td>\n",
       "      <td>affection</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27705</th>\n",
       "      <td>5732</td>\n",
       "      <td>24h</td>\n",
       "      <td>I picked my daughter up from the airport and w...</td>\n",
       "      <td>I picked my daughter up from the airport and w...</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>bonding</td>\n",
       "      <td>affection</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27715</th>\n",
       "      <td>2272</td>\n",
       "      <td>24h</td>\n",
       "      <td>when i received flowers from my best friend</td>\n",
       "      <td>when i received flowers from my best friend</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "      <td>bonding</td>\n",
       "      <td>bonding</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        wid reflection_period  \\\n",
       "hmid                            \n",
       "27676   206               24h   \n",
       "27678    45               24h   \n",
       "27697   498               24h   \n",
       "27705  5732               24h   \n",
       "27715  2272               24h   \n",
       "\n",
       "                                             original_hm  \\\n",
       "hmid                                                       \n",
       "27676  We had a serious talk with some friends of our...   \n",
       "27678                            I meditated last night.   \n",
       "27697  My grandmother start to walk from the bed afte...   \n",
       "27705  I picked my daughter up from the airport and w...   \n",
       "27715        when i received flowers from my best friend   \n",
       "\n",
       "                                                  moment  modified  \\\n",
       "hmid                                                                 \n",
       "27676  We had a serious talk with some friends of our...      True   \n",
       "27678                            I meditated last night.      True   \n",
       "27697  My grandmother start to walk from the bed afte...      True   \n",
       "27705  I picked my daughter up from the airport and w...      True   \n",
       "27715        when i received flowers from my best friend      True   \n",
       "\n",
       "       num_sentence     target predicted_category  \n",
       "hmid                                               \n",
       "27676             2    bonding            bonding  \n",
       "27678             1    leisure            leisure  \n",
       "27697             1  affection          affection  \n",
       "27705             1    bonding          affection  \n",
       "27715             1    bonding            bonding  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/cleaned_hm.csv\", index_col=0)\n",
    "sample_df = df.dropna()\n",
    "sample_df.head()\n",
    "sample_df = sample_df.rename(\n",
    "    columns={\"cleaned_hm\": \"moment\", \"ground_truth_category\": \"target\"}\n",
    ")\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "affection           4810\n",
       "achievement         4276\n",
       "bonding             1750\n",
       "enjoy_the_moment    1514\n",
       "leisure             1306\n",
       "nature               252\n",
       "exercise             217\n",
       "Name: target, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_df[\"target\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a multiclass classification problem! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "train_df, test_df = train_test_split(sample_df, test_size=0.2, random_state=123)\n",
    "X_train_happy, y_train_happy = train_df[\"moment\"], train_df[\"target\"]\n",
    "X_test_happy, y_test_happy = test_df[\"moment\"], test_df[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr = make_pipeline(\n",
    "    CountVectorizer(stop_words=\"english\"),\n",
    "    LogisticRegression(max_iter=2000),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_lr.fit(X_train_happy, y_train_happy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9528318584070796"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr.score(X_train_happy, y_train_happy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8169911504424778"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr.score(X_test_happy, y_test_happy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training on the full corpus\n",
    "\n",
    "- Ideally, you'll try all different models, fine tune the most promising models and deploy the best performing model.  \n",
    "- Sometimes before deploying a model people train it on the full dataset. \n",
    "    - This is probably a good idea, because more data is better.\n",
    "    - It's also a little scary, because we can't test this new model.\n",
    "- Here I'm just deploying the model trained on the training set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Saving the model \n",
    "\n",
    "- If we want to deploy a model, we need to save it. \n",
    "- we are using `joblib` for that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "\n",
    "with open(\"web_api/moment_predictor.joblib\", \"wb\") as f:\n",
    "    joblib.dump(pipe_lr, f)\n",
    "with open(\"web_application/moment_predictor.joblib\", \"wb\") as f:\n",
    "    joblib.dump(pipe_lr, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define a function that accepts input data as a dictionary and returns a prediction:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading our saved model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function to get predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def return_prediction(model, text):\n",
    "    prediction = model.predict([text])[0]\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'affection'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = joblib.load(\"web_api/moment_predictor.joblib\") # load the saved model\n",
    "text = \"I love my students!\"\n",
    "return_prediction(model, text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function appears in the `app.py` that we'll be using shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Setting up a directory structure and environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We need a specific directory structure to help us easily deploy our machine learning model. \n",
    "- This is already set up in this repo.\n",
    "\n",
    "```\n",
    "├── web_api\n",
    "│   └── moment_predictor.joblib  # this is the machine learning model we have built locally\n",
    "│   └── app.py  # the file that defines our flask API\n",
    "│   └── Procfile  # required to help start flask app\n",
    "│   └── requirements.txt  # file containing required packages\n",
    "│   \n",
    "└── web_application\n",
    "    └── moment_predictor.joblib  # this is the machine learning model we have built locally\n",
    "    └── app.py  # the file that defines our flask API\n",
    "    └── Procfile  # required to help start flask app\n",
    "    └── requirements.txt  # file containing required packages\n",
    "    └── templates  # contains HTML templates to help us build the web application\n",
    "    │   └── style.css  # css template to be used in web application\n",
    "    └── static  # this subdirectory contains CSS style sheets\n",
    "        └── home.html  # html template to be used in web application\n",
    "        └── prediction.html  # html template to be used in web application\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two options for deploying our moment prediction model. We can:\n",
    "\n",
    "1. Build a web application (app) with a HTML user-interface that interacts directly with our model.\n",
    "2. Develop a RESTful (REST stands for REpresentational State Transfer) web API that accepts HTTP requests in the form of input data and returns a prediction.\n",
    "\n",
    "We'll explore both options below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Building and deploying a web app\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |        ❓      |                          |\n",
    "| API  |                          |                          |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Flask can create entire web applications.\n",
    "- We need to link our code to some html and css to create our web application.\n",
    "- We will use Flask to create a html form, accept data submitted to the form, and return a prediction using the submitted data. \n",
    "- Again, I won't go into too much detail here, but we can open up `web_application/` and take a quick look."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- If you want to learn more about Flask, see:\n",
    "  - [Flask tutorial video series by Corey Schafer](https://www.youtube.com/playlist?list=PL-osiE80TeTs4UjLw5MM6OjgkjFeUxCYH)\n",
    "  - [Flask docs](https://flask.palletsprojects.com/en/1.1.x/)\n",
    "  - [Flask tutorial by Miguel Grinberg](https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's try `web_application/app.py` that handles this part.\n",
    "- We can open it up here in Jupyter and take a look.\n",
    "- If we run `python app.py` we'll bring it to life."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Web app on local server\n",
    "\n",
    "1. Go to the terminal. \n",
    "2. Navigate to the `web_application` directory. \n",
    "3. Run the following to make the app alive: `python app.py`. \n",
    "    - If you get an error, you may need to install those extra packages and make sure you have the environment loaded.\n",
    "4. Now you should be able to access the app at: `http://127.0.0.1:5000/` or `http://localhost:5000/`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Web app on a real server\n",
    "\n",
    "- If you want people to use your app/model, you would probably want to put it on a real server and not your laptop so that it's live all the time. \n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs)|\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |       ❓       |\n",
    "| API  |                          |                          |\n",
    "\n",
    "- We'll use [Render](https://render.com/) for this. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Render set-up (I already did these):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "1. Go to [render](https://dashboard.render.com/), log-in\n",
    "2. Click on \"New\" and create a web service.  \n",
    "\n",
    "![](img/render-new-app.png)\n",
    "<!-- <img src=\"img/render-new-app.png\" width=\"600\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Demo of creating a new web service\n",
    "- Create an associated GitHub repository. \n",
    "- I've put the contents from `web_application` folder [in this repository](https://github.com/kvarada/ml-deployment-example).\n",
    "- Then you connect that repository to your new web service in render. \n",
    "\n",
    "![](img/github-to-render.png)\n",
    "- Once you fill in all the information, it'll create an environment for you using `requirement.txt` and if everything goes well, your web service will be live.\n",
    "- You'll see a link at the top which you can share with your friends. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Try it out for your own machine learning applications! \n",
    "- This is nice! If you develop a model and you want your friends to try it out without installing anything on their local computers, you can do this.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### API on the localhost \n",
    "\n",
    "- Often you want other people to be able to use your models in their applications.\n",
    "- We can do this by creating an **API**. \n",
    "- If you don't know what an API is, that's OK.\n",
    "  - For our purposes, it's something that exists at a particular address, that can accept information and return information.\n",
    "  - Sort of like a function but not Python-specific and potentially accessible by anyone on the internet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |        ✅       |\n",
    "| API  |       ❓      |                          |\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Go to the terminal. \n",
    "- Navigate to `web_api` folder in this repo.\n",
    "- Run the following to make the api alive: `python app.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that for more complex applications, you may choose to containerize everything in a Docker container to deploy to render)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sending a request to the API\n",
    "\n",
    "- We have a RESTful (REST stands for REpresentational State Transfer) web API that accepts HTTP requests in the form of input data and returns a prediction.\n",
    "- Now you can send requests to the API and get predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -d '{\"text\":\"The students will be submitting their last homework assignment today!\"}' \\\n",
    "      -H \"Content-Type: application/json\" \\\n",
    "      -X POST http://localhost:5000/predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `curl` (stands for client URL) is a tool for transferring data using various network protocols.\n",
    "- Okay, so we have a working API running on localhost, but we don't want to host this service on my laptop!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Deploying the API on a server\n",
    "\n",
    "- We now want to deploy it on a \"real\" server so others can send it requests. \n",
    "- We will use render to deploy our app but you could also use other services such as AWS.\n",
    "\n",
    "|      | on localhost (my laptop) | on server (the interwebs) |\n",
    "|------|--------------------------|--------------------------|\n",
    "| app  |              ✅            |        ✅       |\n",
    "| API  |       ✅      |                  ❓        |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Follow the instructions [here](https://render.com/docs/api) to try the API on server. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Discussion\n",
    "\n",
    "- There are many ways to deploy a model; a RESTful API is very common and convenient. \n",
    "- As you can see, a simple deployment is fairly straightward. \n",
    "- However, there may be other considerations such as:\n",
    "  - Privacy/security\n",
    "  - Scaling to large number of requests\n",
    "  - Error handling\n",
    "  - Real-time / speed\n",
    "  - Low-resource environments (e.g. edge computing)\n",
    "  - etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (~10 min)\n",
    "\n",
    "- We'll take a longer break today.\n",
    "- Consider taking this time to fill out the instructor/TA evaluations if you haven't already.\n",
    "    - Evaluation link: https://canvas.ubc.ca/courses/101888/external_tools/4732\n",
    "\n",
    "![](img/eva-coffee.png)\n",
    "\n",
    "- Here is [Mike's post on these evaluations](https://www.reddit.com/r/UBC/comments/k18qj7/teaching_evaluations_the_good_the_bad_and_the_ugly/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Course review / conclusion (~20 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/ml-workflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning objectives\n",
    "\n",
    "Here are the course learning outcomes: \n",
    "\n",
    "1. Identify problems that may be addressed with machine learning.\n",
    "2. Select the appropriate machine learning tool for a problem.\n",
    "3. Transform data of various types into usable features.\n",
    "4. Apply standard tools implementing supervised and unsupervised learning techniques.\n",
    "5. Describe core differences between training, validation, and testing regimes.\n",
    "6. Effectively communicate the results of a machine learning pipeline.\n",
    "7. Be realistic about the limitations of individual approaches and machine learning as a whole. \n",
    "8. Identify and avoid scenarios in which training and testing data are accidentally mixed (the \"Golden Rule\").\n",
    "9. Employ good habits for applying ML, such as starting an analysis with a baseline estimator.\n",
    "10. Create reproducible workflows and pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How did we do? \n",
    "- Hopefully OK, except we skipped the last point. \n",
    "- There is a [new course](https://ubc-dsci.github.io/dsci-310-student/) on this topic. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What did we cover?\n",
    "\n",
    "I see the course roughly like this (not in order):\n",
    "\n",
    "Part 1: Supervised learning on tabular data\n",
    "\n",
    "- Overfitting, train/validation/test/deployment, cross-validation\n",
    "- Feature preprocessing, pipelines, imputation, OHE, etc\n",
    "- The Golden Rule, various ways to accidentally violate it\n",
    "- Classification metrics: confusion matrix, precision/recall, ROC, AUC\n",
    "- Regression metrics: MSE, MAPE\n",
    "- Feature importances, feature selection\n",
    "- Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- A bunch of models: \n",
    "  - baselines\n",
    "  - linear models (ridge, logistic regression)\n",
    "  - KNNs and RBF SVMs\n",
    "  - tree-based models (random forest, gradient boosted trees)\n",
    "  - Ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Part 2: Other data types (non-tabular)\n",
    "\n",
    "- Clustering: K-Means, DBSCAN\n",
    "- Recommender systems \n",
    "- Computer vision with pre-trained deep learning models (high level)\n",
    "- Language data, text preprocessing, embeddings, topic modeling\n",
    "- Time series\n",
    "- Right-censored data / survival analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Part 3: Communication and Ethics\n",
    "\n",
    "- Ethics for ML\n",
    "- Communicating your results \n",
    "- ML skepticism"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some key takeaways\n",
    "\n",
    "Some useful guidelines:\n",
    "\n",
    "- Do train-test split right away and only once\n",
    "- Don't look at the test set until the end\n",
    "- Don't call `fit` on test/validation data\n",
    "- Use pipelines\n",
    "- Use baselines\n",
    "- Do not be overconfident about your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Difference between Statistics and Machine Learning\n",
    "- There is no clear boundary. \n",
    "- But loosely\n",
    "    - in statistics the emphasis is on inference (usually framed as hypothesis testing)\n",
    "    - in machine learning the emphasis is on prediction and generalization. We are primarily interested in using our model on new unseen examples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recipe to approach a supervised learning problem with tabular data \n",
    "\n",
    "#### Understanding the problem\n",
    "\n",
    "1. Have a long conversation with the stakeholder(s) who will be using your pipeline. \n",
    "2. Have a long conversation with the person(s) who collected the data.\n",
    "3. Think about \n",
    "    - what do you want to accomplish?\n",
    "    - whether you really need to use machine learning for the problem or not\n",
    "    - how do you want to frame the problem as a supervised machine learning problem? What do you want to predict? \n",
    "    - how do you plan to measure your success? \n",
    "    - is there a baseline? \n",
    "    - is there any operating point? \n",
    "    - what will it buy the stakeholders if you improve over their current baseline?  \n",
    "4. Think about the ethical implications - are you sure you want to do this project? If so, should ethics guide your approach?\n",
    "\n",
    "\n",
    "#### Initial analysis, EDA, preprocessing\n",
    "1. Random train-test split with fixed random seed; do not touch the test data until Step 16.\n",
    "2. Exploratory data analysis, outlier detection.\n",
    "3. Choose a scoring metric -> higher values should make you & your stakeholders happier.\n",
    "4. Feature engineering --> Create new features which might be useful to solve the prediction problem. This is typically a time-consuming process. Also, sometimes you get an idea of good features only after looking at your model results. So it's an iterative process. \n",
    "5. Fit a baseline model, e.g. `DummyClassifier` or `DummyRegressor`.\n",
    "6. Create a preprocessing pipeline. \n",
    "7. (Optional) Incorporate feature selection. \n",
    "\n",
    "#### Model building \n",
    "1. Try a linear model, e.g. `LogisticRegression` for classification or `Ridge`; tune hyperparameters with CV.\n",
    "2. Try other sensible model(s), e.g. LightGBM; tune hyperparameters with CV.\n",
    "3. For each model, look at sub-scores from the folds of cross-validation to get a sense of \"error bars\" on the scores.\n",
    "4. Pick a few reasonable models. Best CV score is a reasonable metric, though you may choose to favour simpler models.\n",
    "5. Carry out hyperparameter optimization for these models, paying attention to whether you are not susceptible to optimization bias. \n",
    "6. Try averaging and stacking to examine whether you improve the results over  your best-performing models.  \n",
    "\n",
    "#### Model transparency and interpretation \n",
    "1. Look at feature importances.\n",
    "2. (optional) Perform some more diagnostics like confusion matrix for classification, or \"predicted vs. true\" scatterplots for regression.\n",
    "3. (optional) Try to calibrate the uncertainty/confidence outputted by your model.\n",
    "4. Test set evaluation.\n",
    "5. Model transparency. Take a random sample from the test data and try to explain your predictions using SHAP plots. \n",
    "6. Question everything again: validity of results, bias/fairness of trained model, etc.\n",
    "7. Concisely summarize your results and discuss them with stakeholders.\n",
    "8. (optional) Retrain on all your data.\n",
    "9. Deployment & integration.\n",
    "\n",
    "PS: the order of steps is approximate, and some steps may need to be repeated during prototyping, experimentation, and as needed over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "The order of steps is approximate, and some steps may need to be repeated during prototyping, experimentation, and as needed over time. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What would I do differently?\n",
    "\n",
    "Lots of room for improvement. Here are some things on my mind.  \n",
    "- Flipped classroom in a more effective way in the first part of the course. \n",
    "- More demos during the lecture time\n",
    "- Worksheets/Practice questions during tutorials \n",
    "- Moving some weight from final exam to a course project \n",
    "- Add more interactive components in the lectures\n",
    "- Some material to cover: dealing with outliers, data collection \n",
    "\n",
    "I'm sure you have other suggestions - feel free to drop me an email or drop them in the course evaluations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What next? \n",
    "\n",
    "If you want to further develop your machine learning skills: \n",
    "- Practice! \n",
    "    - Work on your own projects\n",
    "- Work hard and be consistent.   \n",
    "- If you are interested in research in machine learning\n",
    "    - Take CPSC 340. If you do not have the required prereqs you can try to audit it. \n",
    "    - Get into the habit of reading papers and replicating results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 330 vs. 340\n",
    "\n",
    "- We are hoping lots of people will take both courses.\n",
    "- There is some overlap but not a crazy amount.\n",
    "- If you want to learn how these methods work under the hood, CPSC 340 will give you a lot of that, such as:\n",
    "  - Implementing `Ridge.fit()` from scratch\n",
    "  - Mathematically speaking, what is `C` in `LogisticRegression`?\n",
    "  - How fast do these algorithms run in terms of the number of rows and columns of your dataset? \n",
    "  - Etc.\n",
    "- There are also a bunch of other methods covered. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusion & farewell\n",
    "\n",
    "That's all, folks. We made it! Good luck on your final exam! When you get a chance, please let me know what worked for you and what didn't work for you in this course.   \n",
    "\n",
    "![](img/eva-congrats.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}