commercial-blocks-classification/classifying-commercial-blocks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying Commecial Blocks with Microsoft Azure\n",
    "_**Comparing Automated Machine Learning with three standard Scikit Learn Models.**_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "    1. [Accessing the Azure Workspace](#Accessing-the-Azure-Workspace)\n",
    "    1. [Configuring the Azure Workspace](#Configuring-the-Azure-Workspace)\n",
    "    1. [Importing Libraries](#Importing-Libraries)\n",
    "    1. [Creating an Experiment](#Creating-an-Experiment)\n",
    "1. [Data](#Data)\n",
    "1. [Classifying with Scikit Learn](#Classifying-with-SciKit-Learn)\n",
    "1. [Classifying with Automated Machine Learning](#Classifying-with-Automated-Machine-Learning)\n",
    "1. [Results](#Results)\n",
    "1. [Finding the Best Classification Model](#Finding-the-Best-Classification-Model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "**The Task:** Classify commercial blocks from TV news segments (+1 commercial, -1 Non-commercial).\n",
    "\n",
    "This classification model runs a dataset of broadcast data to classify whether a specific segment is a commercial on television. The dataset was taken from the UCI Machine Learning Repository with over 120,000 instances and 4125 features. For more information about the features in this dataset, check out [the UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/datasets/tv+news+channel+commercial+detection+dataset). \n",
    "\n",
    "**The Method:** This experiment will compare training a classification model with traditional machine learning models (KNN, Random Forest, Neural Networks) to training it with Microsoft Azure Automated Machine Learning. \n",
    "\n",
    "<br>\n",
    "<img src=\"https://sportsradiopd.com/wp-content/uploads/2015/11/commercial-e1447279275378.jpg\" style=\"width:500px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This experiment uses [Microsoft Azure Machine Learning Service](https://docs.microsoft.com/en-us/azure/machine-learning/service/) to implement a machine learning classification model. For more examples of Microsoft Machine Learning Service, check out the [Azure Machine Learning Notebooks GitHub Repo](https://github.com/Azure/MachineLearningNotebooks)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing the Azure Workspace\n",
    "To configure a Microsoft Azure workspace, you must [set up a  Azure subscription](https://azure.microsoft.com/en-us/free/) and manage the subscription from the [Azure portal](https://portal.azure.com/). Once your workspace is configured in the Azure portal, your machine learning service workspace should look like the following screenshot. \n",
    "\n",
    "<img src=\"img/configuration.png\">\n",
    "\n",
    "Then, download the `config.json` file in a directory two levels above anything that is being pushed to GitHub. **Never push your config file to GitHub**. Your Azure notebook will find the config file even if it is not in the present directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring the Azure Workspace\n",
    "Before configuring your workspace, be sure to have the following installed:\n",
    "```\n",
    "$ conda install -y matplotlib tqdm scikit-learn\n",
    "$ pip install azureml-sdk[notebooks,automl]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now you're ready to load your workspace...\n",
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "# IMPORT AZURE LIBRARIES\n",
    "# Azure Notebook Libraries\n",
    "import azureml.core\n",
    "import logging\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an Experiment\n",
    "This command will create a new experiment on Azure's Machine Learning Services Workspace. Experiments track important metrics of each model run, including training time (in seconds) and the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a name for the experiment and specify the project folder.\n",
    "experiment_name = 'commercial_block_classification'\n",
    "project_folder = './commercial_block_classification'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing your Experiment\n",
    "Once you create an experiment, you can view the experiment and all of the metrics on the Azure [Machine Learning Services workspace](https://portal.azure.com/).\n",
    "\n",
    "\n",
    "<img src=\"img/experiments.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "Data is from the UCI Machine Learning Repository with over 120,000 instances and 4125 features to [classify commercial blocks](http://archive.ics.uci.edu/ml/datasets/tv+news+channel+commercial+detection+dataset).\n",
    "\n",
    "Data Citation:\n",
    "Dr. Prithwijit Guha , Raghvendra D. Kannao and Ravishankar Soni \n",
    "Multimedia Analytics Lab, \n",
    "Department of Electrical and Electronics Engineering, \n",
    "Indian Institute of Technology, Guwahati, India \n",
    "rdkannao '@' gmail.com , prithwijit.guha '@' gmail.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data from Datastore to Local Directory\n",
    "To import this data, you can download from your Azure datastore or download the repository from Github. Make sure to download the datasets in `data/` on your local machine and point to your filepath in `get_data()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are your availible datastores...\n",
      "workspaceblobstore AzureBlob\n",
      "workspacefilestore AzureFile\n",
      "commercial_blocks AzureBlob\n",
      "machine_translation AzureBlob\n",
      "commercialblockclassification AzureBlob\n",
      "\n",
      "Make commercial blocks your default...\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Datastore\n",
    "\n",
    "# Make sure the commercialblockclassification datastore is included in your workspace\n",
    "# Otherwise, you will have to download the files from Github\n",
    "print(\"These are your availible datastores...\")\n",
    "datastores = ws.datastores\n",
    "for name, ds in datastores.items():\n",
    "    print(name, ds.datastore_type)\n",
    "\n",
    "print(\"\\nMake commercial blocks your default...\")\n",
    "ws.set_default_datastore('commercialblockclassification')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will download the `train.txt` and `test.txt` files in your `data/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading test_data.txt\n",
      "Downloading train_data.txt\n",
      "Downloaded test_data.txt, 1 files out of an estimated total of 2\n",
      "Downloaded train_data.txt, 2 files out of an estimated total of 2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds.download(target_path='data/',show_progress=True, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions will import your data as a dataframe to use with SciKit learn models in the `train.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Importing Data...\n",
      "Data imported.\n",
      "\n",
      "Training data has 13500 rows\n",
      "Testing data has 5750 rows\n"
     ]
    }
   ],
   "source": [
    "# Data Upload Functions\n",
    "from sklearn.datasets import load_svmlight_file\n",
    "\n",
    "def get_data(filepath):\n",
    "    data = load_svmlight_file(filepath)\n",
    "    return data[0], data[1]\n",
    "\n",
    "# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "# IMPORT DATA\n",
    "print(\"\\nImporting Data...\")\n",
    "\n",
    "X_train, y_train = get_data(\"data/train_data.txt\")\n",
    "X_test, y_test = get_data(\"data/test_data.txt\")\n",
    "\n",
    "X_train = X_train.toarray() # convert sparce matrix to array\n",
    "X_test = X_test.toarray() \n",
    "print(\"Data imported.\")\n",
    "\n",
    "print(\"\\nTraining data has %i rows\" % len(X_train))\n",
    "print(\"Testing data has %i rows\" % len(X_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying with SciKit Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Managing Dependencies\n",
    "This is a local run of the classifcation model, so you must ensure all the necessary packages are available in the Python environment you run in the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.runconfig import RunConfiguration\n",
    "\n",
    "# Editing a run configuration property on-fly.\n",
    "run_config_user_managed = RunConfiguration()\n",
    "\n",
    "run_config_user_managed.environment.python.user_managed_dependencies = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train with `train.py` Script and Log Metrics\n",
    "With Azure, you can train on almost any local `train.py` script! The training script in this scenario implements SciKit learn models with the standard SciKit Learn libraries. This script was taken from a homework assignment that did not originally use Microsoft Azure. Azure Machine Learning Services allows users to run standalone `train.py` scripts on an experiment without any alterations. However, logging variables were added to the original script to track model accuracy and other important metics. To add logging, the following lines were added to the original `train.py` script:\n",
    "\n",
    "**Added to the library imports:**\n",
    "```\n",
    "from azureml.core.run import Run\n",
    "```\n",
    "**Added after the model was trained** \n",
    "\n",
    "This code initializes the logger in your experiment context\n",
    "```\n",
    "run_logger = Run.get_context()\n",
    "```\n",
    "\n",
    "**Logged multiple variables to track!** \n",
    "```\n",
    "run_logger.log(name='Model', value=model_name)\n",
    "run_logger.log(name='Accuracy', value=accuracy)\n",
    "run_logger.log(name='Training_Time', value=train_t)\n",
    "```\n",
    "\n",
    "\n",
    "For more documentation about logging variables, [navigate here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-track-experiments#viewing-charts-in-run-details)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Local Training Script\n",
    "This local training script (with added logging capabilities) iterates through three Scikit Learn models, `RandomForestClassifier`, `MLPClassifier`, `KNeighborsClassifier`, with default parameters and logs the test accuracy of each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "# Manual train.py for Classification of Commericial Blocks\n",
      "# ~ By: Katie House\n",
      "# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "\n",
      "# IMPORT LIBRARIES\n",
      "from sklearn.datasets import load_svmlight_file\n",
      "import numpy as np\n",
      "import time\n",
      "from sklearn.metrics import f1_score\n",
      "import csv\n",
      "from sklearn.ensemble import RandomForestClassifier\n",
      "from sklearn.neural_network import MLPClassifier\n",
      "from sklearn.neighbors import KNeighborsClassifier\n",
      "\n",
      "# IMPORT AZURE LIBRARY\n",
      "from azureml.core.run import Run\n",
      "\n",
      "\n",
      "# DEFINE FUNCTIONS\n",
      "def get_data(filepath):\n",
      "    data = load_svmlight_file(filepath)\n",
      "    return data[0], data[1]\n",
      "\n",
      "\n",
      "def Classifier_Test_Train(model, model_name):\n",
      "    # Training Model...\n",
      "    start_time = time.time()  # track train time\n",
      "    model.fit(X_train, y_train)\n",
      "    train_t = round(time.time() - start_time, 2)\n",
      "\n",
      "    # Testing Model...\n",
      "    predictions = model.predict(X_test)\n",
      "    accuracy = round(f1_score(y_test, predictions), 3)\n",
      "\n",
      "    # AZURE LOGGING VARIABLES\n",
      "    run_logger = Run.get_context()\n",
      "    run_logger.log(name='Model', value=model_name)\n",
      "    run_logger.log(name='Accuracy', value=accuracy)\n",
      "    run_logger.log(name='Training_Time', value=train_t)\n",
      "\n",
      "\n",
      "# MAIN FUNCTION\n",
      "if __name__ == '__main__':\n",
      "    run = Run.get_context()\n",
      "\n",
      "    # IMPORT DATA\n",
      "    X_train, y_train = get_data(\"data/train_data.txt\")\n",
      "    X_test, y_test = get_data(\"data/test_data.txt\")\n",
      "\n",
      "    X_train = X_train.toarray()  # convert sparce matrix to array\n",
      "    X_test = X_test.toarray()\n",
      "\n",
      "    # ITERATE THROUGH MODELS\n",
      "    model_list = [RandomForestClassifier(),\n",
      "                  MLPClassifier(),\n",
      "                  KNeighborsClassifier()]\n",
      "    model_names = [\"Random Forest\",\n",
      "                   \"Neural Networks\",\n",
      "                   \"K Nearest Neighbor\"]\n",
      "\n",
      "    # TRAIN AND TEST MODELS\n",
      "    results = []\n",
      "    for i in range(len(model_list)):\n",
      "        results += [(Classifier_Test_Train(model_list[i], model_names[i]))]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('train.py', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit a Local Run of the Training Script\n",
    "Adding the `script='train.py'` argument to `ScriptRunConfig()` will run your local training script on the experiment on the commercial_block_classification experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import ScriptRunConfig\n",
    "\n",
    "src = ScriptRunConfig(source_directory='./', script='train.py', run_config=run_config_user_managed)\n",
    "run = experiment.submit(src)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View your Evaluation Metrics in Azure\n",
    "Azure automatically visualizes your logged metrics for your convenience. This expermiment logs `Training_Time` and `Accuracy` for the default Random Forest, Neural Networks, K Nearest Neighbor models in SciKit Learn. \n",
    "\n",
    "Use run by itself to access a link to the run in the Azure Portal. **Click `Link to Azure Portal` below** to view your visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/azure-link.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the \"Experiments\" tab you can view your logging metrics as a visualization!\n",
    "\n",
    "<img src=\"img/logging.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get your evaluation metrics as a dictionary, use the following commamd:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Model': ['Random Forest', 'Neural Networks', 'K Nearest Neighbor'],\n",
       " 'Accuracy': [0.882, 0.828, 0.777],\n",
       " 'Training_Time': [1.28, 3.76, 0.31]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_metrics = run.get_metrics()\n",
    "run_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using well-known models in Scikit Learn, we only achieve up to ~88% accuracy on the test set with Random Forest classification. Let's store the best accuracy and see if Automated Machine Learning can beat our best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best accuracy we acheived with Scikit Learn:  88.20%\n"
     ]
    }
   ],
   "source": [
    "best_manual_accuracy = float(max(run_metrics['Accuracy'])) * 100 \n",
    "print(\"The best accuracy we acheived with Scikit Learn:  %.2f%%\" % best_manual_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying with Automated Machine Learning\n",
    "Microsoft Azure's [Automated Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train) functionality trains models for you to help find the best model for your machine learning problem. The following experiment will compare the metrics I achieved with the simple `train.py` script with the automated machine learning metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Automated ML for Classification\n",
    "Automated ML offers many different configurations to match your machine learning task. [This article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#select-your-experiment-type) describes all the possible configurations you can choose from. I decided to iterate through 10 different models with 3 cross validations to attempt to beat my initial 88% accuracy result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_config = AutoMLConfig(task = 'classification',\n",
    "                             debug_log = 'automl_errors.log',\n",
    "                             primary_metric = 'AUC_weighted',\n",
    "                             iteration_timeout_minutes = 60,\n",
    "                             iterations = 20,\n",
    "                             n_cross_validations = 3,\n",
    "                             verbosity = logging.INFO,\n",
    "                             X = X_train, \n",
    "                             y = y_train,\n",
    "                             path = project_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the AML Experiment Locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local machine\n",
      "Parent Run ID: AutoML_5763d76e-ba28-4236-85de-550fa25f1aaf\n",
      "Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.\n",
      "Current status: ModelSelection. Beginning model selection.\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   StandardScalerWrapper SGD                      0:00:13       0.9376    0.9376\n",
      "         1   StandardScalerWrapper SGD                      0:00:11       0.9287    0.9376\n",
      "         2   MinMaxScaler LightGBM                          0:00:13       0.9490    0.9490\n",
      "         3   StandardScalerWrapper SGD                      0:00:11       0.9340    0.9490\n",
      "         4   StandardScalerWrapper ExtremeRandomTrees       0:00:16       0.9110    0.9490\n",
      "         5   MinMaxScaler SGD                               0:00:14       0.9273    0.9490\n",
      "         6   StandardScalerWrapper LightGBM                 0:00:13       0.9481    0.9490\n",
      "         7   StandardScalerWrapper SGD                      0:00:11       0.9268    0.9490\n",
      "         8   MinMaxScaler RandomForest                      0:00:18       0.9030    0.9490\n",
      "         9   StandardScalerWrapper SGD                      0:00:11       0.8518    0.9490\n",
      "        10   MinMaxScaler SGD                               0:00:13       0.9347    0.9490\n",
      "        11   StandardScalerWrapper RandomForest             0:00:20       0.8840    0.9490\n",
      "        12   MinMaxScaler SGD                               0:00:12       0.9278    0.9490\n",
      "        13   MinMaxScaler RandomForest                      0:00:13       0.9097    0.9490\n",
      "        14   MaxAbsScaler LightGBM                          0:00:12       0.9301    0.9490\n",
      "        15   MinMaxScaler BernoulliNaiveBayes               0:00:11       0.7682    0.9490\n",
      "        16   MinMaxScaler LightGBM                          0:00:17       0.9607    0.9607\n",
      "        17   MinMaxScaler ExtremeRandomTrees                0:00:13       0.9035    0.9607\n",
      "        18   VotingEnsemble                                 0:00:28       0.9602    0.9607\n",
      "        19   "
     ]
    }
   ],
   "source": [
    "local_run = experiment.submit(automl_config, show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "Use local_run by itself to access a link to the run in the Azure Portal. The experiment will contain visualizations of each model performance in the AutoML iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>commercial_block_classification</td><td>AutoML_7f256506-3ee7-4182-b8f9-e9f65e8341b4</td><td>automl</td><td>NotStarted</td><td><a href=\"https://mlworkspace.azure.ai/portal/subscriptions/a989cac4-671d-45c6-9d2b-ea7e9d936600/resourceGroups/AmherstRG/providers/Microsoft.MachineLearningServices/workspaces/AmherstWorkspace/experiments/commercial_block_classification/runs/AutoML_7f256506-3ee7-4182-b8f9-e9f65e8341b4\" target=\"_blank\" rel=\"noopener\">Link to Azure Portal</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: commercial_block_classification,\n",
       "Id: AutoML_7f256506-3ee7-4182-b8f9-e9f65e8341b4,\n",
       "Type: automl,\n",
       "Status: NotStarted)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "local_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/azure-link-3.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View your Machine Learning Performance in Azure\n",
    "AzureML widgets automatically generates 20 interactive visualizations of each of the model iterations. View this visualizations by running the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a176666a05244cfd9916532f11611e6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_AutoMLWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(local_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/auc.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select the Best Auto ML Model\n",
    "Now the best classification model from the Automated Machine Learning iterations was selected to compare with the manual `train.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline(memory=None,\n",
      "     steps=[('stackensembleclassifier', StackEnsembleClassifier(base_learners=[('2', PipelineWithYTransformations(Pipeline={'memory': None, 'steps': [('MinMaxScaler', MinMaxScaler(copy=True, feature_range=(0, 1))), ('LightGBMClassifier', <automl.client.core.common.model_wrappers.LightGBMClassifier object...olver='warn',\n",
      "          tol=0.0001, verbose=0, warm_start=False),\n",
      "            training_cv_folds=5))])\n",
      "Y_transformer(['LabelEncoder', LabelEncoder()])\n"
     ]
    }
   ],
   "source": [
    "best_run, fitted_model = local_run.get_output()\n",
    "print(fitted_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the Best Classification Model\n",
    "Let's compare the testing accuracy of the manual training script versus the automated machine learning script to find the best classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train.py Accuracy: 88.20%\n",
      "Automated Machine Learning Accuracy: 88.10%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "y_pred = fitted_model.predict(X_test)\n",
    "aml_accuracy = f1_score(y_test, y_pred) * 100\n",
    "\n",
    "print('Train.py Accuracy: %.2f%%' % best_manual_accuracy)\n",
    "print('Automated Machine Learning Accuracy: %.2f%%' % aml_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using Automated Machine Learning, we increased our testing accuracy by ~3%!\n",
    "\n",
    "\n",
    "<img src=\"https://media.giphy.com/media/YJ5OlVLZ2QNl6/giphy.gif\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}