05_Train_Best_Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Best Model\n",
    "In this notebook, we recover into Python the HyperDrive run that was created in the previous notebook, use it to find the best child run discovered by the HyperDrive search, use that best run's parameters to train a model, and finally register that model.\n",
    "\n",
    "The steps in this notebook are\n",
    "- [import libraries](#import),\n",
    "- [read in the Azure ML workspace](#workspace),\n",
    "- [recover the HyperDrive run](#recover),\n",
    "- [use the best hyperparameters found to train a model](#results), and\n",
    "- [register the trained model](#register).\n",
    "\n",
    "## Imports  <a id='import'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from azureml.core import Workspace, Experiment\n",
    "from azureml.train.hyperdrive import HyperDriveRun\n",
    "from azureml.train.estimator import Estimator\n",
    "from azureml.widgets import RunDetails\n",
    "import azureml.core\n",
    "from get_auth import get_auth\n",
    "print('azureml.core.VERSION={}'.format(azureml.core.VERSION))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in the Azure ML workspace  <a id='workspace'></a>\n",
    "Read in the the workspace created in a previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "auth = get_auth()\n",
    "ws = Workspace.from_config(auth=auth)\n",
    "ws_details = ws.get_details()\n",
    "print('Name:\\t\\t{}\\nLocation:\\t{}'\n",
    "      .format(ws_details['name'],\n",
    "              ws_details['location']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recover the run  <a id='recover'></a>\n",
    "Get an experiment that ran the search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp = Experiment(workspace=ws, name='hypetuning')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the ID of the HyperDrive run created in the last notebook. That ID was printed with the run when it was submitted in the previous notebook, and we also saved it in a file. You can also find that ID in Azure Portal on your experiment's page. To see it, you may need to add a `RunId` column to the experiment's table of runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_id_path = \"run_id.txt\"\n",
    "with open(run_id_path, \"r\") as fp:\n",
    "    run_id = fp.read()\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the ID of the HyperDrive run to get a handle to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = HyperDriveRun(exp, run_id)\n",
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the best hyperparameters to train a model <a id='results'></a>\n",
    "We can automatically select the best run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run = run.get_best_run_by_primary_metric()\n",
    "if best_run is None:\n",
    "    raise Exception(\"No best run was found\")\n",
    "best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the best run's hyperparameter set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameter_values = best_run.get_details()['runDefinition']['arguments']\n",
    "best_parameters = dict(zip(parameter_values[::2], parameter_values[1::2]))\n",
    "pd.Series(best_parameters, name='Value').to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use these parameters to train and save the best model. We will train with a boosted number of estimators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "estimators = 8 * int(best_parameters['--estimators'])\n",
    "estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new set of parameters to train the best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = ws.get_default_datastore()\n",
    "model_parameters = best_parameters.copy()\n",
    "model_parameters['--data-folder'] = ds.as_mount()\n",
    "model_parameters['--estimators'] = estimators\n",
    "model_parameters['--save'] = 'FAQ_ranker'\n",
    "pd.Series(model_parameters, name='Value').to_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the compute target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_name = 'hypetuning'\n",
    "compute_target = ws.compute_targets[cluster_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train and save the best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_est = Estimator(source_directory=os.path.join('.', 'scripts'),\n",
    "                      entry_script='TrainClassifier.py',\n",
    "                      script_params=model_parameters,\n",
    "                      compute_target=compute_target,\n",
    "                      conda_packages=['pandas==0.23.4',\n",
    "                                      'scikit-learn==0.21.3',\n",
    "                                      'lightgbm==2.2.1'])\n",
    "model_run = exp.submit(model_est)\n",
    "model_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for the model to be created and saved. This step can take up to two hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model_run_status = model_run.wait_for_completion(wait_post_processing=True)\n",
    "print(model_run_status['status'])\n",
    "if model_run_status['status'] not in ['Completed', 'Finalizing']:\n",
    "    raise Exception('The run did not successfully complete.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register the best model <a id='register'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model_run.register_model(model_name='FAQ_ranker', model_path=os.path.join('outputs', 'FAQ_ranker.pkl'))\n",
    "print(model.name, model.version, sep = '\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [next notebook](06_Test_Best_Model.ipynb) applies the best model to the test data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}