03_Run_Locally.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure ML Local Run\n",
    "In this notebook, we create an Azure ML workspace, and use it to locally run the training script.\n",
    "\n",
    "The steps in this notebook are\n",
    "- [import libraries](#import),\n",
    "- [set the Azure subscription](#subscription),\n",
    "- [create an Azure ML workspace](#workspace),\n",
    "- [create an estimator](#estimator),\n",
    "- [create an experiment](#experiment),\n",
    "- [submit the estimator](#submit), and\n",
    "- [get the results](#results).\n",
    "\n",
    "## Imports  <a id='import'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.common.credentials import get_cli_profile\n",
    "from azureml.core import Workspace, Experiment\n",
    "from azureml.train.estimator import Estimator\n",
    "import azureml.core\n",
    "from get_auth import get_auth\n",
    "print('azureml.core.VERSION={}'.format(azureml.core.VERSION))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Azure subscription <a id='subscription'></a>\n",
    "If you have multiple subscriptions select the subscription you want to use. You may supply either the subscription's name or the subscription's ID. If you want to run this in a different location that supports HyperDrive, you may enter the one you want to use. You can also set the name of the resource group in which this tutorial will add resources. *IMPORTANT NOTE:* The last notebook in this example will delete this resource group and all associated resources. We also define the number of estimators to use for the local run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "subscription_name=\"YOUR_SUBSCRIPTION_NAME\"\n",
    "subscription_id=\"YOUR_SUBSCRIPTION_ID\"\n",
    "location=\"eastus\"\n",
    "resource_group=\"hypetuning\"\n",
    "estimators = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that we have either a subscription name or ID. If the ID has been supplied, then use that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (subscription_name == \"YOUR_SUBSCRIPTION_NAME\"\n",
    "    and subscription_id == \"YOUR_SUBSCRIPTION_ID\"):\n",
    "    raise Exception(\"At least one of a subscription's name or ID must be supplied\")\n",
    "if subscription_id != \"YOUR_SUBSCRIPTION_ID\":\n",
    "    print(\"A subscription ID has been supplied, so it will be used\")\n",
    "    subscription_name = subscription_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Login to Azure if not already logged in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "list=`az account list -o table`\n",
    "if [ \"$list\" == '[]' ] || [ \"$list\" == '' ]; then \n",
    "  az login -o table\n",
    "else\n",
    "  az account list -o table \n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the selected subscription as the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$subscription_name\"\n",
    "az account set --subscription \"$1\"\n",
    "az account show -o table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the ID for the selected Azure subscription."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "az_profile = get_cli_profile()\n",
    "subscription_id = az_profile.get_subscription_id()\n",
    "print(\"Using subscription ID\", subscription_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an Azure ML workspace <a id='workspace'></a>\n",
    "Create a workspace if it does not already exist or recover it if it does exist, and write out its details to `config.json` to reference it between notebooks. The first time this is run, this can take about a minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "auth = get_auth()\n",
    "ws = Workspace.create(name='hypetuning',\n",
    "                      subscription_id=subscription_id,\n",
    "                      resource_group=resource_group,\n",
    "                      create_resource_group=True,\n",
    "                      exist_ok=True,\n",
    "                      location=location,\n",
    "                      auth=auth)\n",
    "ws.write_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an estimator <a id='estimator'></a>\n",
    "Create an estimator that specifies the location of the script, sets up its parameters, and specifies the packages needed to run the script. It may take a while to prepare the run environment the first time an estimator is used, but that environment will be used until the list of packages is changed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "est = Estimator(source_directory=os.path.join('.', 'scripts'), \n",
    "                entry_script='TrainClassifier.py',\n",
    "                script_params={'--data-folder': os.path.abspath('.'),\n",
    "                               '--estimators': estimators,\n",
    "                               '--match': 5,\n",
    "                               '--ngrams': 2,\n",
    "                               '--min_child_samples': 10,\n",
    "                               \"--save\": \"local_model\"},\n",
    "                compute_target='local',\n",
    "                conda_packages=['pandas==0.23.4',\n",
    "                                'scikit-learn==0.21.3',\n",
    "                                'lightgbm==2.2.1'],\n",
    "                use_docker=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an experiment <a id='experiment'></a>\n",
    "Get an experiment to run the script; create it if it doesn't already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp = Experiment(workspace=ws, name='hypetuning')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit the script <a id='submit'></a>\n",
    "Submit the estimator containing the script to be run. This should return almost immediately, and the value is an object that lets you programmatically control the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "run = exp.submit(est)\n",
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experiment returns a table with a link to the `Details Page` in the Azure Portal. That page will let you monitor the status of this run of the experiment, and that of previous runs of that experiment. By clicking on a particular run, you can see its details, the files output by the script, and the logs of the run, including the `driver.log` with the script's print outs.\n",
    "\n",
    "## Get the results <a id='results'></a>\n",
    "Wait for the run to complete. It will take about three minutes. This returns a `dict` with detailed information about the run. Here, we see that the run is either `Finalizing` or has `Completed`. Other statuses include `Queued`, `Preparing`, `Initializing`, `Running`, and `Failed`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "run_status = run.wait_for_completion()\n",
    "run_status['status']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now get the tune dataset metrics logged by the script during its execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "run.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model using the test data <a id='evaluate'></a>\n",
    "Download the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run.download_file(os.path.join('outputs', 'local_model.pkl'),\n",
    "                  os.path.join('outputs', 'local_model.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the model's performance on the held-aside test data. This can take a couple of minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -t scripts/TestClassifier.py --model local_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In [the next notebook](04_Hyperparameter_Random_Search.ipynb), we use the AML SDK to tune the set of hyperparameters."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}