04A - Working with Datastores.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Datastores\n",
    "\n",
    "In previous labs, you connected to an Azure ML workspace and ran a simple experiment based on data in a local CSV file where the experiment script is being run. Now it's time to work with data stored in the cloud.\n",
    "\n",
    "> **Important**: The code in this notebooks assumes that you have completed the first two tasks in [Lab 4A](labdocs/Lab04A.md). If you have not done so, go and do it now!\n",
    "\n",
    "## Connect to Your Workspace\n",
    "\n",
    "The first thing you need to do is to connect to your workspace using the Azure ML SDK.\n",
    "\n",
    "> **Note**: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View Datastores in the Workspace\n",
    "\n",
    "The workspace contains several datastores, including the **aml_data** datastore you ceated in the [previous task](labdocs/Lab04A.md).\n",
    "\n",
    "Run the following code to retrieve the *default* datastore, and then list all of the datastores indicating which is the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the default datastore\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "# Enumerate all datastores, indicating which is the default\n",
    "for ds_name in ws.datastores:\n",
    "    print(ds_name, \"- Default =\", ds_name == default_ds.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a Datastore to Work With\n",
    "\n",
    "You want to work with the **aml_data** datastore, so you need to get it by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "\n",
    "aml_datastore = Datastore.get(ws, 'aml_data')\n",
    "print(aml_datastore.name,\":\", aml_datastore.datastore_type + \" (\" + aml_datastore.account_name + \")\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the Default Datastore\n",
    "\n",
    "You are primarily going towork with the **aml_data** datastore in this course; so for convenience, you can set it to be the default datastore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws.set_default_datastore('aml_data')\n",
    "default_ds = ws.get_default_datastore()\n",
    "print(default_ds.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload Data to a Datastore\n",
    "\n",
    "Now that you have identified the datastore you want to work with, you can upload files from your local file system so that they will be accessible to experiments running in the workspace, regardless of where the experiment script is actually being run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                       target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                       overwrite=True, # Replace existing files of the same name\n",
    "                       show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a Model from a Datastore\n",
    "\n",
    "When you uploaded the files in the code cell above, note that the code returned a *data reference*. A data reference provides a way to pass the path to a folder in a datastore to a script, regardless of where the script is being run, so that the script can access data in the datastore location.\n",
    "\n",
    "The following code gets a reference to the **diabetes-data** folder where you uploaded the diabetes CSV files, and specifically configures the data reference for *download* - in other words, it can be used to download the contents of the folder to the compute context where the data reference is being used. Downloading data works well for small volumes of data that will be processed on local compute. When working with remote compute, you can also configure a data reference to *mount* the datastore location and read data directly from the data source.\n",
    "\n",
    "> **More Information**: For more details about using datastores, see the [Azure ML documentation](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-access-data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_ref = default_ds.path('diabetes-data').as_download(path_on_compute='diabetes_data')\n",
    "print(data_ref)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the data reference in a training script, you must define a parameter for it. Run the following two code cells to create:\n",
    "\n",
    "1. A folder named **diabetes_training_from_datastore**\n",
    "2. A script that trains a classification model by using the training data in all of the CSV files in the folder referenced by the data reference parameter passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Create a folder for the experiment files\n",
    "experiment_folder = 'diabetes_training_from_datastore'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "print(experiment_folder, 'folder created.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import os\n",
    "import argparse\n",
    "from azureml.core import Run\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# Get parameters\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
    "parser.add_argument('--data-folder', type=str, dest='data_folder', help='data folder reference')\n",
    "args = parser.parse_args()\n",
    "reg = args.reg_rate\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes data from the data reference\n",
    "data_folder = args.data_folder\n",
    "print(\"Loading data from\", data_folder)\n",
    "# Load all files and concatenate their contents as a single dataframe\n",
    "all_files = os.listdir(data_folder)\n",
    "diabetes = pd.concat((pd.read_csv(os.path.join(data_folder,csv_file)) for csv_file in all_files))\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a logistic regression model\n",
    "print('Training a logistic regression model with regularization rate of', reg)\n",
    "run.log('Regularization Rate',  np.float(reg))\n",
    "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
    "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script will load the training data from the data reference passed to it as a parameter, so now you just need to set up the script parameters to pass the file reference when we run the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.sklearn import SKLearn\n",
    "from azureml.core import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Set up the parameters\n",
    "script_params = {\n",
    "    '--regularization': 0.1, # regularization rate\n",
    "    '--data-folder': data_ref # data reference to download files from datastore\n",
    "}\n",
    "\n",
    "\n",
    "# Create an estimator\n",
    "estimator = SKLearn(source_directory=experiment_folder,\n",
    "                    entry_script='diabetes_training.py',\n",
    "                    script_params=script_params,\n",
    "                    compute_target = 'local'\n",
    "                   )\n",
    "\n",
    "# Create an experiment\n",
    "experiment_name = 'diabetes-training'\n",
    "experiment = Experiment(workspace = ws, name = experiment_name)\n",
    "\n",
    "# Run the experiment\n",
    "run = experiment.submit(config=estimator)\n",
    "# Show the run details while running\n",
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time the experiment is run, it may take some time to set up the Python environment - subsequent runs will be quicker.\n",
    "\n",
    "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log to verify that the data files were downloaded.\n",
    "\n",
    "As with all experiments, you can view the details of the experiment run in the [Azure ML Studio web interface](https://ml.azure.com), and you can write code to retrieve the metrics and files generated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get logged metrics\n",
    "metrics = run.get_metrics()\n",
    "for key in metrics.keys():\n",
    "        print(key, metrics.get(key))\n",
    "print('\\n')\n",
    "for file in run.get_file_names():\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, you can register the model that was trained by the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Model\n",
    "\n",
    "# Register the model\n",
    "run.register_model(model_path='outputs/diabetes_model.pkl', model_name='diabetes_model',\n",
    "                   tags={'Training context':'SKLearn Estimator (using Datastore)'}, properties={'AUC': run.get_metrics()['AUC'], 'Accuracy': run.get_metrics()['Accuracy']})\n",
    "\n",
    "# List the registered models\n",
    "print(\"Registered Models:\")\n",
    "for model in Model.list(ws):\n",
    "    print(model.name, 'version:', model.version)\n",
    "    for tag_name in model.tags:\n",
    "        tag = model.tags[tag_name]\n",
    "        print ('\\t',tag_name, ':', tag)\n",
    "    for prop_name in model.properties:\n",
    "        prop = model.properties[prop_name]\n",
    "        print ('\\t',prop_name, ':', prop)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you've explored some options for working with data in the form of *datastores*.\n",
    "\n",
    "Azure Machine Learning offers a further level of abstraction for data in the form of *datasets*, which you'll explore next."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}