14 - Interpret Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interpret Models\n",
    "\n",
    "You can use Azure Machine Learning to interpret a model by using an *explainer* that quantifies the amount of influence each feature contribues to the predicted label. There are many common explainers, each suitable for different kinds of modeling algorithm; but the basic approach to using them is the same.\n",
    "\n",
    "## Install SDK packages\n",
    "\n",
    "In addition to the latest version of the **azureml-sdk** and **azureml-widgets** packages, you'll need the **azureml-explain-model** package to run the code in this notebook. You'll also use the Azure ML Interpretability library (**azureml-interpret**). You can use this to interpret many typical kinds of model, even if they haven't been trained in an Azure ML experiment or registered in an Azure ML workspace.\n",
    "\n",
    "Run the cell below to verify that these packages are installed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "gather": {
     "logged": 1649368363682
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: azureml-explain-model\n",
      "Version: 1.44.0\n",
      "Summary: The package has been deprecated and might not receive future updates.\n",
      "Home-page: https://docs.microsoft.com/python/api/overview/azure/ml/?view=azure-ml-py\n",
      "Author: Microsoft Corp\n",
      "Author-email: None\n",
      "License: https://aka.ms/azureml-sdk-license\n",
      "Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages\n",
      "Requires: azureml-interpret\n",
      "Required-by: \n",
      "---\n",
      "Name: azureml-interpret\n",
      "Version: 1.44.0\n",
      "Summary: Machine Learning interpret package is used to interpret ML models\n",
      "Home-page: https://docs.microsoft.com/python/api/overview/azure/ml/?view=azure-ml-py\n",
      "Author: Microsoft Corp\n",
      "Author-email: None\n",
      "License: https://aka.ms/azureml-sdk-license\n",
      "Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages\n",
      "Requires: numpy, numba, shap, interpret-community, azureml-core\n",
      "Required-by: azureml-train-automl-runtime, azureml-responsibleai, azureml-explain-model\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip show azureml-explain-model azureml-interpret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explain a model\n",
    "\n",
    "Let's start with a model that is trained outside of Azure Machine Learning - Run the cell below to train a decision tree classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Data...\n",
      "Training a decision tree model\n",
      "Accuracy: 0.89\n",
      "AUC: 0.8761281655803771\n",
      "Model trained.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# load the diabetes dataset\n",
    "print(\"Loading Data...\")\n",
    "data = pd.read_csv('data/diabetes.csv')\n",
    "\n",
    "# Separate features and labels\n",
    "features = ['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']\n",
    "labels = ['not-diabetic', 'diabetic']\n",
    "X, y = data[features].values, data['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a decision tree model\n",
    "print('Training a decision tree model')\n",
    "model = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "\n",
    "print('Model trained.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training process generated some model evaluation metrics based on a hold-back validation dataset, so you have an idea of how accurately it predicts; but how do the features in the data influence the prediction?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get an explainer for the model\n",
    "\n",
    "Let's get a suitable explainer for the model from the Azure ML interpretability library you installed earlier. There are many kinds of explainer. In this example you'll use a *Tabular Explainer*, which is a \"black box\" explainer that can be used to explain many kinds of model by invoking an appropriate [SHAP](https://github.com/slundberg/shap) model explainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TabularExplainer ready!\n"
     ]
    }
   ],
   "source": [
    "from interpret.ext.blackbox import TabularExplainer\n",
    "\n",
    "# \"features\" and \"classes\" fields are optional\n",
    "tab_explainer = TabularExplainer(model,\n",
    "                             X_train, \n",
    "                             features=features, \n",
    "                             classes=labels)\n",
    "print(tab_explainer, \"ready!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get *global* feature importance\n",
    "\n",
    "The first thing to do is try to explain the model by evaluating the overall *feature importance* - in other words, quantifying the extent to which each feature influences the prediction based on the whole training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pregnancies : 0.2182579329421298\n",
      "Age : 0.10610377061049878\n",
      "BMI : 0.09346910937334589\n",
      "SerumInsulin : 0.0684266455226624\n",
      "PlasmaGlucose : 0.050167463004650706\n",
      "TricepsThickness : 0.021414507176238374\n",
      "DiastolicBloodPressure : 0.01675264961361216\n",
      "DiabetesPedigree : 0.01408361129616792\n"
     ]
    }
   ],
   "source": [
    "# you can use the training data or the test data here\n",
    "global_tab_explanation = tab_explainer.explain_global(X_train)\n",
    "\n",
    "# Get the top features by importance\n",
    "global_tab_feature_importance = global_tab_explanation.get_feature_importance_dict()\n",
    "for feature, importance in global_tab_feature_importance.items():\n",
    "    print(feature,\":\", importance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature importance is ranked, with the most important feature listed first.\n",
    "\n",
    "### Get *local* feature importance\n",
    "\n",
    "So you have an overall view, but what about explaining individual observations? Let's generate *local* explanations for individual predictions, quantifying the extent to which each feature influenced the decision to predict each of the possible label values. In this case, it's a binary model, so there are two possible labels (non-diabetic and diabetic); and you can quantify the influence of each feature for each of these label values for individual observations in a dataset. You'll just evaluate the first two cases in the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Support for not-diabetic\n",
      "\tObservation 1\n",
      "\t\t SerumInsulin : 0.35622451853971343\n",
      "\t\t Age : 0.2462414592118101\n",
      "\t\t TricepsThickness : 0.026309452333534974\n",
      "\t\t BMI : 0.014286114552844672\n",
      "\t\t DiabetesPedigree : 0.0026804351668510743\n",
      "\t\t DiastolicBloodPressure : -0.013998079121474643\n",
      "\t\t PlasmaGlucose : -0.04142252321716635\n",
      "\t\t Pregnancies : -0.2568928060375418\n",
      "\t\t ----------\n",
      "\t\t Total: 0.3334285714285716 Prediction: not-diabetic\n",
      "\tObservation 2\n",
      "\t\t BMI : 0.3428580806236607\n",
      "\t\t Age : 0.10492329964823668\n",
      "\t\t DiabetesPedigree : 0.007955917022777254\n",
      "\t\t PlasmaGlucose : -0.004869223894000553\n",
      "\t\t DiastolicBloodPressure : -0.013938452364448567\n",
      "\t\t TricepsThickness : -0.021313441344163526\n",
      "\t\t Pregnancies : -0.02602108841923202\n",
      "\t\t SerumInsulin : -0.05616651984425941\n",
      "\t\t ----------\n",
      "\t\t Total: 0.3334285714285706 Prediction: not-diabetic\n",
      "Support for diabetic\n",
      "\tObservation 1\n",
      "\t\t Pregnancies : 0.2568928060375418\n",
      "\t\t PlasmaGlucose : 0.04142252321716636\n",
      "\t\t DiastolicBloodPressure : 0.013998079121474669\n",
      "\t\t DiabetesPedigree : -0.0026804351668510717\n",
      "\t\t BMI : -0.014286114552844556\n",
      "\t\t TricepsThickness : -0.02630945233353491\n",
      "\t\t Age : -0.24624145921181018\n",
      "\t\t SerumInsulin : -0.3562245185397133\n",
      "\t\t ----------\n",
      "\t\t Total: -0.33342857142857124 Prediction: not-diabetic\n",
      "\tObservation 2\n",
      "\t\t SerumInsulin : 0.05616651984425948\n",
      "\t\t Pregnancies : 0.026021088419231512\n",
      "\t\t TricepsThickness : 0.02131344134416356\n",
      "\t\t DiastolicBloodPressure : 0.013938452364448654\n",
      "\t\t PlasmaGlucose : 0.004869223894000574\n",
      "\t\t DiabetesPedigree : -0.007955917022777207\n",
      "\t\t Age : -0.10492329964823668\n",
      "\t\t BMI : -0.34285808062366036\n",
      "\t\t ----------\n",
      "\t\t Total: -0.33342857142857046 Prediction: not-diabetic\n"
     ]
    }
   ],
   "source": [
    "# Get the observations we want to explain (the first two)\n",
    "X_explain = X_test[0:2]\n",
    "\n",
    "# Get predictions\n",
    "predictions = model.predict(X_explain)\n",
    "\n",
    "# Get local explanations\n",
    "local_tab_explanation = tab_explainer.explain_local(X_explain)\n",
    "\n",
    "# Get feature names and importance for each possible label\n",
    "local_tab_features = local_tab_explanation.get_ranked_local_names()\n",
    "local_tab_importance = local_tab_explanation.get_ranked_local_values()\n",
    "\n",
    "for l in range(len(local_tab_features)):\n",
    "    print('Support for', labels[l])\n",
    "    label = local_tab_features[l]\n",
    "    for o in range(len(label)):\n",
    "        print(\"\\tObservation\", o + 1)\n",
    "        feature_list = label[o]\n",
    "        total_support = 0\n",
    "        for f in range(len(feature_list)):\n",
    "            print(\"\\t\\t\", feature_list[f], ':', local_tab_importance[l][o][f])\n",
    "            total_support += local_tab_importance[l][o][f]\n",
    "        print(\"\\t\\t ----------\\n\\t\\t Total:\", total_support, \"Prediction:\", labels[predictions[o]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding explainability to a model training experiment\n",
    "\n",
    "As you've seen, you can generate explanations for models trained outside of Azure Machine Learning; but when you use experiments to train and register models in your Azure Machine Learning workspace, you can generate model explanations and log them.\n",
    "\n",
    "Run the code in the following cell to connect to your workspace.\n",
    "\n",
    "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ready to use Azure ML 1.44.0 to work with mllearningworkspace\n"
     ]
    }
   ],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and explain a model using an experiment\n",
    "\n",
    "OK, let's create an experiment and put the files it needs in a local folder - in this case we'll just use the same CSV file of diabetes data to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'diabetes_train_and_explain/diabetes.csv'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os, shutil\n",
    "from azureml.core import Experiment\n",
    "\n",
    "# Create a folder for the experiment files\n",
    "experiment_folder = 'diabetes_train_and_explain'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "\n",
    "# Copy the data file into the experiment folder\n",
    "shutil.copy('data/diabetes.csv', os.path.join(experiment_folder, \"diabetes.csv\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll create a training script that looks similar to any other Azure ML training script except that is includes the following features:\n",
    "\n",
    "- The same libraries to generate model explanations we used before are imported and used to generate a global explanation\n",
    "- The **ExplanationClient** library is used to upload the explanation to the experiment output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing diabetes_train_and_explain/diabetes_training.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# Import Azure ML run library\n",
    "from azureml.core.run import Run\n",
    "\n",
    "# Import libraries for model explanation\n",
    "from azureml.interpret import ExplanationClient\n",
    "from interpret.ext.blackbox import TabularExplainer\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes dataset\n",
    "print(\"Loading Data...\")\n",
    "data = pd.read_csv('diabetes.csv')\n",
    "\n",
    "features = ['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']\n",
    "labels = ['not-diabetic', 'diabetic']\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = data[features].values, data['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a decision tree model\n",
    "print('Training a decision tree model')\n",
    "model = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
    "joblib.dump(value=model, filename='outputs/diabetes.pkl')\n",
    "\n",
    "# Get explanation\n",
    "explainer = TabularExplainer(model, X_train, features=features, classes=labels)\n",
    "explanation = explainer.explain_global(X_test)\n",
    "\n",
    "# Get an Explanation Client and upload the explanation\n",
    "explain_client = ExplanationClient.from_run(run)\n",
    "explain_client.upload_model_explanation(explanation, comment='Tabular Explanation')\n",
    "\n",
    "# Complete the run\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experiment needs a Python environment in which to run the script, so we'll define a Conda specification for it. Note that the **azureml-interpret** library is included in the training environment so the script can create a **TabularExplainer** and use the **ExplainerClient** class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing diabetes_train_and_explain/interpret_env.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile $experiment_folder/interpret_env.yml\n",
    "name: batch_environment\n",
    "dependencies:\n",
    "- python=3.6.2\n",
    "- scikit-learn\n",
    "- pandas\n",
    "- pip\n",
    "- pip:\n",
    "  - azureml-defaults\n",
    "  - azureml-interpret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can run the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24dabea88bd9436584105c66847804c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/mslearn-diabetes-explain_1671010520_e61c954c?wsid=/subscriptions/376621af-6a06-4df6-b902-d25c6239f015/resourcegroups/learning/workspaces/mllearningworkspace&tid=16b3c013-d300-468d-ac64-7eda0820b6d3\", \"run_id\": \"mslearn-diabetes-explain_1671010520_e61c954c\", \"run_properties\": {\"run_id\": \"mslearn-diabetes-explain_1671010520_e61c954c\", \"created_utc\": \"2022-12-14T09:35:21.855629Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"local\", \"ContentSnapshotId\": \"7990df8f-e062-4cdd-97ed-ec634ce94b7b\"}, \"tags\": {\"model_explanation\": \"True\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2022-12-14T09:38:05.498851Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/60_control_log.txt\": \"https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/azureml-logs/60_control_log.txt?sv=2019-07-07&sr=b&sig=nP8lA3Zrk%2F9jvBPSeP%2FTwBCJE4XUqxo9HqCcqGW%2Bjig%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-13T17%3A50%3A41Z&ske=2022-12-15T02%3A00%3A41Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A33%3A16Z&se=2022-12-14T17%3A43%3A16Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/azureml-logs/70_driver_log.txt?sv=2019-07-07&sr=b&sig=kirkYXXzREf5ckmer%2F7XnM5XCxwtDNlVl1rYDnDJaYY%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-13T17%3A50%3A41Z&ske=2022-12-15T02%3A00%3A41Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A33%3A16Z&se=2022-12-14T17%3A43%3A16Z&sp=r\", \"logs/azureml/8_azureml.log\": \"https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/logs/azureml/8_azureml.log?sv=2019-07-07&sr=b&sig=WYnaS9ofZNfLZlDUg8IRtA%2Br06cazHAY6o4gvNWm%2FYI%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-14T09%3A28%3A04Z&ske=2022-12-15T17%3A38%3A04Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A33%3A16Z&se=2022-12-14T17%3A43%3A16Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/8_azureml.log\"], [\"azureml-logs/60_control_log.txt\"], [\"azureml-logs/70_driver_log.txt\"]], \"run_duration\": \"0:02:43\", \"run_number\": \"1671010521\", \"run_queued_details\": {\"status\": \"Completed\", \"details\": null}}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [{\"name\": \"Accuracy\", \"run_id\": \"mslearn-diabetes-explain_1671010520_e61c954c\", \"categories\": [0], \"series\": [{\"data\": [0.8923333333333333]}]}, {\"name\": \"AUC\", \"run_id\": \"mslearn-diabetes-explain_1671010520_e61c954c\", \"categories\": [0], \"series\": [{\"data\": [0.8805686850092044]}]}, {\"name\": \"model_explanation\", \"run_id\": \"mslearn-diabetes-explain_1671010520_e61c954c\", \"categories\": [0], \"series\": [{\"data\": [{\"class_labels\": [\"not-diabetic\", \"diabetic\"], \"overall_summary\": [0.22246638079235992, 0.10511421954825888, 0.09703240451273881, 0.07014026554500949, 0.04900682855284877, 0.021529117936902008, 0.01646453914640017, 0.015141408944072367], \"overall_imp\": [\"Pregnancies\", \"Age\", \"BMI\", \"SerumInsulin\", \"PlasmaGlucose\", \"TricepsThickness\", \"DiastolicBloodPressure\", \"DiabetesPedigree\"], \"per_class_summary\": [[0.22246638079235992, 0.10511421954825888, 0.09703240451273884, 0.07014026554500945, 0.04900682855284877, 0.021529117936902008, 0.016464539146400163, 0.01514140894407237], [0.22246638079235992, 0.10511421954825888, 0.0970324045127388, 0.07014026554500953, 0.049006828552848776, 0.021529117936902008, 0.016464539146400173, 0.015141408944072364]], \"per_class_imp\": [[\"Pregnancies\", \"Age\", \"BMI\", \"SerumInsulin\", \"PlasmaGlucose\", \"TricepsThickness\", \"DiastolicBloodPressure\", \"DiabetesPedigree\"], [\"Pregnancies\", \"Age\", \"BMI\", \"SerumInsulin\", \"PlasmaGlucose\", \"TricepsThickness\", \"DiastolicBloodPressure\", \"DiabetesPedigree\"]]}]}]}], \"run_logs\": \"[2022-12-14T09:37:58.597991] Entering context manager injector.\\nCannot provide tracer without any exporter configured.\\n/azureml-envs/azureml_4612c5564caffc186e5bd2016e43147c/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.\\n  from cryptography.hazmat.backends import default_backend\\n[2022-12-14T09:37:59.188006] context_manager_injector.py Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError'], invocation=['diabetes_training.py'])\\nScript type = None\\n[2022-12-14T09:37:59.190431] Entering Run History Context Manager.\\n/azureml-envs/azureml_4612c5564caffc186e5bd2016e43147c/lib/python3.6/site-packages/azureml/history/_tracking.py:367: FutureWarning: azureml.core: AzureML support for Python 3.6 is deprecated and will be dropped in an upcoming release. At that point, existing Python 3.6 workflows that use AzureML will continue to work without modification, but Python 3.6 users will no longer get access to the latest AzureML features and bugfixes. We recommend that you upgrade to Python 3.7 or newer. To disable SDK V1 deprecation warning set the environment variable AZUREML_DEPRECATE_WARNING to 'False'\\n  from azureml.core.run import Run\\n[2022-12-14T09:37:59.910092] Current directory: /azureml-run\\n[2022-12-14T09:37:59.910122] Preparing to call script [diabetes_training.py] with arguments:[]\\n[2022-12-14T09:37:59.910140] After variable expansion, calling script [diabetes_training.py] with arguments:[]\\n\\nCould not import lightgbm, required if using LGBMExplainableModel\\nLoading Data...\\nTraining a decision tree model\\n\\n\\n[2022-12-14T09:38:03.006903] The experiment completed successfully. Finalizing run...\\n[2022-12-14T09:38:03.006923] Start FinalizingInRunHistory\\n[2022-12-14T09:38:03.007979] Logging experiment finalizing status in history service.\\nStarting the daemon thread to refresh tokens in background for process with pid = 8\\nCleaning up all outstanding Run operations, waiting 300.0 seconds\\n2 items cleaning up...\\nCleanup took 0.11562705039978027 seconds\\n[2022-12-14T09:38:04.025850] Finished context manager injector.\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.44.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'runId': 'mslearn-diabetes-explain_1671010520_e61c954c',\n",
       " 'target': 'local',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2022-12-14T09:37:57.733816Z',\n",
       " 'endTimeUtc': '2022-12-14T09:38:05.498851Z',\n",
       " 'services': {},\n",
       " 'properties': {'_azureml.ComputeTargetType': 'local',\n",
       "  'ContentSnapshotId': '7990df8f-e062-4cdd-97ed-ec634ce94b7b'},\n",
       " 'inputDatasets': [],\n",
       " 'outputDatasets': [],\n",
       " 'runDefinition': {'script': 'diabetes_training.py',\n",
       "  'command': '',\n",
       "  'useAbsolutePath': False,\n",
       "  'arguments': [],\n",
       "  'sourceDirectoryDataStore': None,\n",
       "  'framework': 'Python',\n",
       "  'communicator': 'None',\n",
       "  'target': 'local',\n",
       "  'dataReferences': {},\n",
       "  'data': {},\n",
       "  'outputData': {},\n",
       "  'datacaches': [],\n",
       "  'jobName': None,\n",
       "  'maxRunDurationSeconds': 2592000,\n",
       "  'nodeCount': 1,\n",
       "  'instanceTypes': [],\n",
       "  'priority': None,\n",
       "  'credentialPassthrough': False,\n",
       "  'identity': None,\n",
       "  'environment': {'name': 'explain_env',\n",
       "   'version': 'Autosave_2022-12-14T09:35:21Z_dc3a6559',\n",
       "   'assetId': 'azureml://locations/westeurope/workspaces/8c877630-3281-4ea1-89cf-f8f8040c7370/environments/explain_env/versions/Autosave_2022-12-14T09:35:21Z_dc3a6559',\n",
       "   'autoRebuild': True,\n",
       "   'python': {'interpreterPath': 'python',\n",
       "    'userManagedDependencies': False,\n",
       "    'condaDependencies': {'dependencies': ['python=3.6.2',\n",
       "      'scikit-learn',\n",
       "      'pandas',\n",
       "      'pip',\n",
       "      {'pip': ['azureml-defaults', 'azureml-interpret']}],\n",
       "     'name': 'batch_environment'},\n",
       "    'baseCondaEnvironment': None},\n",
       "   'environmentVariables': {'EXAMPLE_ENV_VAR': 'EXAMPLE_VALUE'},\n",
       "   'docker': {'baseImage': 'mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:20220708.v1',\n",
       "    'platform': {'os': 'Linux', 'architecture': 'amd64'},\n",
       "    'baseDockerfile': None,\n",
       "    'baseImageRegistry': {'address': None, 'username': None, 'password': None},\n",
       "    'enabled': False,\n",
       "    'arguments': []},\n",
       "   'spark': {'repositories': [], 'packages': [], 'precachePackages': True},\n",
       "   'inferencingStackVersion': None},\n",
       "  'history': {'outputCollection': True,\n",
       "   'directoriesToWatch': ['logs'],\n",
       "   'enableMLflowTracking': True,\n",
       "   'snapshotProject': True},\n",
       "  'spark': {'configuration': {'spark.app.name': 'Azure ML Experiment',\n",
       "    'spark.yarn.maxAppAttempts': '1'}},\n",
       "  'parallelTask': {'maxRetriesPerWorker': 0,\n",
       "   'workerCountPerNode': 1,\n",
       "   'terminalExitCodes': None,\n",
       "   'configuration': {}},\n",
       "  'amlCompute': {'name': None,\n",
       "   'vmSize': None,\n",
       "   'retainCluster': False,\n",
       "   'clusterMaxNodeCount': None},\n",
       "  'aiSuperComputer': {'instanceType': 'D2',\n",
       "   'imageVersion': None,\n",
       "   'location': None,\n",
       "   'aiSuperComputerStorageData': None,\n",
       "   'interactive': False,\n",
       "   'scalePolicy': None,\n",
       "   'virtualClusterArmId': None,\n",
       "   'tensorboardLogDirectory': None,\n",
       "   'sshPublicKey': None,\n",
       "   'sshPublicKeys': None,\n",
       "   'enableAzmlInt': True,\n",
       "   'priority': 'Medium',\n",
       "   'slaTier': 'Standard',\n",
       "   'userAlias': None},\n",
       "  'kubernetesCompute': {'instanceType': None},\n",
       "  'tensorflow': {'workerCount': 1, 'parameterServerCount': 1},\n",
       "  'mpi': {'processCountPerNode': 1},\n",
       "  'pyTorch': {'communicationBackend': 'nccl', 'processCount': None},\n",
       "  'hdi': {'yarnDeployMode': 'Cluster'},\n",
       "  'containerInstance': {'region': None, 'cpuCores': 2.0, 'memoryGb': 3.5},\n",
       "  'exposedPorts': None,\n",
       "  'docker': {'useDocker': True,\n",
       "   'sharedVolumes': True,\n",
       "   'shmSize': '2g',\n",
       "   'arguments': []},\n",
       "  'cmk8sCompute': {'configuration': {}},\n",
       "  'commandReturnCodeConfig': {'returnCode': 'Zero',\n",
       "   'successfulReturnCodes': []},\n",
       "  'environmentVariables': {},\n",
       "  'applicationEndpoints': {},\n",
       "  'parameters': []},\n",
       " 'logFiles': {'azureml-logs/60_control_log.txt': 'https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/azureml-logs/60_control_log.txt?sv=2019-07-07&sr=b&sig=vGMBvKYu%2BOhHzgol3ttw94gUOC%2Ft%2B6OYkjOZW06P6P8%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-14T09%3A28%3A04Z&ske=2022-12-15T17%3A38%3A04Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A28%3A12Z&se=2022-12-14T17%3A38%3A12Z&sp=r',\n",
       "  'azureml-logs/70_driver_log.txt': 'https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/azureml-logs/70_driver_log.txt?sv=2019-07-07&sr=b&sig=%2BEoB570DjYUB64GMRFgHOWwWY3z4Z%2F1NH6JXNKAZHws%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-14T09%3A28%3A04Z&ske=2022-12-15T17%3A38%3A04Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A28%3A12Z&se=2022-12-14T17%3A38%3A12Z&sp=r',\n",
       "  'logs/azureml/8_azureml.log': 'https://mllearningwork3345488027.blob.core.windows.net/azureml/ExperimentRun/dcid.mslearn-diabetes-explain_1671010520_e61c954c/logs/azureml/8_azureml.log?sv=2019-07-07&sr=b&sig=%2FJt3vWPUoSoWAfrxS95xI9qKV%2BTlIlfElaPhRvP5ugE%3D&skoid=530a969d-1f1a-4d2a-9866-6f18d23e261b&sktid=16b3c013-d300-468d-ac64-7eda0820b6d3&skt=2022-12-13T17%3A50%3A41Z&ske=2022-12-15T02%3A00%3A41Z&sks=b&skv=2019-07-07&st=2022-12-14T09%3A28%3A04Z&se=2022-12-14T17%3A38%3A04Z&sp=r'},\n",
       " 'submittedBy': 'Carlo Binda'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core import Experiment, ScriptRunConfig, Environment\n",
    "from azureml.core.runconfig import DockerConfiguration\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "\n",
    "# Create a Python environment for the experiment\n",
    "explain_env = Environment.from_conda_specification(\"explain_env\", experiment_folder + \"/interpret_env.yml\")\n",
    "\n",
    "# Create a script config\n",
    "script_config = ScriptRunConfig(source_directory=experiment_folder,\n",
    "                      script='diabetes_training.py',\n",
    "                      environment=explain_env,\n",
    "                      docker_runtime_config=DockerConfiguration(use_docker=True)) \n",
    "\n",
    "# submit the experiment\n",
    "experiment_name = 'mslearn-diabetes-explain'\n",
    "experiment = Experiment(workspace=ws, name=experiment_name)\n",
    "run = experiment.submit(config=script_config)\n",
    "RunDetails(run).show()\n",
    "run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve the feature importance values\n",
    "\n",
    "With the experiment run completed, you can use the **ExplanationClient** class to retrieve the feature importance from the explanation registered for the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature\tImportance\n",
      "Pregnancies \t 0.22246638079235992\n",
      "Age \t 0.10511421954825888\n",
      "BMI \t 0.09703240451273881\n",
      "SerumInsulin \t 0.07014026554500949\n",
      "PlasmaGlucose \t 0.04900682855284877\n",
      "TricepsThickness \t 0.021529117936902008\n",
      "DiastolicBloodPressure \t 0.01646453914640017\n",
      "DiabetesPedigree \t 0.015141408944072367\n"
     ]
    }
   ],
   "source": [
    "from azureml.interpret import ExplanationClient\n",
    "\n",
    "# Get the feature explanations\n",
    "client = ExplanationClient.from_run(run)\n",
    "engineered_explanations = client.download_model_explanation()\n",
    "feature_importances = engineered_explanations.get_feature_importance_dict()\n",
    "\n",
    "# Overall feature importance\n",
    "print('Feature\\tImportance')\n",
    "for key, value in feature_importances.items():\n",
    "    print(key, '\\t', value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the model explanation in Azure Machine Learning studio\n",
    "\n",
    "You can also click the **View run details** link in the Run Details widget to see the run in Azure Machine Learning studio, and view the **Explanations** tab. Then:\n",
    "\n",
    "1. Select the explanation ID for your tabular explainer.\n",
    "2. View the **Aggregate feature importance** chart, which shows the overall global feature importance.\n",
    "3. View the **Individual feature importance** chart, which shows each data point from the test data.\n",
    "4. Select an individual point to see the local feature importance for the individual prediction for the selected data point.\n",
    "5. Use the **New Cohort** button to define a subset of the data with the following settings:\n",
    "    - **Dataset cohort name**: Under 25s\n",
    "    - **Select filter**: Dataset\n",
    "        - Age less than 25 (Make sure you add this filter before saving the new cohort).\n",
    "6. Create a second new cohort named **25 and over** with a filter on Age greater than or equal to 25.\n",
    "6. Review the **Aggregate feature importance** visualization and compare the relative feature importance for the two cohorts you have defined. The ability to compare cohorts makes it possible to see how the features influence preedictions differently for multiple subsets of the data population.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**More Information**: For more information about using explainers in Azure ML, see [the documentation](https://docs.microsoft.com/azure/machine-learning/how-to-machine-learning-interpretability). "
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python38-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}