Designer-Sample1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build AML Pipeline with custom module and built-in module\n",
    "\n",
    "In this tutorial you will learn how to use Designer built-in module and custom module together to create a pipeline.\n",
    "\n",
    "1. Setup enrivonment - install module CLI and module/pipeline SDK\n",
    "2. Register custom modules into your aml workspace using CLI\n",
    "3. Use module/pipeline SDK to create a pipeline with modules registered in step 2 and the built-in module available in AML designer\n",
    "\n",
    "## Prerequisite\n",
    "* Install Azure CLI, please follow [the Azure CLI installation instructions](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) to install."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "* Install Azure CLI AML extension which includes the _module_ command group\n",
    "* Install Azure ML SDK including the APIs to work with _module_ and _pipeline_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uninstall azure-cli-ml (the `az ml` commands)\n",
    "!az extension remove -n azure-cli-ml \n",
    "\n",
    "# Install local version of azure-cli-ml (which includes `az ml module` commands)\n",
    "!az extension add --source https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101/azure_cli_ml-0.1.0.15510101-py3-none-any.whl --pip-extra-index-urls https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101 --yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify the availability of `az ml module` commands\n",
    "!az ml module -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install azureml-sdk with Pipeline, Module\n",
    "# Important! After install succeed, need to restart kernel\n",
    "\n",
    "%config IPCompleter.greedy=True \n",
    "!pip install azureml-pipeline-wrapper[notebooks]==0.1.0.15510101 --extra-index-url https://azuremlsdktestpypi.azureedge.net/CLI-SDK-Runners-Validation/15510101 --user --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you need to configure your ws information here\n",
    "\n",
    "subscription_id = '74eccef0-4b8d-4f83-b5f9-fa100d155b22'\n",
    "workspace_name = 'lisal-amlservice'\n",
    "resource_group = 'lisal-dev'\n",
    "\n",
    "# Specify available aml compute in workspace\n",
    "pipeline_compute = \"always-on-ds2v2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register azureml module\n",
    "\n",
    "You can manage AML module through [azure-cli-ml](https://aka.ms/moduledoc) or [ml.azure.com](https://ml.azure.com/). <br>\n",
    "\n",
    "Module could be registered from:\n",
    "- local path\n",
    "- public Github url\n",
    "- Azure DevOps build artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure your aml workspace \n",
    "\n",
    "!az login \n",
    "!az account set -s $subscription_id \n",
    "!az ml folder attach -w $workspace_name -g $resource_group "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# register a custom module\n",
    "\n",
    "!az ml module register --spec-file=https://github.com/lisagreenview/hello-aml-modules/blob/master/train-score-eval/eval.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from azureml.data.data_reference import DataReference\n",
    "from azureml.core import Workspace, Run, Dataset, Datastore\n",
    "from azureml.pipeline.wrapper import Pipeline, Module, dsl\n",
    "\n",
    "ws = Workspace.get(name=workspace_name, subscription_id=subscription_id, resource_group=resource_group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get built-in module\n",
    "select_column_func = Module.load(ws, namespace='azureml', name='Select Columns in Dataset')\n",
    "clean_data_func = Module.load(ws, namespace='azureml', name='Clean Missing Data')\n",
    "split_data_func = Module.load(ws, namespace='azureml', name='Split Data')\n",
    "linear_regression_func = Module.load(ws, namespace='azureml', name='Linear Regression')\n",
    "train_func = Module.load(ws, namespace='azureml', name='Train Model')\n",
    "score_func = Module.load(ws, namespace='azureml', name='Score Model')\n",
    "eval_func = Module.load(ws, namespace='azureml', name='Evaluate Model')\n",
    "\n",
    "#get custom module\n",
    "my_eval = Module.load(ws, namespace='microsoft.com/aml/samples', name='Evaluate')\n",
    "\n",
    "\n",
    "data_name = 'Automobile_price_data'\n",
    "\n",
    "if data_name not in ws.datasets:\n",
    "    global_datastore = Datastore(ws, name=\"azureml_globaldatasets\")\n",
    "    automobile_data = Dataset.File.from_files(global_datastore.path('GenericCSV/Automobile_price_data_(Raw)')).register(workspace=ws, name='Automobile_price_data', description='Automobile_price_data')\n",
    "    print('Registerd')\n",
    "\n",
    "blob_input_data = ws.datasets[data_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define your pipeline\n",
    "\n",
    "@dsl.pipeline(name = 'Designer Sample 1', \n",
    "              description = 'Regression - Automobile Price Prediction',\n",
    "              default_compute_target = pipeline_compute)\n",
    "def sample1_pipeline():\n",
    "    select = select_column_func(\n",
    "        dataset=blob_input_data, \n",
    "        select_columns='[{\"ColumnNames\":[\"normalized-losses\"],\"Exclude\":true}]'\n",
    "    )   \n",
    "    \n",
    "    clean = clean_data_func(\n",
    "        dataset=select.outputs.results_dataset,\n",
    "        columns_to_be_cleaned='{\"isFilter\":true,\"rules\":[{\"ruleType\":\"AllColumns\",\"exclude\":false}]}',\n",
    "        minimum_missing_value_ratio=0.0,\n",
    "        maximum_missing_value_ratio=1.0,\n",
    "        cleaning_mode='Remove entire row'\n",
    "    )\n",
    "    \n",
    "    split = split_data_func(\n",
    "        dataset=clean.outputs.cleaned_dataset,\n",
    "        splitting_mode='Split Rows',\n",
    "        fraction_of_rows_in_the_first_output_dataset=0.7,\n",
    "        randomized_split='True',\n",
    "        stratified_split='False'\n",
    "    )\n",
    "    \n",
    "    algo = linear_regression_func(\n",
    "        solution_method='Ordinary Least Squares',\n",
    "        l2_regularization_weight=0.001,\n",
    "        include_intercept_term='True',\n",
    "        random_number_seed=0\n",
    "    )\n",
    "    \n",
    "    train = train_func(\n",
    "        dataset=split.outputs.results_dataset1,\n",
    "        untrained_model=algo.outputs.untrained_model,\n",
    "        label_column='[{\"ColumnNames\":[\"price\"]}]'\n",
    "    )\n",
    "    \n",
    "    score = score_func(\n",
    "        trained_model=train.outputs.trained_model,\n",
    "        dataset=split.outputs.results_dataset2,\n",
    "        append_score_columns_to_output='True'\n",
    "    )\n",
    "    \n",
    "    eval_1 = my_eval(\n",
    "        scoring_result=score.outputs.scored_dataset\n",
    "    )\n",
    "       \n",
    "    eval_2 = eval_func(\n",
    "        scored_dataset = score.outputs.scored_dataset\n",
    "    ) \n",
    "    \n",
    "    return {**eval_1.outputs, **eval_2.outputs}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a pipeline\n",
    "pipeline = sample1_pipeline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# validate pipeline and visualize the graph\n",
    "pipeline.validate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save as a draft\n",
    "pipeline.save(experiment_name = 'SDK-Created-Samples', id='ef7acf77-cd4b-4336-b4a2-c85536faa778')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = pipeline.submit(experiment_name = 'SDK-Created-Samples',tags={'mode':'SDK'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run.wait_for_completion()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}