01_setup_azure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Setup Azure\n",
    "In this notebook, we will set up the project in Azure by creating the resources we need.\n",
    "\n",
    "We start by creating the resource group that we'll put all resources into. Then we create our storage account which will be used to store all data (include logs). Finally we set up our Service Bus namespace, and a queue attached to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages and load .env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from dotenv import set_key, get_key, find_dotenv, load_dotenv\n",
    "from pathlib import Path\n",
    "import os\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env_path = find_dotenv()\n",
    "if env_path=='':\n",
    "    Path('.env').touch()\n",
    "    env_path = find_dotenv()\n",
    "load_dotenv(env_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set variables for the project:\n",
    "\n",
    "- `subscription_id` - the subscription id for your Azure account. Use `az account list -o table` to list all subscriptions\n",
    "- `resource_group` - the name for the resource group you'll be using for this project. You can think of resource groups as logical containers for the resources you'll create in this tutorial.\n",
    "- `region` - the region you wish to deploy your resources in:\n",
    "    - You can see a list of the regions under the key 'name' when running the command `az account list-locations`. \n",
    "    - Not all regions support GPU enabled VMs. You can check [here](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/)\n",
    "    \n",
    "Set your storage account variables\n",
    "\n",
    "- `storage_account_name` - the desired name of your storage account\n",
    "- `storage_container_name` - the desired container name in the storage account is a logical container for individual blobs. For this project, we'll store all content into a single container for ease-of-use\n",
    "- `model_dir` - the desired name of directory you wish to store your models\n",
    "\n",
    "Set the Service Bus variables:\n",
    "\n",
    "- `namespace` - the desired namespace for your Service Bus - this can simply be thought of as a logical container for Service Bus\n",
    "- `queue` - the desired name of the queue. The queue belongs to the namespace.\n",
    "\n",
    "Set your docker container variables:\n",
    "- `docker_login` - the login username for your docker account\n",
    "- `aks_image_repo` - the desired image repo to use for your aks container\n",
    "- `aci_image_repo` - the desired image repo to use for your aci container\n",
    "\n",
    "AKS variables:\n",
    "- `aks_cluster` - the desired name of the aks cluster\n",
    "\n",
    "Logic App deployment:\n",
    "- `logic_app` - the desired name of your logic apps\n",
    "- `aci_group` - the desired name of your aci group\n",
    "- `aci_display_name` - the display name used for your aci group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "subscription_id = \"<your-subscription-id>\"           # fill in\n",
    "resource_group = \"<a-resource-group-name>\"           # fill in\n",
    "region = \"<selected-region-name>\"                    # fill in\n",
    "\n",
    "storage_account_name = \"batchscoringdlsa\"            # feel free to replace or use this default\n",
    "storage_container_name = \"aks\"                       # feel free to replace or use this default\n",
    "model_dir = \"models\"                                 # feel free to replace or use this default\n",
    "\n",
    "namespace = \"batchscoringdlnamespace\"                # feel free to replace or use this default\n",
    "queue = \"batchscoringdlqueue\"                        # feel free to replace or use this default\n",
    "\n",
    "docker_login = \"<docker-login>\"                      # fill in\n",
    "scoring_image_repo = \"batchscoringdl_scoring_app\"    # feel free to replace or use this default\n",
    "flask_image_repo = \"batchscoringdl_flask_app\"        # feel free to replace or use this default\n",
    "\n",
    "aks_cluster = \"batchscoringdlcluster\"                # feel free to replace or use this default\n",
    "logic_app = \"batchscoringdlla\"                       # feel free to replace or use this default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "set_key(env_path, \"SUBSCRIPTION_ID\", subscription_id)\n",
    "set_key(env_path, \"RESOURCE_GROUP\", resource_group)\n",
    "set_key(env_path, \"REGION\", region)\n",
    "set_key(env_path, \"STORAGE_ACCOUNT_NAME\", storage_account_name)\n",
    "set_key(env_path, \"STORAGE_CONTAINER_NAME\", storage_container_name)\n",
    "set_key(env_path, \"STORAGE_MODEL_DIR\", model_dir)\n",
    "set_key(env_path, \"SB_NAMESPACE\", namespace)\n",
    "set_key(env_path, \"SB_QUEUE\", queue)\n",
    "set_key(env_path, \"DOCKER_LOGIN\", docker_login)\n",
    "set_key(env_path, \"SCORING_IMAGE\", scoring_image_repo)\n",
    "set_key(env_path, \"FLASK_IMAGE\", flask_image_repo)\n",
    "set_key(env_path, \"AKS_CLUSTER\", aks_cluster)\n",
    "set_key(env_path, \"LOGIC_APP\", logic_app)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up your resource group\n",
    "Make sure you've identified which subscription_id and region to use. Create a new resource group to contain all the resources that we create.\n",
    "\n",
    "This section of the notebook will walk through setting up the resource group using the __az cli__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!az account set -s {subscription_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the resource group which we'll put our storage account and all other resources for this project into."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!az group create -l {region} -n {resource_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure blob storage\n",
    "In this section of the notebook, we'll create an Azure blob storage that we'll use throughout the tutorial. This object store will be used to store input and output images as well as any supplementary data such as logs and other scripts that will be used in this workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the __az cli__ to create the account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K\"Succeeded\" ..\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!az storage account create -n {storage_account_name} -g {resource_group} --query 'provisioningState'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the __az cli__ to grab the keys of the storage account that was just created. The `--quote '[0].value'` part of the command simply means to select the _value_ of the _zero-th indexed_ of the set of keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "key = !az storage account keys list --account-name {storage_account_name} -g {resource_group} --query '[0].value'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The stdout from the command above is stored in a string array of 1. Select the element in the array and ttrip opening and closing quotation marks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage_account_key = str(key[0][1:-1]) # this is used to strip opening and closing quotation marks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the __az cli__ to create the container in the storage account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"created\": true\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!az storage container create \\\n",
    "    --account-name {storage_account_name} \\\n",
    "    --account-key {storage_account_key} \\\n",
    "    --name {storage_container_name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set storage account key to dotenv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "set_key(env_path, \"STORAGE_ACCOUNT_KEY\", storage_account_key) # generated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Service Bus & Generated Keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!az servicebus namespace create \\\n",
    "    --resource-group {resource_group} \\\n",
    "    --name {namespace} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a service bus queue. Set the lock duration to 5 minutes. This means that the lock for each queue message will last for 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!az servicebus queue create \\\n",
    "    --resource-group {resource_group} \\\n",
    "    --namespace-name {namespace} \\\n",
    "    --name {queue} \\\n",
    "    --lock-duration PT5M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've created our Service Bus namespace and queue, we need to get the key-name/key-value pair so that we can access it.\n",
    "\n",
    "By default, your Service Bus resource will come with an key-value \"authorization rule\" pair - its key name will have the value: \"RootManageSharedAccessKey\". The following command will get the key name for the \"authorization rule\", and assign it as `sb_key_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "sb_key_name = !az servicebus namespace authorization-rule list \\\n",
    "    --resource-group {resource_group} \\\n",
    "    --namespace-name {namespace} \\\n",
    "    -o json --query \"[0].name\"\n",
    "\n",
    "sb_key_name = str(sb_key_name[0][1:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the primary key value to \"RootManageSharedAccessKey\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sb_credentials = !az servicebus namespace authorization-rule keys list \\\n",
    "    --resource-group {resource_group} \\\n",
    "    --namespace-name {namespace} \\\n",
    "    --name {sb_key_name} \\\n",
    "    -o json --query \"primaryKey\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sb_key_value = re.findall(r'\"(.*?)\"', str(sb_credentials))[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the service bus key value to the dotenv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "set_key(env_path, \"SB_SHARED_ACCESS_KEY_VALUE\", sb_key_value) # generated\n",
    "set_key(env_path, \"SB_SHARED_ACCESS_KEY_NAME\", sb_key_name) # generated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set environment variables to be used by later notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that our `.env` file looks correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!cat .env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up Azure Blob as a File System on our local machine using Blobfuse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install blob fuse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "ubuntu_version = !lsb_release -r | grep -o '[0-9][0-9].[0-9][0-9]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-01-22 16:59:27--  https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb\n",
      "Resolving packages.microsoft.com (packages.microsoft.com)... 40.76.35.62\n",
      "Connecting to packages.microsoft.com (packages.microsoft.com)|40.76.35.62|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3132 (3.1K) [application/octet-stream]\n",
      "Saving to: ‘packages-microsoft-prod.deb’\n",
      "\n",
      "packages-microsoft- 100%[===================>]   3.06K  --.-KB/s    in 0s      \n",
      "\n",
      "2019-01-22 16:59:27 (664 MB/s) - ‘packages-microsoft-prod.deb’ saved [3132/3132]\n",
      "\n",
      "(Reading database ... 233115 files and directories currently installed.)\n",
      "Preparing to unpack packages-microsoft-prod.deb ...\n",
      "Unpacking packages-microsoft-prod (1.0-ubuntu18.04.2) over (1.0-ubuntu18.04.2) ...\n",
      "Setting up packages-microsoft-prod (1.0-ubuntu18.04.2) ...\n",
      "Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease\n",
      "Hit:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease          \n",
      "Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n",
      "Hit:4 https://packages.microsoft.com/repos/azure-cli bionic InRelease          \n",
      "Hit:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease        \n",
      "Hit:6 https://nvidia.github.io/libnvidia-container/ubuntu18.04/amd64  InRelease\n",
      "Hit:7 https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/amd64  InRelease\n",
      "Get:8 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]    \n",
      "Hit:9 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease      \n",
      "Hit:10 https://download.docker.com/linux/ubuntu bionic InRelease               \n",
      "Ign:11 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
      "Hit:12 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
      "Hit:13 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
      "Fetched 158 kB in 1s (189 kB/s)\n",
      "Reading package lists... Done\n"
     ]
    }
   ],
   "source": [
    "!wget https://packages.microsoft.com/config/ubuntu/{ubuntu_version[0]}/packages-microsoft-prod.deb\n",
    "!sudo dpkg -i packages-microsoft-prod.deb\n",
    "!sudo apt-get update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "blobfuse is already the newest version (1.0.2).\n",
      "The following packages were automatically installed and are no longer required:\n",
      "  linux-azure-cloud-tools-4.15.0-1031 linux-azure-cloud-tools-4.15.0-1032\n",
      "  linux-azure-headers-4.15.0-1031 linux-azure-headers-4.15.0-1032\n",
      "  linux-azure-tools-4.15.0-1031 linux-azure-tools-4.15.0-1032\n",
      "Use 'sudo apt autoremove' to remove them.\n",
      "0 upgraded, 0 newly installed, 0 to remove and 114 not upgraded.\n"
     ]
    }
   ],
   "source": [
    "!sudo apt-get install blobfuse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup ramdisk tmp dir for blob fuse for low latency buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove '/mnt/ramdisk': Device or resource busy\r\n"
     ]
    }
   ],
   "source": [
    "!sudo rm -rf /mnt/ramdisk && sudo mkdir /mnt/ramdisk\n",
    "!sudo mount -t tmpfs -o size=4g tmpfs /mnt/ramdisk\n",
    "!sudo mkdir /mnt/ramdisk/blobfusetmp\n",
    "!sudo chown $(whoami) /mnt/ramdisk/blobfusetmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup credentials for blob connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -f fuse_connection.cfg\n",
    "!touch fuse_connection.cfg\n",
    "!echo accountName {get_key(env_path, \"STORAGE_ACCOUNT_NAME\")} >> fuse_connection.cfg\n",
    "!echo accountKey {get_key(env_path, \"STORAGE_ACCOUNT_KEY\")} >> fuse_connection.cfg\n",
    "!echo containerName {get_key(env_path, \"STORAGE_CONTAINER_NAME\")} >> fuse_connection.cfg\n",
    "!chmod 700 fuse_connection.cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create mount directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf data && mkdir data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mount blob to mount directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "!blobfuse data \\\n",
    "    --tmp-path=/mnt/ramdisk/blobfusetmp  \\\n",
    "    --config-file=fuse_connection.cfg \\\n",
    "    -o attr_timeout=240 \\\n",
    "    -o entry_timeout=240 \\\n",
    "    -o negative_timeout=120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Model Dir and Video to storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy `models` dir to the `data` folder we just mounted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp -r -n models data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download an `orangutan.mp4` video to use throughout the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-01-22 16:59:34--  https://happypathspublic.blob.core.windows.net/videos/orangutan.mp4\n",
      "Resolving happypathspublic.blob.core.windows.net (happypathspublic.blob.core.windows.net)... 52.239.214.164\n",
      "Connecting to happypathspublic.blob.core.windows.net (happypathspublic.blob.core.windows.net)|52.239.214.164|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 7961293 (7.6M) [video/mp4]\n",
      "Saving to: ‘orangutan.mp4’\n",
      "\n",
      "orangutan.mp4       100%[===================>]   7.59M  --.-KB/s    in 0.05s   \n",
      "\n",
      "2019-01-22 16:59:34 (140 MB/s) - ‘orangutan.mp4’ saved [7961293/7961293]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://happypathspublic.blob.core.windows.net/videos/orangutan.mp4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move that video into the mounted `data` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv orangutan.mp4 data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that our models dir and orangutan.mp4 is uploaded to our mounted blob storage container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models\torangutan.mp4\r\n"
     ]
    }
   ],
   "source": [
    "!ls data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continue to the next [notebook](/notebooks/02_local_testing.ipynb)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [conda env:batchscoringdl_aks]",
   "language": "python",
   "name": "conda-env-batchscoringdl_aks-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}