05_deploy_logic_app.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Logic App"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will perform the final deployment so that the whole architecture is on Azure.\n",
    "\n",
    "### Recap of the architecture:\n",
    "The architecture for this workflow is centered on two main components:\n",
    "- The AKS cluster that contains the flask app and the scoring app\n",
    "- Logic App that will trigger the workflow by hitting the flask app on the AKS cluster\n",
    "\n",
    "At this point, we already have an AKS cluster up and running. It is continuously polling out Service Bus queue to check if there are any incoming messages to process. \n",
    "\n",
    "The next step is to deploy our Logic App so that the workflow can be triggered by new videos that appear in blob storage. Once a new video is found in blob, Logic App will hit the AKS endpoint that will run the video preprocessing script, the add-to-queue script, and finally the postprocessing script. \n",
    "\n",
    "As soon as new items are added to the queue from the flask_app, our scoring pods in the AKS cluster will start pulling those items off and start processing them.\n",
    "\n",
    "### Steps in this notebook:\n",
    "1. Deploy the Logic App. The Logic App will go live once deployed. \n",
    "2. Test the Logic App deployment. To do so, we upload a new video to blob, which will triger the entire workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages and load .env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import set_key, get_key, find_dotenv, load_dotenv\n",
    "from pathlib import Path\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env_path = find_dotenv(raise_error_if_not_found=True)\n",
    "load_dotenv(env_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Logic App"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Logic Apps](https://happypathspublic.blob.core.windows.net/assets/batch_scoring_for_dl/azure_logic_app.jpg)\n",
    "\n",
    "The *logic* behind the Logic App deployment is shown above:\n",
    "1. When a blob is added, begin the workflow.\n",
    "2. Check the blob name. \n",
    "    - if the blob name ends with `.mp4`:\n",
    "        - make a request to the AKS endpoint\n",
    "    - otherwise:\n",
    "        - terminate in cancellation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the deployment for the Azure blob storage connector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!az group deployment create \\\n",
    "    --name blob_connector \\\n",
    "    --resource-group {get_key(env_path, \"RESOURCE_GROUP\")} \\\n",
    "    --template-file template.blob_connector.json \\\n",
    "    --parameters \\\n",
    "        location={get_key(env_path, \"REGION\")} \\\n",
    "        subscription_id={get_key(env_path, \"SUBSCRIPTION_ID\")} \\\n",
    "        storage_account_name={get_key(env_path, \"STORAGE_ACCOUNT_NAME\")} \\\n",
    "        storage_account_key={get_key(env_path, \"STORAGE_ACCOUNT_KEY\")}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the deployment for the Logic App."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": [
     "strip_out"
    ]
   },
   "outputs": [],
   "source": [
    "!az group deployment create \\\n",
    "    --name logic_app \\\n",
    "    --resource-group {get_key(env_path, \"RESOURCE_GROUP\")} \\\n",
    "    --template-file template.logic_app.json \\\n",
    "    --parameters \\\n",
    "        name={get_key(env_path, \"LOGIC_APP\")} \\\n",
    "        location={get_key(env_path, \"REGION\")} \\\n",
    "        resource_group={get_key(env_path, \"RESOURCE_GROUP\")} \\\n",
    "        subscription_id={get_key(env_path, \"SUBSCRIPTION_ID\")} \\\n",
    "        storage_container_name={get_key(env_path, \"STORAGE_CONTAINER_NAME\")} \\\n",
    "        url_endpoint={\"http://{}:8080/process\".format(get_key(env_path, \"AKS_EXTERNAL_IP\"))}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigger logic app by adding a new video to the Azure blob container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-upload the orangutan video to blob with a new name: `logic_app_test_orangutan.mp4`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_video_name = \"logic_app_test_orangutan.mp4\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a copy the old `orangutan.mp4` video but named with the `<new_video_name>`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp data/orangutan.mp4 data/{new_video_name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The appearance of the new video will trigger Logic Apps to kick off the whole workflow. Inspect the state of your cluster using `kubectl` or check on the files in your container in blob to monitor the progress.\n",
    "\n",
    "When the video is finished processing, we can view it directly on the mounted blob container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video width=\"320\" height=\"240\" controls>\n",
       "  <source src=\"data/logic_app_test_orangutan/logic_app_test_orangutan_processed.mp4\" type=\"video/mp4\">\n",
       "</video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<video width=\"320\" height=\"240\" controls>\n",
    "  <source src=\"data/logic_app_test_orangutan/logic_app_test_orangutan_processed.mp4\" type=\"video/mp4\">\n",
    "</video>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continue to the next [notebook](/notebooks/06_clean_up.ipynb)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [conda env:batchscoringdl_aks]",
   "language": "python",
   "name": "conda-env-batchscoringdl_aks-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}