hands-on-aws-3-stepfunctions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on AWS - 3 - Step Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing dependencies\n",
    "**Here we install relevant dependencies to run serverless framework.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install awscli --upgrade --user\n",
    "!npm install -g serverless\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting AWS environmental variables\n",
    "**Here we set up AWS environmental variables so that we will be able to deploy to our AWS account. Please replace it with provided credentials.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>\n",
    "%env AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>\n",
    "%env AWS_ACCOUNT_ID=<AWS_ACCOUNT_ID>"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Setting up identifier\n",
    "**Here we set up personal identifier. You will need to replace \"FirstName LastName\" with your first name and lastname.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import hashlib\n",
    "import datetime\n",
    "res=hashlib.sha224(bytes(\"FirstName LastName\"+str(datetime.datetime.now().timestamp()),'utf-8')).hexdigest()[:5]\n",
    "%env HASH=$res\n",
    "%env AWS_DEFAULT_REGION=us-west-1"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Step Functions + Lambda deployment\n",
    "\n",
    "**Let's take a look at the code which will run within lambda. The code of map handler will generate events, task handler will process them and reduce handler will aggregate the results.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!cat hands-on-aws-3-stepfunctions/index.py"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's deploy the lambda**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd hands-on-aws-3-stepfunctions;npm install\n",
    "!cd hands-on-aws-3-stepfunctions;serverless deploy --verbose > logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Let's view the output of the deployment process. Serverless framework will return endpoint for the step funtions which we created**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!cd hands-on-aws-3-stepfunctions;tail -15 logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Invoking Step Functions and looking at the logs\n",
    "\n",
    "**Now let's make simple request to the endpoint and check that it starts Step Function.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl $(cd hands-on-aws-3-stepfunctions;tail -15 logs.txt | grep \"ServiceEndpoint:\" | cut -f2- -d:)/startFunction\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's take a look at the latest logs for the function which generates events.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!cd hands-on-aws-3-stepfunctions;serverless logs -f LambdaMap\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's take a look at the latest logs for the function which processes events in parallel.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!cd hands-on-aws-3-stepfunctions;serverless logs -f LambdaTask\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's take a look at the latest logs for the function which assembles the results from the parallel lambdas.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!cd hands-on-aws-3-stepfunctions;serverless logs -f LambdaReduce\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove infrastructure\n",
    "\n",
    "**Now let's do the cleanup process and remove the infrastructure which we created.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!cd hands-on-aws-3-stepfunctions;serverless remove > logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}