GTC_2018_CoLab.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -*- coding: utf-8 -*-"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Detection of Anomalies in Financial Transactions using Deep Autoencoder Networks\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The content of this \"GPU Technology Conference (GTC) 2018, Silicon Valley\" lab was jointly developed by Marco Schreyer and Timur Sattarov. Please don't hesitate to contact us in case of any questions via <a href=\"mailto:marco.schreyer@dfki.de\">marco.schreyer@dfki.de</a> and <a href=\"mailto:sattarov.timur@pwc.com\">sattarov.timur@pwc.com</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction and Lab Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Fraud and Accounting Information Systems (AIS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Association of Certified Fraud Examiners estimates in its Global Fraud Study 2016 [1] that the typical organization loses 5% of its annual revenues due to fraud. According to Joseph T. Wells [2] the term **\"fraud\"** refers to, \n",
    "\n",
    ">_\"the abuse of one's occupation for personal enrichment through the deliberate misuse of an organization's resources or assets\"_. \n",
    "\n",
    "A similar more recent study, conducted by the auditors of PwC, revealed that 30% of the study respondents experienced losses of between \\$100,000 and \\$5 million USD [3] in the last 24 months. The study also showed that financial statement fraud caused by far the greatest median loss of the surveyed fraud schemes.\n",
    "\n",
    "At the same time organizations accelerate the digitization and reconfiguration of business processes [4] affecting in particular Accounting Information Systems (AIS) or more general Enterprise Resource Planning (ERP) systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 550px; height: auto\" src=\"images/accounting.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 1:** Hierarchical view of an Accounting Information System (AIS) that records distinct layers of abstraction, namely (1) the business process information, (2) the accounting information as well as the (3) technical journal entry information in designated database tables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Steadily, these systems collect vast quantities of electronic evidence at an almost atomic level. This holds in particular for the journal entries of an organization recorded in its general ledger and sub-ledger accounts. SAP, one of the most prominent ERP software providers, estimates that approx. 76% of the world's transaction revenue touches one of their systems [5].\n",
    "\n",
    "The illustration in **Figure 1** depicts a hierarchical view of an Accounting Information System (AIS) recording process and journal entry information in designated database tables. In the context of fraud examinations, the data collected by such systems may contain valuable traces of a potential fraud scheme."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Classification of Financial Anomalies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When conducting a detailed examination of real-world journal entries, usually recorded in large-scaled AIS or ERP systems, two prevalent characteristics can be observed:\n",
    "\n",
    "> - specific transactions attributes exhibit **a high variety of distinct attribute values** e.g. customer information, posted sub-ledgers, amount information, and \n",
    "> - the transactions exhibit **strong dependencies between specific attribute values** e.g. between customer information and type of payment, posting type and general ledgers. \n",
    "\n",
    "Derived from this observation we distinguish two classes of anomalous journal entries, namely **\"global\"** and **\"local\" anomalies** as illustrated in **Figure 2** below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 450px; height: auto\" src=\"images/anomalies.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 2:** Illustrative example of global and local anomalies portrait in a feature space of the two transaction features \"Posting Amount\" (Feature 1) and \"Posting Positions\" (Feature 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Global Anomalies***, are financial transactions that exhibit **unusual or rare individual attribute values**. These anomalies usually relate to highly skewed attributes e.g. seldom posting users, rarely used ledgers, or unusual posting times. \n",
    "\n",
    "Traditionally \"red-flag\" tests, performed by auditors during annual audits, are designed to capture those types of anomalies. However, such tests might result in a high volume of false positive alerts due to e.g. regular reverse postings, provisions and year-end adjustments usually associated with a low fraud risk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Local Anomalies***, are financial transactions that exhibit an **unusual or rare combination of attribute values** while the individual attribute values occur quite frequently e.g. unusual accounting records. \n",
    "\n",
    "This type of anomaly is significantly more difficult to detect since perpetrators intend to disguise their activities trying to imitate a regular behaviour. As a result, such anomalies usually pose a high fraud risk since they might correspond to e.g. misused user accounts, irregular combinations of general ledger accounts and posting keys that don't follow an usual activity pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Lab Objective and Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this lab is to walk you through a deep learning based methodology that can be used to detect of global and local anomalies in financial datasets. The proposed method is based on the following assumptions: \n",
    "\n",
    ">1. the majority of financial transactions recorded within an organizations’ ERP-system relate to regular day-to-day business activities and perpetrators need to deviate from the ”regular” in order to conduct fraud,\n",
    ">2. such deviating behaviour will be recorded by a very limited number of financial transactions and their respective attribute values or combination of attribute values and we refer to such deviation as \"anomaly\".\n",
    "\n",
    "Concluding from these assumptions we can learn a model of regular journal entries with minimal ”harm” caused by the potential anomalous ones.\n",
    "\n",
    "In order to detect such anomalies, we will train deep autoencoder networks to learn a compressed but \"lossy\" model of regular transactions and their underlying posting pattern. Imposing a strong regularization onto the network hidden layers limits the networks' ability to memorize the characteristics of anomalous journal entries. Once the training process is completed, the network will be able to reconstruct regular journal entries, while failing to do so for the anomalous ones.\n",
    "\n",
    "After completing the lab you should be familiar with:\n",
    "\n",
    ">1. the basic concepts, intuitions and major building blocks of autoencoder neural networks,\n",
    ">2. the techniques of pre-processing financial data in order to learn a model of its characteristics,\n",
    ">3. the application of autoencoder neural networks to detect anomalies in large-scale financial data, and,\n",
    ">4. the interpretation of the detection results of the networks as well as its reconstruction loss. \n",
    "\n",
    "Please note, that this lab is neither a complete nor comprehensive forensic data analysis approach or fraud examination strategy. However, the methodology and code provided in this lab can be modified or adapted to detect anomalous records in a variety of financial datasets. Subsequently, the detected records might serve as a starting point for a more detailed and substantive examination by auditors or compliance personnel. \n",
    "\n",
    "For this lab, we assume that you are familiar with the general concepts of deep neural networks (DNN) and GPUs as well as PyTorch and Python. For more information on these concepts please check the relevant labs of NVIDIA's Deep Learning Institute (DLI). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think about potential fraud scenarios of your organization:\n",
    "\n",
    ">1. What scenarios or fraudulent activities you could think of? [3 min]\n",
    ">2. What data sources might affect or record those potential fraudulent activities? [5 min]\n",
    ">3. What kind of data analytics techniques could be applied to detect those activities? [5 min]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Environment Setup and Verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Python Verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let's verify that Python is working on your system. To do this, execute the cell block below by giving it focus (clicking on it with your mouse), and hitting Shift-Enter, or pressing the play button in the toolbar above. If all goes well, you should see some output returned below the grey cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('The answer should be forty-two: {}'.format(str(40+2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Python Libraries Import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a next step let's import the libraries needed throughout the lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing utilities\n",
    "import os\n",
    "import sys\n",
    "from datetime import datetime\n",
    "import io\n",
    "import urllib\n",
    "\n",
    "# importing data science libraries\n",
    "import pandas as pd\n",
    "import random as rd\n",
    "import numpy as np\n",
    "\n",
    "# importing pytorch libraries\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import autograd\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# import visualization libraries\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import seaborn as sns\n",
    "from IPython.display import Image, display\n",
    "sns.set_style('darkgrid')\n",
    "\n",
    "# ignore potential warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Allow for Jupyter notebook inline plotting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 CUDNN and GPU Verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To determine if CDNN is available on the server let's execute the cell below to display information about the available CUDNN version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print CUDNN backend version\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] The CUDNN backend version: {}'.format(now, torch.backends.cudnn.version()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, let's display information about the potential GPUs running on the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If CUDNN and GPU's are available let's still specify if we want to use both:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_CUDA = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Python and PyTorch Verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's execute the cell below to display information about the Python and PyTorch version running on the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print current Python version\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] The Python version: {}'.format(now, sys.version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print current PyTorch version\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] The PyTorch version: {}'.format(now, torch.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Random Seed Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let' set the seeds of random elements in the code e.g. the initialization of the network parameters to guarantee deterministic computation and results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init deterministic seed\n",
    "seed_value = 1234 #4444 #3333 #2222 #1111 #1234\n",
    "rd.seed(seed_value) # set random seed\n",
    "np.random.seed(seed_value) # set numpy seed\n",
    "torch.manual_seed(seed_value) # set pytorch seed CPU\n",
    "if (torch.backends.cudnn.version() != None and USE_CUDA == True):\n",
    "    torch.cuda.manual_seed(seed_value) # set pytorch seed GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Financial Fraud Detection Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will conduct a descriptive analysis of the labs financial dataset. Furthermore, we will apply some necessary pre-processing steps to train a deep neural network. The lab is based on a derivation of the **\"Synthetic Financial Dataset For Fraud Detection\"** by Lopez-Rojas [6] available via the Kaggle predictive modelling and analytics competitions platform that can be obtained using the following link: https://www.kaggle.com/ntnu-testimon/paysim1.\n",
    "\n",
    "Let's start loading the dataset and investigate its structure and attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset into the notebook kernel\n",
    "url = 'https://raw.githubusercontent.com/GitiHubi/deepAI/master/data/fraud_dataset_v2.csv'\n",
    "ori_dataset = pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect the datasets dimensionalities\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] Transactional dataset of {} rows and {} columns loaded'.format(now, ori_dataset.shape[0], ori_dataset.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Initial Data and Attribute Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We augmented the dataset and renamed the attributes to appear more similar to a real-world dataset that one usually observes in SAP-ERP systems as part of SAP's Finance and Cost controlling (FICO) module. \n",
    "\n",
    "The dataset contains a subset of in total 7 categorical and 2 numerical attributes available in the FICO BKPF (containing the posted journal entry headers) and BSEG (containing the posted journal entry segments) tables. Please, find below a list of the individual attributes as well as a brief description of their respective semantics:\n",
    "\n",
    ">- `BELNR`: the accounting document number,\n",
    ">- `BUKRS`: the company code,\n",
    ">- `BSCHL`: the posting key,\n",
    ">- `HKONT`: the posted general ledger account,\n",
    ">- `PRCTR`: the posted profit center,\n",
    ">- `WAERS`: the currency key,\n",
    ">- `KTOSL`: the general ledger account key,\n",
    ">- `DMBTR`: the amount in local currency,\n",
    ">- `WRBTR`: the amount in document currency.\n",
    "\n",
    "Let's also have a closer look into the top 10 rows of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect top rows of dataset\n",
    "ori_dataset.head(10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may also have noticed the attribute `label` in the data. We will use this field throughout the lab to evaluate the quality of our trained models. The field describes the true nature of each individual transaction of either being a **regular** transaction (denoted by `regular`) or an **anomaly** (denoted by `global` and `local`). Let's have closer look into the distribution of the regular vs. anomalous transactions in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of anomalies vs. regular transactions\n",
    "ori_dataset.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, the statistic reveals that, similar to real world scenarios, we are facing a highly \"unbalanced\" dataset. Overall, the dataset contains only a small fraction of **100 (0.018%)** anomalous transactions. While the 100 anomalous entries encompass **70 (0.013%)** \"global\" anomalies and **30 (0.005%)** \"local\" anomalies as introduced in section 1.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove the \"ground-truth\" label information for the following steps of the lab\n",
    "label = ori_dataset.pop('label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Pre-Processing of Categorical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the initial data assessment above we can observe that the majority of attributes recorded in AIS- and ERP-systems correspond to categorical (discrete) attribute values, e.g. the posting date, the general-ledger account, the posting type, the currency. Let's have a more detailed look into the distribution of two dataset attributes, namely (1) the posting key `BSCHL` as well as (2) the general ledger account `HKONT`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare to plot posting key and general ledger account side by side\n",
    "fig, ax = plt.subplots(1,2)\n",
    "fig.set_figwidth(20)\n",
    "\n",
    "# plot the distribution of the posting key attribute\n",
    "g = sns.countplot(x=ori_dataset['BSCHL'], ax=ax[0])\n",
    "g.set_xticklabels(g.get_xticklabels(), rotation=90)\n",
    "g.set_title('Distribution of BSCHL attribute values')\n",
    "\n",
    "# plot the distribution of the general ledger account attribute\n",
    "g = sns.countplot(x=ori_dataset['HKONT'], ax=ax[1])\n",
    "g.set_xticklabels(g.get_xticklabels(), rotation=90)\n",
    "g.set_title('Distribution of HKONT attribute values')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, neural networks are in general not designed to be trained directly on categorical data and require the attributes to be trained on to be numeric. One simple way to meet this requirement is by applying a technique referred to as **\"one-hot\" encoding**. Using this encoding technique, we will derive a numerical representation of each of the categorical attribute values. One-hot encoding creates new binary columns for each categorical attribute value present in the original data. \n",
    "\n",
    "Let's work through a brief example: The **categorical attribute “Receiver”** below contains the names \"John\", \"Timur\" and \"Marco\". We \"one-hot\" encode the names by creating a separate binary column for each possible name value observable in the \"Receiver\" column. Now, we encode for each transaction that contains the value \"John\" in the \"Receiver\" column this observation with 1.0 in the newly created \"John\" column and 0.0 in all other created name columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 430px; height: auto\" src=\"images/encoding.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this technique will \"one-hot\" encode the 6 categorical attributes in the original transactional dataset. This can be achieved using the `get_dummies()` function available in the Pandas data science library:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select categorical attributes to be \"one-hot\" encoded\n",
    "categorical_attr_names = ['KTOSL', 'PRCTR', 'BSCHL', 'HKONT']\n",
    "\n",
    "# encode categorical attributes into a binary one-hot encoded representation \n",
    "ori_dataset_categ_transformed = pd.get_dummies(ori_dataset[categorical_attr_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's inspect the encoding of 10 sample transactions to see if we have been successfull."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect encoded sample transactions\n",
    "ori_dataset_categ_transformed.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Pre-Processing of Numerical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now inspect the distributions of the two numerical attributes contained in the transactional dataset namely, the (1) local currency amount `DMBTR` and the (2) document currency amount `WRBTR`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the log-scaled \"DMBTR\" as well as the \"WRBTR\" attribute value distribution\n",
    "fig, ax = plt.subplots(1,2)\n",
    "fig.set_figwidth(20)\n",
    "\n",
    "# plot distribution of the local amount attribute\n",
    "g = sns.distplot(ori_dataset['DMBTR'].tolist(), ax=ax[0])\n",
    "g.set_title('Distribution of DMBTR amount values')\n",
    "\n",
    "# plot distribution of the document amount attribute\n",
    "g = sns.distplot(ori_dataset['WRBTR'].tolist(), ax=ax[1])\n",
    "g.set_title('Distribution of WRBTR amount values')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, it can be observed, that for both attributes the distributions of amount values are heavy tailed. In order to approach faster a potential global minimum scaling and normalization of numerical input values is good a practice. Therefore, we first log-scale both variables and second min-max normalize the scaled amounts to the interval [0, 1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select \"DMBTR\" vs. \"WRBTR\" attribute\n",
    "numeric_attr_names = ['DMBTR', 'WRBTR']\n",
    "\n",
    "# add a small epsilon to eliminate zero values from data for log scaling\n",
    "numeric_attr = ori_dataset[numeric_attr_names] + 1e-7\n",
    "numeric_attr = numeric_attr.apply(np.log)\n",
    "\n",
    "# normalize all numeric attributes to the range [0,1]\n",
    "ori_dataset_numeric_attr = (numeric_attr - numeric_attr.min()) / (numeric_attr.max() - numeric_attr.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now visualize the log-scaled and min-max normalized distributions of both attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# append 'label' attribute for colour distinction\n",
    "numeric_attr_vis = ori_dataset_numeric_attr.copy()\n",
    "numeric_attr_vis['label'] = label\n",
    "\n",
    "# plot the log-scaled and min-max normalized numeric attributes\n",
    "g = sns.pairplot(data=numeric_attr_vis, vars=numeric_attr_names, hue='label')\n",
    "g.fig.suptitle('Distribution of DMBTR vs. WRBTR amount values')\n",
    "g.fig.set_size_inches(15, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, as anticipated the numeric attribute values of the \"global\" anomalies (green) fall outside the range of the regular amount distributions due to their unusual high amount values. In contrast, the numeric attribute values of the \"local\" anomalies (orange) are much more commingled within the regular transaction amounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Merge Categorical and Numerical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we merge both pre-processed numerical and categorical attributes into a single dataset that we will use for training our deep autoencoder neural network (explained an implemented in the following section 4.):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge categorical and numeric subsets\n",
    "ori_subset_transformed = pd.concat([ori_dataset_categ_transformed, ori_dataset_numeric_attr], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's again have a look at the dimensionality of the dataset after we applied the distinct pre-processing steps to the attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect final dimensions of pre-processed transactional data\n",
    "ori_subset_transformed.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the pre-processing steps above you may have noticed, that we didn't encode the attributes `WAERS` and `BUKRS` yet. This we left as an exercise for you:\n",
    "\n",
    ">1. Plot and inspect the distribution of the values of both attributes `WAERS` and `BUKRS`. [3 min]\n",
    ">2. Encode both variables using the `get_dummies()` method provided by the Pandas library. [5 min]\n",
    ">3. Merge your encoding results with the Pandas `ori_subset_transformed` data frame. [5 min]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, upon completion of all the pre-processing steps (incl. the exercises) we should end up with an encoded dataset consisting of a total number of 533,009 records (rows) and **618 encoded attributes** (columns). Let's keep the number number of columns in mind since it will define the dimensionality of the input- and output-layer of our deep autoencoder network which we will now implement in the following section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Autoencoder Neural Networks (AENNs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this section is to familiarize ourselves with the underlying idea and concepts of building a deep autoencoder neural network (AENN). We will cover the major building blocks and the specific network structure of AENNs as well as an exemplary implementation using the open source machine learning library PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Autoencoder Neural Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AENNs or \"Replicator Neural Networks\" are a variant of general feed-forward neural networks that have been initially introduced by Hinton and Salakhutdinov in [6]. AENNs usually comprise a **symmetrical network architecture** as well as a central hidden layer, referred to as **\"latent\"** or **\"coding\" layer**, of lower dimensionality. The design is chosen intentionally since the training objective of an AENN is to reconstruct its input in a \"self-supervised\" manner. \n",
    "\n",
    "**Figure 3** below illustrates a schematic view of an autoencoder neural network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 600px; height: auto\" src=\"images/autoencoder.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 3:** Schematic view of an autoencoder network comprised of two non-linear mappings (fully connected feed forward neural networks) referred to as encoder $f_\\theta: \\mathbb{R}^{dx} \\mapsto \\mathbb{R}^{dz}$ and decoder $g_\\theta: \\mathbb{R}^{dz} \\mapsto \\mathbb{R}^{dx}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, AENNs can be interpreted as \"lossy\" data **compression algorithms**. They are \"lossy\" in a sense that the reconstructed outputs will be degraded compared to the original inputs. The difference between the original input $x^i$ and its reconstruction $\\hat{x}^i$ is referred to as **reconstruction error**. In general, AENNs encompass three major building blocks:\n",
    "\n",
    "\n",
    ">   1. an encoding mapping function $f_\\theta$, \n",
    ">   2. a decoding mapping function $g_\\theta$, \n",
    ">   3. and a loss function $\\mathcal{L_{\\theta}}$.\n",
    "\n",
    "Most commonly the encoder and the decoder mapping functions consist of **several layers of neurons followed by a non-linear function** and shared parameters $\\theta$. The encoder mapping $f_\\theta(\\cdot)$ maps an input vector (e.g. an \"one-hot\" encoded transaction) $x^i$ to a compressed representation $z^i$ referred to as latent space $Z$. This hidden representation $z^i$ is then mapped back by the decoder $g_\\theta(\\cdot)$ to a re-constructed vector $\\hat{x}^i$ of the original input space (e.g. the re-constructed encoded transaction). Formally, the nonlinear mappings of the encoder- and the decoder-function can be defined by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>$f_\\theta(x^i) = s(Wx^i + b)$, and $g_\\theta(z^i) = s′(W′z^i + d)$,</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $s$ and $s′$ denote non-linear activations with model parameters $\\theta = \\{W, b, W', d\\}$, $W \\in \\mathbb{R}^{d_x \\times d_z}, W' \\in \\mathbb{R}^{d_z \\times d_y}$ are weight matrices and $b \\in \\mathbb{R}^{dx}$, $d \\in \\mathbb{R}^{dz}$ are offset bias vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Autoencoder Neural Network Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's start implementing an AENN by first implementing the encoder network using PyTorch. For the encoder, we aim to implement a network consisting of **nine fully-connected layers**. Furthermore, the encoder is specified by the following number of neurons per layer: \"618-256-128-64-32-16-8-4-3\". Meaning the first layer consists of 618 neurons (specified by the dimensionality of our input data), the second layer of 256 neurons and the subsequent layers of 128, 64, 32, 16, 8, 4 and 3 neurons respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some elements of the encoder network code below should be given particular attention:\n",
    "\n",
    ">- `self.encoder_Lx`: defines the linear transformation of the layer applied to the incoming input: $Wx + b$.\n",
    ">- `nn.init.xavier_uniform`: inits the layer weights using a uniform distribution according to [9]. \n",
    ">- `self.encoder_Rx`: defines the non-linear transformation of the layer: $\\sigma(\\cdot)$.\n",
    ">- `self.dropout`: randomly zeros some of the elements of the input tensor with probability $p$ according to [8].\n",
    "\n",
    "We use **\"Leaky ReLUs\"** as introduced by Xu et al. in [7] to avoid \"dying\" non-linearities and to speed up training convergence. Leaky ReLUs allow a small gradient even when a particular neuron is not active. In addition, we include the **\"drop-out\" probability**, as introduced by [8], which defines the probability rate for each neuron to be set to zero at a forward pass to prevent the network from overfitting. However we explore its effect on the model later in the exercise section of the lab. Initially, we set the dropout probability of each neuron to $p=0.0$ (0%), meaning that none of the neuron activiations will be set to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the encoder network\n",
    "class encoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(encoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 618, out 512\n",
    "        self.encoder_L1 = nn.Linear(in_features=ori_subset_transformed.shape[1], out_features=512, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.encoder_L1.weight) # init weights according to [9]\n",
    "        self.encoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "        # specify layer 2 - in 512, out 256\n",
    "        self.encoder_L2 = nn.Linear(512, 256, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L2.weight)\n",
    "        self.encoder_R2 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 3 - in 256, out 128\n",
    "        self.encoder_L3 = nn.Linear(256, 128, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L3.weight)\n",
    "        self.encoder_R3 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 4 - in 128, out 64\n",
    "        self.encoder_L4 = nn.Linear(128, 64, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L4.weight)\n",
    "        self.encoder_R4 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 5 - in 64, out 32\n",
    "        self.encoder_L5 = nn.Linear(64, 32, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L5.weight)\n",
    "        self.encoder_R5 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 6 - in 32, out 16\n",
    "        self.encoder_L6 = nn.Linear(32, 16, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L6.weight)\n",
    "        self.encoder_R6 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 7 - in 16, out 8\n",
    "        self.encoder_L7 = nn.Linear(16, 8, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L7.weight)\n",
    "        self.encoder_R7 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 8 - in 8, out 4\n",
    "        self.encoder_L8 = nn.Linear(8, 4, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L8.weight)\n",
    "        self.encoder_R8 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 9 - in 4, out 3\n",
    "        self.encoder_L9 = nn.Linear(4, 3, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L9.weight)\n",
    "        self.encoder_R9 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # init dropout layer with probability p\n",
    "        self.dropout = nn.Dropout(p=0.0, inplace=True)\n",
    "        \n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.encoder_R1(self.dropout(self.encoder_L1(x)))\n",
    "        x = self.encoder_R2(self.dropout(self.encoder_L2(x)))\n",
    "        x = self.encoder_R3(self.dropout(self.encoder_L3(x)))\n",
    "        x = self.encoder_R4(self.dropout(self.encoder_L4(x)))\n",
    "        x = self.encoder_R5(self.dropout(self.encoder_L5(x)))\n",
    "        x = self.encoder_R6(self.dropout(self.encoder_L6(x)))\n",
    "        x = self.encoder_R7(self.dropout(self.encoder_L7(x)))\n",
    "        x = self.encoder_R8(self.dropout(self.encoder_L8(x)))\n",
    "        x = self.encoder_R9(self.encoder_L9(x)) # don't apply dropout to the AE bottleneck\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to instantiate the encoder model to be trained on the CPU or to be trained on any of the available GPUs (if CUDNN is available and `USE_CUDA` is set to `True`) by execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init training network classes / architectures\n",
    "encoder_train = encoder()\n",
    "\n",
    "# push to cuda if cudnn is available\n",
    "if (torch.backends.cudnn.version() != None and USE_CUDA == True):\n",
    "    encoder_train = encoder().cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is initialized we can visualize the model structure and review the implemented network architecture by execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the initialized architectures\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] encoder architecture:\\n\\n{}\\n'.format(now, encoder_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks, great? Excellent!\n",
    "\n",
    "Let's, now as a next step, complete the AENN by implementing the corresponding decoder network. The decoder also consists of nine fully-connected layers. Futhermore, the decoder network is intended to **symmetrically mirror** the encoder networks architecture by a layer wise inversion \"3-4-8-16-32-64-128-256-618\" of the encoder network layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the decoder network\n",
    "class decoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(decoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 3, out 4\n",
    "        self.decoder_L1 = nn.Linear(in_features=3, out_features=4, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.decoder_L1.weight)  # init weights according to [9]\n",
    "        self.decoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "        # specify layer 2 - in 4, out 8\n",
    "        self.decoder_L2 = nn.Linear(4, 8, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L2.weight)\n",
    "        self.decoder_R2 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 3 - in 8, out 16\n",
    "        self.decoder_L3 = nn.Linear(8, 16, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L3.weight)\n",
    "        self.decoder_R3 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 4 - in 16, out 32\n",
    "        self.decoder_L4 = nn.Linear(16, 32, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L4.weight)\n",
    "        self.decoder_R4 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 5 - in 32, out 64\n",
    "        self.decoder_L5 = nn.Linear(32, 64, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L5.weight)\n",
    "        self.decoder_R5 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 6 - in 64, out 128\n",
    "        self.decoder_L6 = nn.Linear(64, 128, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L6.weight)\n",
    "        self.decoder_R6 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "        \n",
    "        # specify layer 7 - in 128, out 256\n",
    "        self.decoder_L7 = nn.Linear(128, 256, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L7.weight)\n",
    "        self.decoder_R7 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 8 - in 256, out 512\n",
    "        self.decoder_L8 = nn.Linear(256, 512, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L8.weight)\n",
    "        self.decoder_R8 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 9 - in 512, out 618\n",
    "        self.decoder_L9 = nn.Linear(in_features=512, out_features=ori_subset_transformed.shape[1], bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L9.weight)\n",
    "        self.decoder_R9 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # init dropout layer with probability p\n",
    "        self.dropout = nn.Dropout(p=0.0, inplace=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.decoder_R1(self.dropout(self.decoder_L1(x)))\n",
    "        x = self.decoder_R2(self.dropout(self.decoder_L2(x)))\n",
    "        x = self.decoder_R3(self.dropout(self.decoder_L3(x)))\n",
    "        x = self.decoder_R4(self.dropout(self.decoder_L4(x)))\n",
    "        x = self.decoder_R5(self.dropout(self.decoder_L5(x)))\n",
    "        x = self.decoder_R6(self.dropout(self.decoder_L6(x)))\n",
    "        x = self.decoder_R7(self.dropout(self.decoder_L7(x)))\n",
    "        x = self.decoder_R8(self.dropout(self.decoder_L8(x)))\n",
    "        x = self.decoder_R9(self.decoder_L9(x)) # don't apply dropout to the AE output\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also instantiate the encoder model for CPU or GPU training and convince ourselves that it was successfully initialized by printing and reviewing its architecture by execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init training network classes / architectures\n",
    "decoder_train = decoder()\n",
    "\n",
    "# push to cuda if cudnn is available\n",
    "if (torch.backends.cudnn.version() != None) and (USE_CUDA == True):\n",
    "    decoder_train = decoder().cuda()\n",
    "    \n",
    "# print the initialized architectures\n",
    "now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "print('[LOG {}] decoder architecture:\\n\\n{}\\n'.format(now, decoder_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like intended? Brilliant!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Autoencoder Neural Network Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have implemented the AENN we are ready to train the network. Prior to starting the training, we need to define an apropriate loss function. Remember, we aim to train our model to learn a set of encoder-decoder model parameters $\\theta$ that minimize the dissimilarity of a given financial transaction $x^{i}$ and its reconstruction $\\hat{x}^{i} = g_\\theta(f_\\theta(x^{i}))$ as faithfully as possible. \n",
    "\n",
    "Thereby, the training objective is to learn a set of optimal shared encoder-decoder model parameters $\\theta^*$ that optimize $\\arg\\min_{\\theta} \\|X - g_\\theta(f_\\theta(X))\\|$ over all transactions or more specifically journal entries $X$. To achieve this optimization objective, one typically minimizes a loss function $\\mathcal{L_{\\theta}}$ as part of the network training. In this lab we use the **binary-cross-entropy error (BCE)** loss, defined by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> $\\mathcal{L^{BCE}_{\\theta}}(x^{i};\\hat{x}^{i}) = \\frac{1}{n}\\sum_{i=1}^{n}\\sum_{j=1}^{k} x^{i}_{j} ln(\\hat{x}^{i}_{j}) + (1-x^{i}_{j}) ln(1-\\hat{x}^{i}_{j})$, </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for a set of $n$-journal entries $x^{i}$, $i=1,...,n$ and their respective reconstructions $\\hat{x}^{i}$ and all journal entry attributes $j=1,...,k$. During trainig the BCE loss will penalize models that result in a high dissimilarity between input journal entries and their respective reconstructions. \n",
    "\n",
    "Luckily, an implementation of the BCE loss is already available in PyTorch! It can be instantiated \"off-the-shelf\" via execution of the following PyTorch command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the optimization criterion / loss function\n",
    "loss_function = nn.BCEWithLogitsLoss(reduction='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note: Enabling the parameter `reduction` specifies that the losses are averaged over all observations for each minibatch).\n",
    "\n",
    "Based on the loss magnitude of a certain mini-batch PyTorch automatically computes the gradients. But even better, based on the gradient, the library also helps us in the optimization and update of the AENN parameters $\\theta$.\n",
    "\n",
    "We will use the **Adam optimization** as proposed in [11] and set the learning-rate $l = 0.001$. Each mini-batch step the optimizer will update the encoder- and decoder-parameters $\\theta$ values according to degree of reconstruction error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define learning rate and optimization strategy\n",
    "learning_rate = 1e-3\n",
    "encoder_optimizer = torch.optim.Adam(encoder_train.parameters(), lr=learning_rate)\n",
    "decoder_optimizer = torch.optim.Adam(decoder_train.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have successfully implemented and defined the three AENN building blocks let's take some time to review the `encoder` and `decoder` model definition as well as the `loss`. Please, read the above code and comments carefully and don't hesitate to let us know any questions you might have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training the Autoencoder Neural Network (AENN) Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will train our deep autoencoder neural network (as implemented in section 4. of the lab) using the encoded transactional data (created in section 3. of the lab). More specifically, we will have a detailed look into the distinct training steps as well as how to monitor the training progress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Preparing the Network Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have pre-processed the dataset, implemented the AENN and defined the reconstruction error. Let's now start to train a corresponding model for **5 epochs** and a **mini-batch size of 128** journal entries per batch. This implies that the whole dataset will be fed to the AENN 5 times in chunks of 128 journal entries yielding to 4,165 mini-batches (533,009 journal entries / 128 journal entries per mini-batch) per epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify training parameters\n",
    "num_epochs = 5\n",
    "mini_batch_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the training phase, we will fetch the individual mini-batches of the entire population of journal entries. To achieve this, we will use PyTorch's `DataLoader` that provides single- or multi-process iterators over a given dataset to load one mini-batch at a time. By enabling `shuffle=True` the data will be reshuffled at every epoch prior to feeding it to the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert pre-processed data to pytorch tensor\n",
    "torch_dataset = torch.from_numpy(ori_subset_transformed.values).float()\n",
    "\n",
    "# convert to pytorch tensor - none cuda enabled\n",
    "dataloader = DataLoader(torch_dataset, batch_size=mini_batch_size, shuffle=True, num_workers=0)\n",
    "# note: we set num_workers to zero to retrieve deterministic results\n",
    "\n",
    "# determine if CUDA is available at compute node\n",
    "if (torch.backends.cudnn.version() != None) and (USE_CUDA == True):\n",
    "    dataloader = DataLoader(torch_dataset.cuda(), batch_size=mini_batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Running the Network Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we start training the model. The training procedure of each mini-batch is performed as follows: \n",
    "\n",
    ">1. do a forward pass through the encoder-decoder part, \n",
    ">2. compute the binary-cross-entropy reconstruction loss $\\mathcal{L^{BCE}_{\\theta}}(x^{i};\\hat{x}^{i})$, \n",
    ">3. do a backward pass through the encoder-decoder part, and \n",
    ">4. update the parameters of the encoder $f_\\theta(\\cdot)$ and decoder $g_\\theta(\\cdot)$ networks.\n",
    "\n",
    "To ensure learning while training our AENN model we will monitor whether the loss decreases with progressing training. Therefore, we obtain and evaluate the reconstruction performance of the entire dataset after each training epoch. Based on this evaluation we can conclude on the training progress and whether the loss is converging (indicating that the model might not improve any further).\n",
    "\n",
    "The following elements of the network training code below should be given particular attention:\n",
    " \n",
    ">- `reconstruction_loss.backward()` computes the gradients based on the magnitude of the reconstruction loss,\n",
    ">- `encoder_optimizer.step()` and `decoder_optimizer.step()` updates the network parameters based on the gradient.\n",
    "\n",
    "Please also note, that the mini-batch training of the AENN will be executed on the GPU (if CUDNN is available and set accordingly by USE_CUDA=True). However, the evaluation of the reconstruction performance over the entire set of journal entries will be performed at CPU level. Using PyTorch this can be easily achieved with the following commands:\n",
    "\n",
    ">- `encoder_train.cuda()`: moves all model parameters and buffers to the GPU.\n",
    ">- `encoder_train.cpu()`: moves all model parameters and buffers to the CPU.\n",
    "\n",
    "The reason for the switch to the CPU in the evaluation phase is the size of the entire dataset. We aim to compute the reconstruction error over the entire dataset, which in most of the cases does not fit into the GPU's memory.\n",
    "\n",
    "In addition, after each training epoch we want to save a checkpoint for both the actual `encoder` and `decoder` model. The saved model checkpoints contain a snapshot of the trained model parameter values upon completion of a training epoch. In general, it is good practice, to save checkpoints at regular intervals during training. In case your system crashes during training you are able continue from the last checkpoint rather than start over from scratch.\n",
    "\n",
    ">- `torch.save()`: saves a checkpoint of the actual encoder and decoder model parameter values to disc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init collection of epoch losses\n",
    "epoch_losses = []\n",
    "\n",
    "# convert encoded transactional data to torch Variable\n",
    "data = autograd.Variable(torch_dataset)\n",
    "\n",
    "# train autoencoder model\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    # init collection of epoch losses\n",
    "    mini_batch_losses = []\n",
    "    \n",
    "    # init mini batch counter\n",
    "    mini_batch_count = 0\n",
    "    \n",
    "    # determine if CUDA is available at compute node\n",
    "    if(torch.backends.cudnn.version() != None) and (USE_CUDA == True):\n",
    "        \n",
    "        # set networks / models in GPU mode\n",
    "        encoder_train.cuda()\n",
    "        decoder_train.cuda()\n",
    "\n",
    "    # set networks in training mode (apply dropout when needed)\n",
    "    encoder_train.train()\n",
    "    decoder_train.train()\n",
    "\n",
    "    # start timer\n",
    "    start_time = datetime.now()\n",
    "        \n",
    "    # iterate over all mini-batches\n",
    "    for mini_batch_data in dataloader:\n",
    "\n",
    "        # increase mini batch counter\n",
    "        mini_batch_count += 1\n",
    "\n",
    "        # convert mini batch to torch variable\n",
    "        mini_batch_torch = autograd.Variable(mini_batch_data)\n",
    "\n",
    "        # =================== (1) forward pass ===================================\n",
    "\n",
    "        # run forward pass\n",
    "        z_representation = encoder_train(mini_batch_torch) # encode mini-batch data\n",
    "        mini_batch_reconstruction = decoder_train(z_representation) # decode mini-batch data\n",
    "        \n",
    "        # =================== (2) compute reconstruction loss ====================\n",
    "\n",
    "        # determine reconstruction loss\n",
    "        reconstruction_loss = loss_function(mini_batch_reconstruction, mini_batch_torch)\n",
    "        \n",
    "        # =================== (3) backward pass ==================================\n",
    "\n",
    "        # reset graph gradients\n",
    "        decoder_optimizer.zero_grad()\n",
    "        encoder_optimizer.zero_grad()\n",
    "\n",
    "        # run backward pass\n",
    "        reconstruction_loss.backward()\n",
    "        \n",
    "        # =================== (4) update model parameters ========================\n",
    "\n",
    "        # update network parameters\n",
    "        decoder_optimizer.step()\n",
    "        encoder_optimizer.step()\n",
    "\n",
    "        # =================== monitor training progress ==========================\n",
    "\n",
    "        # print training progress each 1'000 mini-batches\n",
    "        if mini_batch_count % 1000 == 0:\n",
    "            \n",
    "            # print the training mode: either on GPU or CPU\n",
    "            mode = 'GPU' if (torch.backends.cudnn.version() != None) and (USE_CUDA == True) else 'CPU'\n",
    "            \n",
    "            # print mini batch reconstuction results\n",
    "            now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "            end_time = datetime.now() - start_time\n",
    "            print('[LOG {}] training status, epoch: [{:04}/{:04}], batch: {:04}, loss: {}, mode: {}, time required: {}'.format(now, (epoch+1), num_epochs, mini_batch_count, np.round(reconstruction_loss.item(), 4), mode, end_time))\n",
    "\n",
    "            # reset timer\n",
    "            start_time = datetime.now()\n",
    "            \n",
    "        # collect mini-batch loss\n",
    "        mini_batch_losses.extend([np.round(reconstruction_loss.item(), 4)])\n",
    "\n",
    "    # =================== evaluate model performance =============================\n",
    "                                 \n",
    "    # collect mean training epoch loss\n",
    "    epoch_losses.extend([np.mean(mini_batch_losses)])\n",
    "    \n",
    "    # print training epoch results\n",
    "    now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "    print('[LOG {}] training status, epoch: [{:04}/{:04}], loss: {:.10f}'.format(now, (epoch+1), num_epochs, np.mean(mini_batch_losses)))\n",
    "\n",
    "    # =================== save model snapshot to disk ============================\n",
    "    \n",
    "    # save trained encoder model file to disk\n",
    "    encoder_model_name = \"ep_{}_encoder_model.pth\".format((epoch+1))\n",
    "    torch.save(encoder_train.state_dict(), encoder_model_name)\n",
    "\n",
    "    # save trained decoder model file to disk\n",
    "    decoder_model_name = \"ep_{}_decoder_model.pth\".format((epoch+1))\n",
    "    torch.save(decoder_train.state_dict(), decoder_model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now evaluate if the loss function is indeed going down with progressing training of the model. Therefore, let's visualize the magnitudes of the losses obtained per training epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the training progress\n",
    "plt.plot(range(0, len(epoch_losses)), epoch_losses)\n",
    "plt.xlabel('[training epoch]')\n",
    "plt.xlim([0, len(epoch_losses)])\n",
    "plt.ylabel('[reconstruction-error]')\n",
    "#plt.ylim([0.0, 1.0])\n",
    "plt.title('AENN training performance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the reconstruction loss change as we progress in training our model? After 5 epochs, we can observe that our reconstruction loss already went down significantly and starts to converge nicely. This indicates that our network did a pretty good job in learning the structure and attributes of the journal entries.\n",
    "\n",
    "But, from the plot we also observe that the model could probably be trained a couple more epochs as the trend of the reconstruction error still decreases for the last few epochs. In order to save time, we will continue the lab using a pre-trained model already trained by 20 epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we continue with the next notebook section it's time for some exercises:\n",
    "\n",
    ">1. Set the `USE_CUDA` flag to `False` and re-start the training procedure. What impact do you observe in terms of training time needed for training a single epoch? Please, note that after altering the `USE_CUDA` flag you need to execute all successive cells starting from section 4.2. [5-10 min]\n",
    ">2. Set the `dropout` probability to `0.8` (80%) and re-start the training procedure. What impact do you observe in terms of training performance / reconstruction loss? Please, note that after altering the `dropout` probability you need to execute all successive cells starting from section 4.2. [5-10 min]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluating the Autoencoder Neural Network (AENN) Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to train our autoencoder model, we will explore how we can use it to detect anomalies within the entire population of journal entries. Initially, we will start by loading a pre-trained model of 10 epochs and assess its reconstruction capability on the entire dataset. \n",
    "\n",
    "The pre-trained model is stored in the same directory as the lab notebook and can be loaded by executing the cell below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# restore pretrained model checkpoint\n",
    "encoder_model_name = 'https://raw.githubusercontent.com/GitiHubi/deepAI/master/models/ep_10_encoder_model.pth'\n",
    "decoder_model_name = 'https://raw.githubusercontent.com/GitiHubi/deepAI/master/models/ep_10_decoder_model.pth'\n",
    "\n",
    "# Read stored model from the remote location\n",
    "encoder_bytes = urllib.request.urlopen(encoder_model_name)\n",
    "decoder_bytes = urllib.request.urlopen(decoder_model_name)\n",
    "\n",
    "# Load tensor from io.BytesIO object\n",
    "encoder_buffer = io.BytesIO(encoder_bytes.read())\n",
    "decoder_buffer = io.BytesIO(decoder_bytes.read())\n",
    "\n",
    "# init training network classes / architectures\n",
    "encoder_eval = encoder()\n",
    "decoder_eval = decoder()\n",
    "\n",
    "# load trained models\n",
    "encoder_eval.load_state_dict(torch.load(encoder_buffer))\n",
    "decoder_eval.load_state_dict(torch.load(decoder_buffer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Assessment of the Pre-Trained Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once obtained, let's use the model to reconstruct the entire dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert encoded transactional data to torch Variable\n",
    "data = autograd.Variable(torch_dataset)\n",
    "\n",
    "# set networks in evaluation mode (don't apply dropout)\n",
    "encoder_eval.eval()\n",
    "decoder_eval.eval()\n",
    "\n",
    "# reconstruct encoded transactional data\n",
    "reconstruction = decoder_eval(encoder_eval(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, nice. Comparing the overall reconstruction loss of the pre-trained model trained for **10 epochs** to the one we initially trained for **5 epochs** reveals, that the pre-trained model results in a significantly lower reconstruction error. We can therefore conclude that the pre-trained model outperforms our initial model in capturing the inherent characteristics of the journal entries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Assessment of the Individual Transactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now that we are convinced that the pre-trained model is of decent quality let's assess the individual journal entries of the dataset. To achieve this, we collect the reconstruction errors of each individual journal entry by executing the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init binary cross entropy errors\n",
    "reconstruction_loss_transaction = np.zeros(reconstruction.size()[0])\n",
    "\n",
    "# iterate over all detailed reconstructions\n",
    "for i in range(0, reconstruction.size()[0]):\n",
    "\n",
    "    # determine reconstruction loss - individual transactions\n",
    "    reconstruction_loss_transaction[i] = loss_function(reconstruction[i], data[i]).item()\n",
    "\n",
    "    if(i % 100000 == 0):\n",
    "\n",
    "        ### print conversion summary\n",
    "        now = datetime.utcnow().strftime(\"%Y%m%d-%H:%M:%S\")\n",
    "        print('[LOG {}] collected individual reconstruction loss of: {:06}/{:06} transactions'.format(now, i, reconstruction.size()[0]))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have collected individual reconstruction errors let's visualize them accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# assign unique id to transactions\n",
    "plot_data = np.column_stack((np.arange(len(reconstruction_loss_transaction)), reconstruction_loss_transaction))\n",
    "\n",
    "# obtain regular transactions as well as global and local anomalies\n",
    "regular_data = plot_data[label == 'regular']\n",
    "global_outliers = plot_data[label == 'global']\n",
    "local_outliers = plot_data[label == 'local']\n",
    "\n",
    "# plot reconstruction error scatter plot\n",
    "ax.scatter(regular_data[:, 0], regular_data[:, 1], c='C0', alpha=0.4, marker=\"o\", label='regular') # plot regular transactions\n",
    "ax.scatter(global_outliers[:, 0], global_outliers[:, 1], c='C1', marker=\"^\", label='global') # plot global outliers\n",
    "ax.scatter(local_outliers[:, 0], local_outliers[:, 1], c='C2', marker=\"^\", label='local') # plot local outliers\n",
    "\n",
    "# add plot legend of transaction classes\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization reveals that the pre-trained model is able to reconstruct the majority of regular journal entries, while failing to do so, for the anomalous ones. As a result, the model reconstruction error can be used to distinguish both \"global\" anomalies (orange) and \"local\" anomalies (green) from the regular journal entries (blue).\n",
    "\n",
    "To further investigate our observation and confirm the initial assumption, let's have a closer look into the journal entries exhibiting a \"high\" binary cross-entropy reconstruction error >= 0.1. We assume that these journal entries correspond to the \"global\" anomalies of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# append labels to original dataset\n",
    "ori_dataset['label'] = label\n",
    "\n",
    "# inspect transactions exhibiting a reconstruction error >= 0.1\n",
    "ori_dataset[reconstruction_loss_transaction >= 0.1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now also have a closer look into the journal entries exhibiting a \"medium\" binary cross-entropy reconstruction error >= 0.02 and < 0.1. We assume that these journal entries mostly correspond to the \"local\" anomalies of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect transactions exhibiting a reconstruction error < 0.1 and >= 0.05\n",
    "ori_dataset[(reconstruction_loss_transaction >= 0.05) & (reconstruction_loss_transaction < 0.1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Optional Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please read ahead and only come back to these optional exercises if time permits.\n",
    "\n",
    "**1. Train the autoencoder model from scratch** [15 mins]\n",
    "\n",
    "First, change the number of training epochs `num_epochs` to **30** in the corresponding cell of section 5.1. Second, you might also want to try different learning rates other than the initial learning rate of **0.001** in the corresponding cell of section 4.3. Third, comment out the two lines in the first cell of section 6. where the pre-trained model is defined (under \"restore pre-trained model checkpoint\") as well as two lines where it is loaded (under \"load trained models\"). \n",
    "\n",
    "Please, note that after altering the number of training epochs in section 5.1 and/or the learning rate in section 4.3 you need to execute all successive cells starting from section 5.1 or 4.3.\n",
    "\n",
    "**2. What would happen if we remove a few fully-connected layers?** [15 mins]\n",
    "\n",
    "We designed a specific model for the lab because experiments show that the structure provided result in a good detection accuracy. Let's see how the reconstruction performance change if we would **remove several of the hidden layers**. First, adjust the encoder and decoder model definitions in section 4.2 accordingly (you may want to use the code snippets shown below). Then, follow all the instructions for training from scratch.\n",
    "\n",
    "Please, note that after altering the encoder and / or decoder network architecture in section 4.2 you need to execute all successive cells starting from section 4.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the shallow encoder network \n",
    "# containing only a single layer\n",
    "class shallow_encoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(encoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 618, out 3\n",
    "        self.encoder_L1 = nn.Linear(in_features=ori_subset_transformed.shape[1], out_features=3, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.encoder_L1.weight) # init weights according to [9]\n",
    "        self.encoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "        \n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.encoder_R1(self.encoder_L1(x)) # don't apply dropout to the AE bottleneck\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the shallow decoder network \n",
    "# containing only a single layer\n",
    "class shallow_decoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(decoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 3, out 618\n",
    "        self.decoder_L1 = nn.Linear(in_features=3, out_features=ori_subset_transformed.shape[1], bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.decoder_L1.weight)  # init weights according to [9]\n",
    "        self.decoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.decoder_R1(self.decoder_L1(x)) # don't apply dropout to the AE output\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Explore the latent space representation** [15 min]\n",
    "\n",
    "In a real world scenario it is usually beneficial to visualize the data manifolds to:\n",
    "\n",
    "> 1. determine if the autoencoder learned a meaningful representation; and,\n",
    "> 2. obtain an impression of the structural characteristics and dependencies in the data.\n",
    "\n",
    "To achieve this, we will propagate the data through the trained model and capture for each transaction the respective representation in the latent space as referred to as \"embeddings\" (the activation pattern of each journal entry at the bottleneck neurons). \n",
    "\n",
    "Therefore, we intentionally chose a bottleneck layer comprised of 3 neurons. This provides us the ability to visualize the distinctive activation pattern using the 3d plotting capabilities of Pythons matplotlib library. In the subsequent cells we already prepared a plotting function that draws a 3d scatter plot of the latent space representation of each transaction at a particular epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot latent space representation of all samples given latent variable and label class\n",
    "def plot_latent_space(latent_variable, label, epoch):\n",
    "    \"\"\" Plots latent space activations as a 3d scatter plot at particular epoch\n",
    "    :param latent_space: activations of latent space\n",
    "    :param label: 1-d array of labels defining type of anomaly\n",
    "    :param epoch: training epoch\n",
    "    \"\"\"\n",
    "    # prepare plot\n",
    "    fig = plt.figure(figsize=(10,8))\n",
    "    ax = fig.add_subplot(111, projection='3d')\n",
    "    ax.view_init(30, 45)\n",
    "\n",
    "    # set axis paramaters of subplot\n",
    "    ax.grid(linestyle='dotted')\n",
    "    \n",
    "    # set label and title details\n",
    "    ax.set_xlabel(r'activation [$z_1$]', weight='normal', fontsize=12)\n",
    "    ax.set_ylabel(r'activation [$z_2$]', weight='normal', fontsize=12)\n",
    "    ax.set_zlabel(r'activation [$z_3$]', weight='normal', fontsize=12)\n",
    "    plt.title('latent space activations at epoch ' + str(epoch), fontsize=12)\n",
    "\n",
    "    # plot regular transactions\n",
    "    regular = latent_variable[np.where(label == 'regular')]\n",
    "    ax.scatter(regular[:, 0], regular[:, 1], regular[:, 2], c='C0', alpha=0.4, marker=\"o\")\n",
    "\n",
    "    # plot first order anomalous transactions\n",
    "    anomalies_1 = latent_variable[np.where(label == 'global')]\n",
    "    ax.scatter(anomalies_1[:, 0], anomalies_1[:, 1], anomalies_1[:, 2], c='C1', s=100, marker=\"^\")\n",
    "\n",
    "    # plot second order anomalous transactions\n",
    "    anomalies_2 = latent_variable[np.where(label == 'local')]\n",
    "    ax.scatter(anomalies_2[:, 0], anomalies_2[:, 1], anomalies_2[:, 2], c='C2', s=100, marker=\"^\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, before plotting the embeddings we load the model of the shallow autoencoder network you have been asked to train in exercise 7.2 of the optional exercises. Since we are interested on the latent space representation of each transaction we only need to load the pre-trained encoder part of the autoencoder (ignoring the decoder part) and propagate all transactions through it.\n",
    "\n",
    "The function below will accomplish these steps for a given pre-trained encoder model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract latent space representation of all samples given the name of encoder model to be loaded\n",
    "def get_latent_space(encoder_model_name):\n",
    "    ''' Extracts the latent space representation\n",
    "    :param encoder_model_name: file name of the pretrained encoder model\n",
    "    :return: latent space representation\n",
    "    '''\n",
    "    # init training network classes / architectures\n",
    "    encoder_eval = encoder()\n",
    "\n",
    "    # load trained models\n",
    "    encoder_eval.load_state_dict(torch.load(encoder_model_name))\n",
    "\n",
    "    # convert encoded transactional data to torch Variable\n",
    "    data = autograd.Variable(torch_dataset)\n",
    "\n",
    "    # set networks in training mode (don't apply dropout)\n",
    "    encoder_eval.eval()\n",
    "\n",
    "    # extract encoded latent space representation\n",
    "    latent_variable = encoder_eval(data).data.numpy()\n",
    "    \n",
    "    return latent_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check how the latent space representation \"embeddings\" of the individual transactions changes with progressing training epochs. In order to do so, you can load any checkpoint of a particular training epoch and visualize the latent representations you want to have a look at using the code snipped below. \n",
    "\n",
    "If you didn't find the time to accomplish exercise 7.2 you may want to plot the embeddings using the pre-trained model we prepared for this lab as stated in the cell below. This will also give you an idea of the transactional manifolds learned by the autoencoder network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine epoch of model to be evaluated\n",
    "eval_epoch = 5\n",
    "\n",
    "# collect latent space representation at a certain epochs \n",
    "z_representation = get_latent_space(encoder_model_name='ep_{}_encoder_model.pth'.format(eval_epoch))\n",
    "\n",
    "# plot the latent space at a particular epoch\n",
    "plot_latent_space(z_representation, label, epoch=eval_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you may want to save the content of the lab to your local machine. Therefore, please execute the cell below to retrieve an archived version of your current lab content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -czf GTC_2018_Lab.tar.gz *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may also want to execute the content of your lab outside of the jupyter notebook environment e.g. on compute node or server. The cell below converts the lab notebook into a standalone and executable python script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter nbconvert --to script GTC_2018_Lab.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Lab Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we presented a step by step implementation of an autoencoder deep neural network based methodology to detect anomalies in financial data. The degree of a financial transaction \"abnormity\" is evaluated based on its respective reconstruction error. The code provided in this lab can be tailored to meet more complex fraud detection scenarios and datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Post-Lab Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend you to try the following exercises after the lab:\n",
    "\n",
    "**1. Evaluation of shallow and deep autoencoder models** \n",
    "\n",
    "Try to train and evaluate further (shallow and deeper) autoencoder models (by removing and adding of fully-connected layers). Analyse the performance in terms of training time and reconstruction error.\n",
    "\n",
    "**2. Comparison to other dimensionality reduction techniques**\n",
    "\n",
    "Try using other dimensionality reduction techniques such as principal component analysis, non-negative matrix factorization or sparse coding and compare the detected anomalies with the ones detected by the autoencoder.\n",
    "\n",
    "**3. Review of additional autoencoder concepts**\n",
    "\n",
    "Try using other autoencoder architectures such as variational [13] or adversarial [14] autoencoder and compare the results with the autoencoder architecture implemented above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Major elements of the lab content are inspired by the publication \"Detection of Anomalies in Large Scale Accounting Data using Deep Autoencoder Networks\", of M. Schreyer, T. Sattarov, D. S. Borth, A. Dengel, and B. Reimer, 2017 (arXiv preprint available under: https://arxiv.org/abs/1709.05254).\n",
    "\n",
    "[1] ACFE, \"Report to the Nations on Occupational Fraud and Abuse\", The 2016 Global Fraud Study, Association of Certified Fraud Examiners (ACFE), 2016.\n",
    "\n",
    "[2] J. T. Wells, \"Corporate Fraud Handbook: Prevention and Detection\", John Wiley & Sons, 2017.\n",
    "\n",
    "[3] PwC, \"Pulling Fraud Out of the Shadows\", The Global Economic Crime and Fraud Survey 2018, PricewaterhouseCoopers LLP, 2018.\n",
    "\n",
    "[4] S. Markovitch, P. Willmott, \"Accelerating the digitization of business processes\", McKinsey & Company (2014) 1–5.\n",
    "\n",
    "[5] SAP, SAP Global Corporate Affairs, Corporate Factsheet 2017, 2017.\n",
    "\n",
    "[6] E. A. Lopez-Rojas , A. Elmir, and S. Axelsson, \"PaySim: A financial mobile money simulator for fraud detection\", In: The 28th European Modeling and Simulation Symposium-EMSS, Larnaca, Cyprus, 2016.\n",
    "\n",
    "[7] G. E. Hinton, and R. R. Salakhutdinov, \"Reducing the dimensionality of data with neural networks\", science 313, no. 5786: 504-507, 2006.\n",
    "\n",
    "[8] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, \"Dropout: A simple way to prevent neural networks from overfitting\", The Journal of Machine Learning Research, 15(1), 1929-1958, 2014.\n",
    "\n",
    "[9] X. Glorot and Y. Bengio, \"Understanding the difficulty of training deep feedforward neural networks\", Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 9:249–256, 2010.\n",
    "\n",
    "[10] B. Xu, N. Wang, T. Chen, and M. Li, \"Empirical Evaluation of Rectified Activations in Convolution Network\", ICML Deep Learning Workshop, pages 1–5, 2015.\n",
    "\n",
    "[11] D. P. Kingma and J. Ba, \"Adam: A method for stochastic optimization\", International Conference on Learning Representations (ICLR). 2015.\n",
    "\n",
    "[12] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. R. Salakhutdinov. \"Improving neural networks by preventing co-adaptation of feature detectors\", Technical Report, 2012.\n",
    "\n",
    "[13] D. P. Kingma, M. Welling. \"Auto-encoding variational bayes\", arXiv preprint arXiv:1312.6114, 2013.\n",
    "\n",
    "[14] Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B., \"Adversarial autoencoders\", arXiv preprint arXiv:1511.05644, 2015.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}