RNN_forecasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent Neural Network forecasting intro\n",
    "\n",
    "- Reference:`keras.layers.SimpleRNN`\n",
    "\n",
    "**Basic Architecture diagram**  \n",
    "\n",
    "<img src=\"./pic/RNN.png\" alt=\"RNN Architecture\" width=\"500\"/>\n",
    "\n",
    "- U,V,W are weight matrix.\n",
    "- Input: vector $X_t$ is input for network at time step t.  \n",
    "- Hidden state: vector $h(t)=\\tanh \\left(W h(t-1)+U_{x}(t)\\right)$\n",
    "- Output: $y_t$ is the output for the network at time step t. $y_t=\\operatorname{softmax}(V s(t))$\n",
    "\n",
    "Every neuron is assigned to a fixed step. The output of the hidden layer of one time step is part of the input of next time step.\n",
    "\n",
    "- The algorithm is to find the optimal weight matrix U,V,W that gives the best prediction or minimizes the loss function $J$. \n",
    "$$J(\\theta)=\\frac{1}{m} \\sum_{i=1}^{m} \\sum_{t=1}^{N_{i}} D(y_t, Y_t)$$\n",
    "- Forward/Backward propagation\n",
    "\n",
    "## Vanishing gradiant problems \n",
    "\n",
    "- LSTM `keras.layers.LSTM`, first proposed in\n",
    "[Hochreiter & Schmidhuber, 1997](https://www.bioinf.jku.at/publications/older/2604.pdf).\n",
    "- GRU `keras.layers.GRU`, first proposed in\n",
    "[Cho et al., 2014](https://arxiv.org/abs/1406.1078)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN-LSTM Crypto-forcasting \n",
    "- Thanks to [jeffheaton](https://github.com/jeffheaton/t81_558_deep_learning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.6.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Asset_ID</th>\n",
       "      <th>Count</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>VWAP</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1886282</th>\n",
       "      <td>1</td>\n",
       "      <td>1009.0</td>\n",
       "      <td>38463.446686</td>\n",
       "      <td>38484.00</td>\n",
       "      <td>38405.18</td>\n",
       "      <td>38428.484286</td>\n",
       "      <td>34.940985</td>\n",
       "      <td>38451.705205</td>\n",
       "      <td>0.000094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886283</th>\n",
       "      <td>1</td>\n",
       "      <td>935.0</td>\n",
       "      <td>38423.084753</td>\n",
       "      <td>38467.72</td>\n",
       "      <td>38408.31</td>\n",
       "      <td>38453.424286</td>\n",
       "      <td>23.979506</td>\n",
       "      <td>38434.534545</td>\n",
       "      <td>-0.000699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886284</th>\n",
       "      <td>1</td>\n",
       "      <td>946.0</td>\n",
       "      <td>38458.731429</td>\n",
       "      <td>38500.40</td>\n",
       "      <td>38447.64</td>\n",
       "      <td>38484.997143</td>\n",
       "      <td>38.255848</td>\n",
       "      <td>38479.665203</td>\n",
       "      <td>-0.000319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886285</th>\n",
       "      <td>1</td>\n",
       "      <td>699.0</td>\n",
       "      <td>38482.981429</td>\n",
       "      <td>38497.00</td>\n",
       "      <td>38439.62</td>\n",
       "      <td>38463.950000</td>\n",
       "      <td>24.634812</td>\n",
       "      <td>38467.956129</td>\n",
       "      <td>0.000082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886286</th>\n",
       "      <td>1</td>\n",
       "      <td>671.0</td>\n",
       "      <td>38457.484286</td>\n",
       "      <td>38503.00</td>\n",
       "      <td>38439.63</td>\n",
       "      <td>38491.732857</td>\n",
       "      <td>19.218220</td>\n",
       "      <td>38470.259211</td>\n",
       "      <td>-0.000140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Asset_ID   Count          Open      High       Low         Close  \\\n",
       "1886282         1  1009.0  38463.446686  38484.00  38405.18  38428.484286   \n",
       "1886283         1   935.0  38423.084753  38467.72  38408.31  38453.424286   \n",
       "1886284         1   946.0  38458.731429  38500.40  38447.64  38484.997143   \n",
       "1886285         1   699.0  38482.981429  38497.00  38439.62  38463.950000   \n",
       "1886286         1   671.0  38457.484286  38503.00  38439.63  38491.732857   \n",
       "\n",
       "            Volume          VWAP    Target  \n",
       "1886282  34.940985  38451.705205  0.000094  \n",
       "1886283  23.979506  38434.534545 -0.000699  \n",
       "1886284  38.255848  38479.665203 -0.000319  \n",
       "1886285  24.634812  38467.956129  0.000082  \n",
       "1886286  19.218220  38470.259211 -0.000140  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import IPython\n",
    "import IPython.display\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import tensorflow as tf\n",
    "from sklearn import metrics\n",
    "mpl.rcParams['figure.figsize'] = (8, 6)\n",
    "mpl.rcParams['axes.grid'] = False\n",
    "print(tf.__version__)\n",
    "\n",
    "##data and date\n",
    "csv_path = './data/codetest/btc.csv'\n",
    "df = pd.read_csv(csv_path)\n",
    "df = df[-70000:]\n",
    "date_time = pd.to_datetime(df.pop('timestamp'),unit='s')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the data\n",
    "\n",
    "split for the training(70%), validation(20%), and test sets(10%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from script.RNN.data_split import mysplit\n",
    "train_df, val_df, test_df = mysplit(df= df, features =['Count','Open','High','Low','Close','Volume','VWAP']\n",
    "                                    ,train_size=0.7,val_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize the data\n",
    "scale features before training a neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_mean = train_df.mean()\n",
    "train_std = train_df.std()\n",
    "\n",
    "train_df = (train_df - train_mean) / train_std #scipy.stats.zscore(train_df,ddof=1)\n",
    "val_df = (val_df - train_mean) / train_std\n",
    "test_df = (test_df - train_mean) / train_std\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Count</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>VWAP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1886282</th>\n",
       "      <td>-0.567846</td>\n",
       "      <td>-2.624313</td>\n",
       "      <td>-2.632213</td>\n",
       "      <td>-2.627502</td>\n",
       "      <td>-2.635777</td>\n",
       "      <td>-0.362491</td>\n",
       "      <td>-2.627832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886283</th>\n",
       "      <td>-0.613876</td>\n",
       "      <td>-2.637358</td>\n",
       "      <td>-2.637476</td>\n",
       "      <td>-2.626491</td>\n",
       "      <td>-2.627716</td>\n",
       "      <td>-0.484490</td>\n",
       "      <td>-2.633382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886284</th>\n",
       "      <td>-0.607034</td>\n",
       "      <td>-2.625837</td>\n",
       "      <td>-2.626912</td>\n",
       "      <td>-2.613786</td>\n",
       "      <td>-2.617511</td>\n",
       "      <td>-0.325597</td>\n",
       "      <td>-2.618796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886285</th>\n",
       "      <td>-0.760674</td>\n",
       "      <td>-2.618000</td>\n",
       "      <td>-2.628011</td>\n",
       "      <td>-2.616377</td>\n",
       "      <td>-2.624314</td>\n",
       "      <td>-0.477196</td>\n",
       "      <td>-2.622580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886286</th>\n",
       "      <td>-0.778090</td>\n",
       "      <td>-2.626240</td>\n",
       "      <td>-2.626071</td>\n",
       "      <td>-2.616373</td>\n",
       "      <td>-2.615334</td>\n",
       "      <td>-0.537482</td>\n",
       "      <td>-2.621836</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Count      Open      High       Low     Close    Volume      VWAP\n",
       "1886282 -0.567846 -2.624313 -2.632213 -2.627502 -2.635777 -0.362491 -2.627832\n",
       "1886283 -0.613876 -2.637358 -2.637476 -2.626491 -2.627716 -0.484490 -2.633382\n",
       "1886284 -0.607034 -2.625837 -2.626912 -2.613786 -2.617511 -0.325597 -2.618796\n",
       "1886285 -0.760674 -2.618000 -2.628011 -2.616377 -2.624314 -0.477196 -2.622580\n",
       "1886286 -0.778090 -2.626240 -2.626071 -2.616373 -2.615334 -0.537482 -2.621836"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data for LSTM: windowing\n",
    "\n",
    "The models make a set of predictions based on a window of consecutive samples from the data.\n",
    "- The `width` (number of time steps) of the `input` and `label` windows.\n",
    "- The `shift=offset` between them.\n",
    "- Which `features` are used as `inputs`, `labels`, or both.\n",
    "\n",
    "Example: to make a single prediction 24 hours into the future, given 24 hours of history, you might define a window like this:\n",
    "<img src=\"./pic/raw_window_24h.png\" alt=\"window24_24\" width=\"500\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data used for training the model is formated as `tf.data.Dataset` which is splited into inputs and labels. Similarly for valuation, test and example data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Long Short-Term Memory  \n",
    "<img src=\"./pic/LSTM-1.png\" alt=\"LSTM Architecture\" width=\"500\"/> \n",
    "\n",
    "- [https://keras.io](https://keras.io)\n",
    "- ` tf.keras.layers.LSTM`\n",
    "\n",
    "\n",
    "### Model Design\n",
    "- `LSTM(units = 20)`: LSTM layer with 20 internal units.\n",
    "- `Dense(units=2)`: Dense layer with 2 units.\n",
    "    dense models handled each time step independently.\n",
    "- Choose units by cross validation.\n",
    "\n",
    "Graph below demonstrates what layer and unit (or neuron) are, and the rightmost image shows the internal structure of a single LSTM unit.\n",
    "<img src=\"./pic/lstm-units.png\" alt=\"lstmunits\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#model design and fit\n",
    "from script.RNN.myModel import mymodel, myfittedmodel\n",
    "mylstm = mymodel()\n",
    "wide_window,lstm_model = myfittedmodel(model = mylstm[0], input_width=30, label_width=30, shift=1,\n",
    "                                                train_df=train_df, val_df=val_df, test_df=test_df,\n",
    "                                                label_columns=['Close'])\n",
    "IPython.display.clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input shape: (32, 30, 7)\n",
      "Output shape: (32, 30, 1)\n",
      "Label shape: (32, 30, 1)\n"
     ]
    }
   ],
   "source": [
    "print('Input shape:', wide_window.example[0].shape)#[batch_size, timesteps, feature]\n",
    "print('Output shape:', lstm_model(inputs= wide_window.example[0]).shape)\n",
    "print('Label shape:', wide_window.example[1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization the architecture\n",
    "- `lstm_model.summary()`\n",
    "- Graphs dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "#Op-level graph\n",
    "%tensorboard --logdir logs\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm (LSTM)                  (None, 30, 20)            2240      \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 30, 1)             21        \n",
      "=================================================================\n",
      "Total params: 2,261\n",
      "Trainable params: 2,261\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "lstm_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loss', 'mean_absolute_error', 'mse']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_performance={}\n",
    "performance={}\n",
    "val_performance['LSTM0'] = lstm_model.evaluate(x = wide_window.val,verbose=0)\n",
    "performance['LSTM0'] = lstm_model.evaluate(wide_window.test, verbose=0,return_dict=False)\n",
    "lstm_model.metrics_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.005833860021084547, 0.06880035251379013, 0.005833861883729696]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "performance['LSTM0']#test evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.005364998709410429, 0.039690278470516205, 0.005364999640733004]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_performance['LSTM0']#val/train evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use prediction to calculate the metrics as above:  \n",
    "\n",
    "```python\n",
    "import itertools\n",
    "\n",
    "pred = []\n",
    "label = []\n",
    "for input_batch,label_batch in wide_window.test:\n",
    "    pred_batch = lstm_model(input_batch)\n",
    "    pred.append(list(pred_batch.numpy().flatten()))\n",
    "    label.append(list(label_batch.numpy().flatten()))\n",
    "\n",
    "\n",
    "#flatten list of lists\n",
    "pred = list(itertools.chain(*pred))\n",
    "label = list(itertools.chain(*label))\n",
    "#sklearn version\n",
    "print(metrics.mean_absolute_error(label, pred), metrics.mean_squared_error(label, pred))\n",
    "\n",
    "#tf.keras.loss version\n",
    "mse = tf.keras.losses.MeanSquaredError()\n",
    "mae = tf.keras.losses.MeanAbsoluteError()\n",
    "print(mae(label, pred).numpy(), mse(label, pred).numpy()) \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAF5CAYAAACcBlM8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAltklEQVR4nO3dfZhdZX3v//eHQIjyFItpDyWURBrUUCDE8CBWxAcUUAERj0lVBHqkWFCsrS3U81PUY6XKqRZL4WBFxYNGy0GIGEHrkQerSMJTICBtivTHFNRAJYA8Br7nj70Th2Ems5LZKzPsvF/Xta/Z61732vs780c+ue91r7VSVUiSpP6z2XgXIEmS2mHIS5LUpwx5SZL6lCEvSVKfMuQlSepTm493Ab30/Oc/v2bMmDHeZUiStNFcd91191bVtOH29VXIz5gxg6VLl453GZIkbTRJ/n2kfU7XS5LUpwx5SZL6lCEvSVKf6qtz8sN54oknGBgY4NFHHx3vUvrGlClTmD59OltsscV4lyJJWoe+D/mBgQG22WYbZsyYQZLxLudZr6q47777GBgYYObMmeNdjiRpHVqfrk9ycJLbk6xIcsow+5PkzO7+ZUnmdttfmOTGQa8Hkrxvfb//0UcfZfvttzfgeyQJ22+/vTMjkvQs0OpIPskk4CzgIGAAWJJkUVXdOqjbIcCs7mtf4Gxg36q6HZgz6HP+A/jGBtaxob+ChuHfU5KeHdoeye8DrKiqO6rqcWAhcPiQPocD51fHNcDUJDsM6fNq4N+qasRrASVJ0tO1fU5+R+CuQdsDdEbro/XZEbhnUNt84Ku9KGjGKd/qxcesdefprx+1z4EHHsipp57K6173urVtn/nMZ/iXf/kX/v7v/37Y/meccQbz5s3j0EMP5Stf+QpTp059Wp/TTjuNrbfemj/7sz8b8Xsvvvhidt11V2bPng3Ahz70IQ444ABe85rXNPztJEnPZm2P5Ieb16316ZNkMnAY8I/DfkFyfJKlSZauXLlygwtt04IFC1i4cOHT2hYuXMiCBQtGPXbx4sXPCPimLr74Ym699ddnRj760Y8a8JK0CWk75AeAnQZtTwfuXs8+hwDXV9XPh/uCqjq3quZV1bxp04a9de+4O+qoo7j00kt57LHHALjzzju5++67+cpXvsK8efPYbbfd+PCHPzzssTNmzODee+8F4OMf/zgvfOELec1rXsPtt9++ts/nPvc59t57b/bcc0/e/OY38/DDD/PDH/6QRYsW8YEPfIA5c+bwb//2bxxzzDFceOGFAHzve99jr732Yvfdd+e4445bW9uMGTP48Ic/zNy5c9l99935yU9+0uafRpLUorZDfgkwK8nM7oh8PrBoSJ9FwNHdVfb7AauqavBU/QJ6NFU/Xrbffnv22WcfLrvsMqAzin/rW9/Kxz/+cZYuXcqyZcu48sorWbZs2Yifcd1117Fw4UJuuOEGLrroIpYsWbJ235FHHsmSJUu46aabePGLX8znP/959t9/fw477DA+9alPceONN7LLLrus7f/oo49yzDHH8LWvfY2bb76Z1atXc/bZZ6/d//znP5/rr7+ed7/73Zxxxhkt/EUkSRtDqyFfVauBk4DLgduAr1fV8iQnJDmh220xcAewAvgc8Mdrjk/yXDor8y9qs86NYfCU/Zqp+q9//evMnTuXvfbai+XLlz9tan2oq6++mje96U0897nPZdttt+Wwww5bu++WW27h5S9/ObvvvjsXXHABy5cvX2ctt99+OzNnzmTXXXcF4J3vfCdXXXXV2v1HHnkkAC95yUu48847N/RXliSNs9ZvhlNVi+kE+eC2cwa9L+DEEY59GNi+1QI3kiOOOIL3v//9XH/99TzyyCM873nP44wzzmDJkiU873nP45hjjhn12vORLl075phjuPjii9lzzz354he/yBVXXLHOz+n8yUe25ZZbAjBp0iRWr169zr6SpImr7+94N1FsvfXWHHjggRx33HEsWLCABx54gK222ortttuOn//853z729/mwAMPHPH4Aw44gGOOOYZTTjmF1atX881vfpM/+qM/AuDBBx9khx124IknnuCCCy5gxx13BGCbbbbhwQcffMZnvehFL+LOO+9kxYoV/O7v/i5f/vKXecUrXtHK7y2N6LTtxrsCaXyctmqjfdUmF/JNLnlry4IFCzjyyCNZuHAhL3rRi9hrr73YbbfdeMELXsDLXvaydR47d+5c3vrWtzJnzhx23nlnXv7yl6/d97GPfYx9992XnXfemd13331tsM+fP593vetdnHnmmWsX3EHn3vNf+MIXeMtb3sLq1avZe++9OeGEE57xnZKkZ7eMNnX7bDJv3rxaunTp09puu+02XvziF49TRf3Lv6vGzJG8NlU9Hsknua6q5g23z0fNSpLUpwx5SZL6lCEvSVKfMuQlSepThrwkSX3KkJckqU9tctfJ9/yynVEuhbjvvvt49atfDcDPfvYzJk2axJoH6Vx77bVMnjx5ncdfccUVTJ48mf3337839UqSNhmbXshvZNtvvz033ngj0OwZ8ENdccUVbL311oa8JGm9OV0/Dq677jpe8YpX8JKXvITXve513HNP56F7Z555JrNnz2aPPfZg/vz53HnnnZxzzjl8+tOfZs6cOVx99dXjXLkk6dnEkfxGVlW85z3v4ZJLLmHatGl87Wtf44Mf/CDnnXcep59+Oj/96U/Zcsstuf/++5k6dSonnHDCeo/+JUkCQ36je+yxx7jllls46KCDAHjyySfZYYcdANhjjz1429vexhFHHMERRxwxjlVKkvqBIb+RVRW77bYbP/rRj56x71vf+hZXXXUVixYt4mMf+9ioz4WXJGldPCe/kW255ZasXLlybcg/8cQTLF++nKeeeoq77rqLV77ylXzyk5/k/vvv56GHHhrxcbGSJI1m0xvJb8Tn+A5ns80248ILL+S9730vq1atYvXq1bzvfe9j11135e1vfzurVq2iqviTP/kTpk6dyhvf+EaOOuooLrnkEj772c8+7RGzkiSty6YX8uPotNNOW/v+qquuesb+H/zgB89o23XXXVm2bFmbZUmS+pTT9ZIk9SlDXpKkPrVJhHxVjXcJfcW/pyQ9O/R9yE+ZMoX77rvPYOqRquK+++5jypQp412KJGkUfb/wbvr06QwMDLBy5crxLqVvTJkyhenTp493GZKkUfR9yG+xxRbMnDlzvMuQJGmj6/vpekmSNlWGvCRJfcqQlySpTxnykiT1KUNekqQ+ZchLktSnDHlJkvqUIS9JUp8y5CVJ6lOGvCRJfcqQlySpTxnykiT1qdZDPsnBSW5PsiLJKcPsT5Izu/uXJZk7aN/UJBcm+UmS25K8tO16JUnqF62GfJJJwFnAIcBsYEGS2UO6HQLM6r6OB84etO9vgcuq6kXAnsBtbdYrSVI/aXskvw+woqruqKrHgYXA4UP6HA6cXx3XAFOT7JBkW+AA4PMAVfV4Vd3fcr2SJPWNtkN+R+CuQdsD3bYmfV4ArAS+kOSGJP+QZKs2i5UkqZ+0HfIZpq0a9tkcmAucXVV7Ab8Chjunf3ySpUmWrly5cqz1SpLUN9oO+QFgp0Hb04G7G/YZAAaq6sfd9gvphP7TVNW5VTWvquZNmzatZ4VLkvRs13bILwFmJZmZZDIwH1g0pM8i4OjuKvv9gFVVdU9V/Qy4K8kLu/1eDdzacr2SJPWNzdv88KpaneQk4HJgEnBeVS1PckJ3/znAYuBQYAXwMHDsoI94D3BB9z8IdwzZJ0mS1qHVkAeoqsV0gnxw2zmD3hdw4gjH3gjMa7M+SZL6lXe8kySpTxnykiT1qXVO1yc5ssFnPNqdkpckSRPIaOfkPwdcwvDXsq9xAEPOuUuSpPE3Wsh/u6qOW1eHJP+7h/VIkqQeWec5+ap6+2gf0KSPJEna+MZ0Tr6qLuptOZIkqVdGm65/Y/fnbwL7A/+3u/1K4ArAkJckaYJaZ8hX1bEASS4FZlfVPd3tHeg8J16SJE1QTa+Tn7Em4Lt+DuzaQj2SJKlHmt7W9ooklwNfpfMY2PnA91urSpIkjVmjkK+qk5K8ic418QDnVtU32itLkiSN1fo8oOZ64MGq+qckz02yTVU92FZhkiRpbBqdk0/yLuBC4H91m3YELm6pJkmS1ANNF96dCLwMeACgqv6VzmV1kiRpgmoa8o9V1eNrNpJsTmcBniRJmqCahvyVSf4SeE6Sg4B/BL7ZXlmSJGmsmob8KcBK4Gbgj4DFVfXB1qqSJElj1nR1/duAhVX1uTUNSd5QVZe2U5YkSRqrpiP5zwJXJ3nxoLaPtlCPJEnqkaYh/1PgOODCJG/ptqWdkiRJUi80na6vqro+ySuArybZF5jUYl2SJGmMmo7k7wGoqnuB19G5fO732ipKkiSNXaOQr6rXD3r/VFV9oKqa/gdBkiSNg3VO1yf5TFW9L8k3GebmN1V1WGuVSZKkMRntnPyXuz/PaLsQSZLUW+sM+aq6rvvzyo1TjiRJ6pXRputvZh33qK+qPXpekSRJ6onRpuvfsFGqkCRJPTfadP2/b6xCJElSbzW6DC7JfkmWJHkoyeNJnkzyQNvFSZKkDdf0Wve/AxYA/wo8B/hvdO5nL0mSJqimt7WlqlYkmVRVTwJfSPLDFuuSJElj1DTkH04yGbgxySfp3OZ2q/bKkiRJY9V0uv4ddB5IcxLwK2An4M1tFSVJksau0Uh+0Cr7R4CPtFeOJEnqlaar69+Q5IYk/5nkgSQPurpekqSJrel0/WeAdwLbV9W2VbVNVW3b5MAkBye5PcmKJKcMsz9JzuzuX5Zk7qB9dya5OcmNSZY2rFWSJNF84d1dwC1VNeItboeTZBJwFnAQMAAsSbKoqm4d1O0QYFb3tS9wdvfnGq/sPsdekiSth6Yh/+fA4iRXAo+taayqvxnluH2AFVV1B0CShcDhwOCQPxw4v/sfiGuSTE2yQ1Xd0/SXkCRJz9R0uv7jwMPAFGCbQa/R7EhnFmCNgW5b0z4FfCfJdUmOH+4LkhyfZGmSpStXrmxQkiRJm4amI/nfqKrXbsDnZ5i2oVP+6+rzsqq6O8lvAt9N8pOquuppHavOBc4FmDdv3nqdTpAkqZ81Hcn/U5INCfkBOtfUrzEduLtpn6pa8/MXwDfoTP9LkqQGmob8icBlSR5Zz0volgCzkszs3jFvPrBoSJ9FwNHdVfb7Aauq6p4kWyXZBiDJVsBrgVsa1itJ0iZv1On6JJsBB1fVP6/vh1fV6iQnAZfTuWPeeVW1PMkJ3f3nAIuBQ4EVdM77H9s9/LeAbyRZU+dXquqy9a1BkqRN1aghX1VPJTkDeOmGfEFVLaYT5IPbzhn0vujMFAw97g5gzw35TkmS1Hy6/jtJ3pzusFqSJE18TVfXv5/OU+eeTPIInRXx1fSud5IkaeNr+oCaJtfES5KkCaTpSJ4khwEHdDevqKpL2ylJkiT1QtOn0J0OnEzndrS3Aid32yRJ0gTVdCR/KDCnqp4CSPIl4AbgGU+VkyRJE0PT1fUAUwe9367HdUiSpB5rOpL/BHBDku/TWVl/AHBqa1VJkqQxa7q6/qtJrgD2phPyf1FVP2uzMEmSNDbrM12/GXAv8Etg1yQHjNJfkiSNo0Yj+SR/DbwVWA481W0u4KoRD5IkSeOq6Tn5I4AXVtVjLdYiSZJ6qOl0/R3AFm0WIkmSeqvpSP5h4MYk3wPWjuar6r2tVCVJksasacgv6r4kSdKzRNNL6L7UdiGSJKm31ucSOkmS9CxiyEuS1KcMeUmS+lTj58kPluSvgFXAP1TVfb0tSZIk9cKGjuSvBVYDn+5hLZIkqYc2aCRfVRf3uA5JktRjjUbySXZN8r0kt3S390jy39stTZIkjUXT6frP0Xl+/BMAVbUMmN9WUZIkaeyahvxzq+raIW2re12MJEnqnaYhf2+SXeg8XpYkRwH3tFaVJEkas6YL704EzgVelOQ/gJ8Cb2+tKkmSNGZN711/B/CaJFsBm1XVg+2WJUmSxqrp6vqTk2xL55Gzn05yfZLXtluaJEkai6bn5I+rqgeA1wK/CRwLnN5aVZIkacyahny6Pw8FvlBVNw1qkyRJE1DTkL8uyXfohPzlSbYBnmqvLEmSNFZNV9f/ITAHuKOqHk6yPZ0pe0mSNEE1XV3/VJLpwB8kAbiyqr7ZamWSJGlMmq6uPx04Gbi1+3pvkk+0WZgkSRqbptP1hwJzquopgCRfAm6gcz97SZI0Aa3P8+SnDnq/XdODkhyc5PYkK5KcMsz+JDmzu39ZkrlD9k9KckOSS9ejVkmSNnlNR/KfAG5I8n06l84dQINRfJJJwFnAQcAAsCTJoqq6dVC3Q4BZ3de+wNndn2ucDNwGbNuwVkmSRMORfFV9FdgPuKj7emlVLWxw6D7Aiqq6o6oeBxYChw/pczhwfnVcA0xNsgNAd7Hf64F/aPTbSJKktdY5kh86dU5nNA7w20l+u6quH+XzdwTuGnL8vg367EjnKXefAf4c2GYdNR4PHA/wO7/zO6OUI0nSpmO06fr/uY59BbxqlOOHuyteNemT5A3AL6rquiQHjlhE1bl0npDHvHnzhn72mM045Vu9/khpwrvz9NePdwmSemCdIV9Vrxzj5w8AOw3ang7c3bDPUcBhSQ4FpgDbJvnfVeUjbiVJamCd5+STvD3JO4Zpf1eSP2jw+UuAWUlmJpkMzAcWDemzCDi6u8p+P2BVVd1TVadW1fSqmtE97v8a8JIkNTfadP2f0llJP9TXgO8DX1nXwVW1OslJwOXAJOC8qlqe5ITu/nOAxXSuw19B51G23i5XkqQeGC3kJ1XVg0Mbq+qBJFs0+YKqWkwnyAe3nTPofQEnjvIZVwBXNPk+SZLUMdoldFsk2WpoY/cpdJPbKUmSJPXCaCH/eeDCJDPWNHTfL+zukyRJE9Roq+vPSPIQcGWSrelc/vYr4PSqOntjFChJkjbMqLe17Z4/P6cb8hnuHL0kSZp4RruE7g1r3lfVQ8MF/OA+kiRp4hhtJP+pJP/B8HelW+OvAJ8QJ0nSBDNayP8c+JtR+vxrj2qRJEk9NNrCuwM3Uh2SJKnHGj1qVpIkPfsY8pIk9alRQz7JZkn23xjFSJKk3hk15KvqKdb9XHlJkjQBNZ2u/06SNydZ16V0kiRpAhn1jndd7we2Ap5M8gid6+arqrZtrTJJkjQmjUK+qrZpuxBJktRbTUfyJDkMOKC7eUVVeZc7SZImsEbn5JOcDpwM3Np9ndxtkyRJE1TTkfyhwJzuSnuSfAm4ATilrcIkSdLYrM/NcKYOer9dj+uQJEk91nQk/1fADUm+T2dl/QHAqa1VJUmSxmzUkE+yGfAUsB+wN52Q/4uq+lnLtUmSpDEYNeSr6qkkJ1XV14FFG6EmSZLUA03PyX83yZ8l2SnJb6x5tVqZJEkak6bn5I/r/jxxUFsBL+htOZIkqVeanpM/paq+thHqkSRJPdL0KXQnjtZPkiRNLJ6TlySpT3lOXpKkPtX0KXQz2y5EkiT1VtMH1Dw3yX9Pcm53e1aSN7RbmiRJGoum5+S/ADwO7N/dHgD+RysVSZKknmga8rtU1SeBJwCq6hE6t7eVJEkTVNOQfzzJc+gstiPJLsBjrVUlSZLGrOnq+g8DlwE7JbkAeBlwTFtFSZKksWu6uv67Sa6n8yS6ACdX1b1r9ifZraqWt1SjJEnaAE1H8lTVfcC3Rtj9ZWBuTyqSJEk90fSc/GhGXISX5OAktydZkeSUYfYnyZnd/cuSzO22T0lybZKbkixP8pEe1SpJ0iahVyFfwzUmmQScBRwCzAYWJJk9pNshwKzu63jg7G77Y8CrqmpPYA5wcJL9elSvJEl9r1chP5J9gBVVdUdVPQ4sBA4f0udw4PzquAaYmmSH7vZD3T5bdF/D/mdCkiQ9U69C/vER2ncE7hq0PdBta9QnyaQkNwK/AL5bVT8e+gVJjk+yNMnSlStXbmD5kiT1n6a3tU2Styf5UHf7d5Lss2Z/VY00jT7cufqho/ER+1TVk1U1B5gO7JPk957RsercqppXVfOmTZvW4LeRJGnT0HQk//fAS4EF3e0H6ZxrH80AsNOg7enA3evbp6ruB64ADm5YryRJm7ymIb9vVZ0IPApQVb8EJjc4bgkwK8nMJJOB+cCiIX0WAUd3Zwv2A1ZV1T1JpiWZCtC9295rgJ80rFeSpE1e0+vkn+iulF9zW9tpwFOjHVRVq5OcBFwOTALOq6rlSU7o7j8HWAwcCqwAHgaO7R6+A/Cl7vduBny9qi5t/JtJkrSJaxryZwLfAH4zyceBo4D/r8mBVbWYTpAPbjtn0PsCThzmuGXAXg3rkyRJQzS9re0FSa4DXk1nodwRVXVbq5VJkqQxaRTySb5cVe9g0DnxQW2SJGkCarrwbrfBG93z5C/pfTmSJKlX1hnySU5N8iCwR5IHkjzY3f4FcMlGqVCSJG2QdYZ8VX2iqrYBPlVV21bVNt3X9lV16kaqUZIkbYCmq+u/neSAoY1VdVWP65EkST3SNOQ/MOj9FDoPnrkOeFXPK5IkST3R9BK6Nw7eTrIT8MlWKpIkST2xoU+hGwCe8bAYSZI0cTS9Tv6z/PrpcZsBc4CbWqpJkiT1QNNz8ksHvV8NfLWq/rmFeiRJUo80PSf/pbYLkSRJvbXOkE9yM7+epn/aLjrPltmjlaokSdKYjTaSf8NGqUKSJPXcOkO+qv59zfskvwXs3d28tqp+0WZhkiRpbBpdQpfkvwLXAm8B/ivw4yRHtVmYJEkam6ar6z8I7L1m9J5kGvBPwIVtFSZJksam6c1wNhsyPX/fehwrSZLGQdOR/GVJLge+2t1+K7C4nZIkSVIvNL1O/gNJjgR+n87lc+dW1TdarUySJI1J09vabgVcUlUXJXkh8MIkW1TVE+2WJ0mSNlTT8+pXAVsm2ZHOgrtjgS+2VZQkSRq7piGfqnoYOBL4bFW9CZjdXlmSJGmsGod8kpcCbwO+1W1rumhPkiSNg6Yh/z7gVOAbVbU8yQuA77dWlSRJGrOmq+uvBK5Msm2SbarqDuC97ZYmSZLGoultbed1n0i3DLglyU1JXtJuaZIkaSyanlc/D/jjqroaIMnvA18AfNSsJEkTVNNz8g+uCXiAqvoB8GA7JUmSpF5Y50g+ydzu22uT/C86t7UtOre1vaLd0iRJ0liMNl3/P4dsf3jQ++pxLZIkqYfWGfJV9cqNVYgkSeqtxje0SfJ6YDdgypq2qvpoG0VJkqSxa3oJ3Tl0zsO/h85T6N4C7NxiXZIkaYyarq7fv6qOBn5ZVR8BXgrs1F5ZkiRprJqG/CPdnw8n+W3gCWBmOyVJkqReaBrylyaZCnwKuB64k87ldKNKcnCS25OsSHLKMPuT5Mzu/mVrLttLslOS7ye5LcnyJCc3rFWSJNH83vUf6779P0kuBaZU1ao1+5McVFXfHXpckknAWcBBwACwJMmiqrp1ULdDgFnd177A2d2fq4E/rarrk2wDXJfku0OOlSRJI2g6kl+rqh4bHPBdfz1C932AFVV1R1U9DiwEDh/S53Dg/Oq4BpiaZIequqeqru9+54PAbcCO61uvJEmbqvUO+RFkhPYdgbsGbQ/wzKAetU+SGcBewI+f8cXJ8UmWJlm6cuXK9SxbkqT+1auQH+nud8OF/9C+6+yTZGvg/wDvq6oHntGx6tyqmldV86ZNm9a0XkmS+l6vQn4kAzz9UrvpwN1N+yTZgk7AX1BVF7VYpyRJfadXIX/nCO1LgFlJZiaZDMwHFg3pswg4urvKfj9gVVXdkyTA54HbqupvelSnJEmbjPW5re3+wIzBx1TV+d2fRw53TFWtTnIScDkwCTivqpYnOaG7/xxgMXAosAJ4GDi2e/jLgHcANye5sdv2l1W1uGnNkiRtyhqFfJIvA7sANwJPdpsLOH+0Y7uhvHhI2zmD3hdw4jDH/YCRF/RJkqRRNB3JzwNmdwNZkiQ9CzQ9J38L8F/aLESSJPVW05H884Fbk1wLPLamsaoOa6UqSZI0Zk1D/rQ2i5AkSb3X9N71V7ZdiCRJ6q1G5+ST7JdkSZKHkjye5Mkkz7j7nCRJmjiaLrz7O2AB8K/Ac4D/1m2TJEkTVOOb4VTViiSTqupJ4AtJfthiXZIkaYyahvzD3dvS3pjkk8A9wFbtlSVJksaq6XT9O7p9TwJ+ReeBMm9uqyhJkjR2TVfX/3uS5wA7VNVHWq5JkiT1QNPV9W+kc9/6y7rbc5IMfZqcJEmaQJpO158G7APcD1BVN9J5Ip0kSZqgmob86qpa1WolkiSpp5qurr8lyR8Ak5LMAt4LeAmdJEkTWNOR/HuA3eg8nOarwAPA+1qqSZIk9UDT1fUPAx/sviRJ0rNAo5BPMg/4SzqL7dYeU1V7tFOWJEkaq6bn5C8APgDcDDzVXjmSJKlXmob8yqryunhJkp5Fmob8h5P8A/A9OovvAKiqi1qpSpIkjVnTkD8WeBGwBb+eri/AkJckaYJqGvJ7VtXurVYiSZJ6qul18tckmd1qJZIkqaeajuR/H3hnkp/SOScfoLyETpKkiatpyB/cahWSJKnnGj9Pvu1CJElSbzU9Jy9Jkp5lDHlJkvqUIS9JUp8y5CVJ6lOGvCRJfcqQlySpTxnykiT1KUNekqQ+ZchLktSnWg/5JAcnuT3JiiSnDLM/Sc7s7l+WZO6gfecl+UWSW9quU5KkftNqyCeZBJwFHALMBhYM8zS7Q4BZ3dfxwNmD9n0R75svSdIGaXskvw+woqruqKrHgYXA4UP6HA6cXx3XAFOT7ABQVVcB/9lyjZIk9aW2Q35H4K5B2wPdtvXtM6IkxydZmmTpypUrN7hQSZL6Tdshn2HaagP6jKiqzq2qeVU1b9q0aetVnCRJ/aztkB8Adhq0PR24ewP6SJKk9dR2yC8BZiWZmWQyMB9YNKTPIuDo7ir7/YBVVXVPy3VJktT3Wg35qloNnARcDtwGfL2qlic5IckJ3W6LgTuAFcDngD9ec3ySrwI/Al6YZCDJH7ZZryRJ/WTztr+gqhbTCfLBbecMel/AiSMcu6Dd6iRJ6l/e8U6SpD5lyEuS1KcMeUmS+pQhL0lSnzLkJUnqU4a8JEl9ypCXJKlPGfKSJPUpQ16SpD5lyEuS1KcMeUmS+pQhL0lSnzLkJUnqU4a8JEl9ypCXJKlPGfKSJPUpQ16SpD5lyEuS1KcMeUmS+pQhL0lSnzLkJUnqU4a8JEl9ypCXJKlPGfKSJPUpQ16SpD5lyEuS1KcMeUmS+pQhL0lSnzLkJUnqU4a8JEl9ypCXJKlPGfKSJPUpQ16SpD5lyEuS1KcMeUmS+pQhL0lSn2o95JMcnOT2JCuSnDLM/iQ5s7t/WZK5TY+VJEkjazXkk0wCzgIOAWYDC5LMHtLtEGBW93U8cPZ6HCtJkkbQ9kh+H2BFVd1RVY8DC4HDh/Q5HDi/Oq4BpibZoeGxkiRpBG2H/I7AXYO2B7ptTfo0OVaSJI1g85Y/P8O0VcM+TY4lyfF0pvkBHkpy+3pVKOkZ8tcbdNjzgXt7W4nUhz4yXLyNyc4j7Wg75AeAnQZtTwfubthncoNjqapzgXN7UaykDZdkaVXNG+86JP1a29P1S4BZSWYmmQzMBxYN6bMIOLq7yn4/YFVV3dPwWEmSNIJWR/JVtTrJScDlwCTgvKpanuSE7v5zgMXAocAK4GHg2HUd22a9kiT1k1Q94zS3JK23JMd3T59JmiAMeUmS+pS3tZUkqU8Z8pIk9SlDXpKkPtX2dfKSNmFJdgNWAZtV1f8/3vVImxoX3klqRZLXA58G/hn4L8DXquqL41qUtIlxJC+pp5IE2A44FXhvVV2W5OXAPyZ5TlWdPb4VSpsOQ15ST1VnevD+JNcC93fbrk5yMPDtJA9V1ZfHs0ZpU+HCO0k9k+RlSQ7pbq4CTlmzr6puBN4GHJnkt7ojfkktMuQl9USSqcC3gA8leV1VfQTYIsniQd2uAh4BnioXBEmtc7peUq+sAv4O+D1g3ySbVdXrk1ya5BvAWcAuwB74b4+0UTiSlzQmSabA2nPxPwRmAc8BXtUd0b8B+BHwauAoYH73SZOSWuYldJI2WJLXAX8OLKqqv+22HQts2+3yYuCSqvp2d99zquqRcSlW2gQ5ZSZpgyTZCfgN4HeB9yfZBbgGmE5nVf3fAe8C3pJkSlV9A3h0nMqVNkmGvKT11h3Bnwy8D/hT4EjgucB/drdnA8uBC7qH/BDWTulL2kgMeUnrpRvwZwDvrqp/SfKfdNb3vIPOSP0N3dfdVfWrJJ8z3KXx4Tl5SY0leS1wEZ1b1P7hoPapdIL9bcAnq+r7g/bFkJfGh6vrJTWS5BXAZ4GTgO2T/HmS5wFU1f3AN4EvAx/t3t2O7j4DXhonTtdLGlWSGXRWzB9dVT9Och3wN0B1p+Pvr6pV3RvfPEHnfLykceZ0vaR1SnIocCLwP6rqR0m2qKonkswG/hb4DnBuVa3q9nd6XpognK6XNKLuIrtP0Flo9+Nu82qAqrqVzur6VwEnJ9mu227ASxOEIS9pWN072b0b+Fh3Id1zkjwfOCTJ5t0R+3LgL4A5wKTxq1bScDwnL2kkTwD3Avck2Rn4E2AmsA+dm968C7i3qpYlmV9Vj49fqZKG40he0lqDH/9aVU8Ct9CZqr+azsK7LwG/Tefe9CcNOvSJjVimpIYcyUsabBKwOsnkqnq8qj6T5NvAlKq6KcnmVVVJLgOmrDnI8/DSxORIXhIA3fPtK5L8RlU9nmQyQFXdXlU3dd+vTvIO4BjgkvGrVlIThrwkAKrqXuA9wA+TPK8b9JuvmcJPMjXJX9BZUf/2qrptHMuV1IDT9ZLWqqpvJlkNLE0yr6p+mWRzOpfN7Qr8Eji8qgbGtVBJjTiSl/Q03We/n0Qn6J/XnaI/ic4T5RYb8NKzh3e8kzSsJIcAfw18kc7lcguq6sbxrEnS+jHkJY0oyevpPHhmrzWL7yQ9exjyktYpyXOr6uHxrkPS+jPkJUnqUy68kySpTxnykiT1KUNekqQ+ZchLktSnDHlJkvqUIS9JUp/6f4cGBitaB4taAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(len(performance))\n",
    "width = 0.3\n",
    "metric_name = 'mean_absolute_error'\n",
    "mae_index = lstm_model.metrics_names.index('mean_absolute_error')\n",
    "val_mae = [v[mae_index] for v in val_performance.values()]\n",
    "test_mae = [v[mae_index] for v in performance.values()]\n",
    "\n",
    "plt.ylabel('mean_absolute_error [Close, normalized]')\n",
    "plt.bar(x - 0.17, val_mae, width, label='Validation')\n",
    "plt.bar(x + 0.17, test_mae, width, label='Test')\n",
    "plt.xticks(ticks=x, labels=performance.keys(),\n",
    "           rotation=45)\n",
    "_ = plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the trained LSTM\n",
    "\n",
    "```\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = lstm_model.to_json()\n",
    "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save neural network structure to YAML (no weights)\n",
    "model_yaml = lstm_model.to_yaml()\n",
    "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save entire network to HDF5 (save everything, suggested)\n",
    "save_path = \"./trainedNN\"\n",
    "lstm_model.save(os.path.join(save_path,\"lstm-base.h5\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reload\n",
    "save_path = \"./trainedNN\"\n",
    "lstm_model_reload = tf.keras.models.load_model(os.path.join(save_path,\"lstm-notebook.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning hyperparameters\n",
    "\n",
    "Some of the hyperparameters seen so far include:\n",
    "\n",
    "* Number of layers in the neural network\n",
    "* How many neurons in each layer\n",
    "* What activation functions to use on each layer\n",
    "* Dropout percent on each layer\n",
    "* L1 and L2 values on each layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overfitting and Early Stopping \n",
    "\n",
    "It is important to segment the original dataset into several datasets(already done in the previous section):\n",
    "\n",
    "* **Training Set** : `wide_window.train`\n",
    "* **Validation Set** : `wide_window.val`\n",
    "* **Holdout Set**: `wide_window.test`\n",
    "\n",
    "The training data(`wide_window.train`) are used to train the neural network until the validation set no longer improves.  This attempts to stop at a near optimal training point.  This method will only give accurate \"**out of sample**\" predictions for the validation set(`wide_window.val`), this is usually 20% or so of the data.  The predictions for the training data will be overly optimistic, as these were the data that the neural network was trained on.\n",
    "\n",
    "I design this data split method in the class `WindowGenerator` as shown in section data windowing. The model fitting takes the early stopping through `tf.keras.callbacks.EarlyStopping`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I design the earlystopping based on validation set as:\n",
    "\n",
    "```python\n",
    "early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss',\n",
    "                                                    patience=2,\n",
    "                                                    mode='min',restore_best_weights=False)\n",
    "```\n",
    "\n",
    "- details: [https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-fold CrossValidation for hyperparameters\n",
    "- reference:[https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning))  \n",
    "\n",
    "- K-fold for sectional data: `sklearn.model_selection.KFold`\n",
    "- K-fold for time series needs rolling basis: `sklearn.model_selection.TimeSeriesSplit`\n",
    "- K-fold for time series needs blocking rolling basis:[https://goldinlocks.github.io/Time-Series-Cross-Validation/](https://goldinlocks.github.io/Time-Series-Cross-Validation/)\n",
    "\n",
    "<img src=\"./pic/class_1_kfold.png\" alt=\"RNN Architecture\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11670, 7) (11666, 7)\n",
      "(23336, 7) (11666, 7)\n",
      "(35002, 7) (11666, 7)\n",
      "(46668, 7) (11666, 7)\n",
      "(58334, 7) (11666, 7)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFgCAYAAAD3tH5OAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgPElEQVR4nO3dfXRU9b3v8c83D4RgkIcSCgoaRJLwZIogLSpWaQUU5Z7W45Kqx9pbDirtql0cEVx6tPXSu3xa1mUtRUq91J620GqxXmxRLFJ75FQbMKmRR7FwQBsI8mwFSfK9f+yd6xgDmcBMfmHm/VprVmb27Pnt73cE/fj77b1j7i4AAACEkRO6AAAAgGxGGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGNAEszMk3hcbGZbzOyhdqwr38xmmFmNmf3DzHaZ2atmNjuFx1hpZk+laryEcS80s+VmVmdm75vZJjNbaGb92jjOxfH3Pyxhm5vZNxNeTzOzf0ph+QCQMnmhCwBOEmMSnhdKWiFpjqTnEravlfQlSe+1Y12PSbpO0vckvSqpu6TPSbpS0n0pOsZ0SUdSNJakKIhJWinpGUlfl/SBpMGSrpV0pqTtJ3iIMZL+lvB6mqSa+HgA0KEYN30F2sbMiiQdkPQ1d18YsI4ukvZKutPdH2z2nvkJ/uU2s0J3/+BExjjG2D+XdI6kc5rX2dbazexiSS9JGu7uNUfZp1JSjbvfeLw1A0C6sEwJpFDzZcp42a3SzCaZ2dp4KfE5M+tpZmeb2UvxEl2lmZ3TbKwcM5ttZm+Z2WEz22hmX03Y5RRJ+ZJqm9fRQsDpaWaPm9kOMztkZqvM7LPN9vF4yfMRM6uT9Ea8/RPLlGY2LO7jQPz4tZn1SXg/38weMrP/jmt/18yWmFmneJfukna2FLoStzV9n2b272ZWa2YHzeznZtbtKP8IEnv5ZlP9kkZK+mrCkvKNx/o8ALQnwhiQfmdIulfSXYqWy86XNF/Sovjxz4pOGVhkZpbwuR/En5kvaZKkJZKeMLMrJMnd6yRtk/QdM/uymXVt6eBmViDpRUmXSpop6Z8k1Ul6MTFAxWZK6ivpXyR96yjjnS3pFUmd4/1ulDRU0v9NqP8ORcun/x4f99uS9knKjd9fI+mSOGSd1dJxEnxF0hcl/aukGfF3saCVzySaLmm9pN8pWr4co48vLwNAUJwzBqRfT0lj3H2zJMUzYDMlfdXdn4y3maKAUC5pXRx4blG0FPrTeJwXzayvpHskLY233ago0D0tqdHMXo9fP+ruH8b7XC9pmKSh7r4pPt6LkjZI+re4lia17n5NK/3co2g27rKmY5jZXxUFnsvjPkZL+kVC7ZL0q4TnD0q6QFFIvdfM/i7pWUkPu/vGZscrlDTJ3Q/Gx3pf0s/MbLC7r2ulVrn72vgzde7+59b2B4D2xswYkH5bmoJY7K3454oWtp0e//yCpEZJS8wsr+kh6Q+SPmNmuZLk7iskDVQ0e/SEpE8pCjorzKzp7/cXJa2W9LeEcSTpj5JGNas1mRmjLyqapWtMGO9vkrYkjFcl6UYzu93Mzmk24yd33x/3eL6k/y1ps6SpktaY2bnNjre8KYjFfiPJJJ2XRK0A0OERxoD029vs9YctbG/a1jn+2UvRkt4+RVcyNj0WKprR7tv0QXc/4O6L3P1fJZ0l6X8pmnW6MmGszzUb54ikr0nq36y2HUn000vSrBbGOythvDmSfqhoibBa0jYzuzVxEI/8l7vf6e5jFQW5RkVLm4l2NvvcB5IOJn4HAHAyY5kS6Jh2S6pXFKoaW3h/Zwvb5O5uZg8qCjTlkn4bj1WpaNmzucPNh0iytiVq+bytXXEdhyTdLeluMxsk6WZJj5jZBndfdpTaq8xsuaQhzd7qnfjCzAolFUn6exK1AkCHRxgDOqYVimbGurn78pZ2MLN8Sae4+95mbw2KfzbNcv1B0nhJ/+3uLYa4NvqDonPQVidzCwp332Rmt0n6hqKgtczMejevJV7KHKhPzs5damZFCUuVX1YUGivbUPOH+mjWEQA6FMIY0AG5+wYzm6foCssHFAWPzoquWix196mSuknaaGY/VXSfrX2SyhRdyfiOotkrSXpS0czUyvi2G28rOrdstKIT9r/fxvK+I+k1Sc+Z2ROKZsNOV3TV5EJ3X2lmSxSdp/a6ohu6Nl0x+nI8xoL4nLanFZ0v1kPRsmmFpKubHe+D+FgPKlqafFDSEndf24aa10uaYGYTFN2U92/u3p435wWAoyKMAR3XNyRtVHRLh3sl7Vd0l/+fxO/vl/SAoisYr5V0qqIQ9rykOe6+T4qWDM3skniM70r6tKJlztcUXcHYJu6+0cw+p+i8sPmKrnZ8R9GMWdOFCKskXaPoSs2cuO6r3L1pNmuuoitB71YUsPZKelPSBHd/odkhFym6ye5PFC1PPquWl1yPZY6iW4z8StH39DVF598BQHDcgR9Ah2VmWyQ95e63ha4FANKFqykBAAACIowBAAAExDIlAABAQMyMAQAABJSWqyl79erlJSUl6RgaAAC0YPXq1bvcvTh0HWi7tISxkpISVVa25X6MAADgRJjZ1tA14PiwTAkAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAElJb7jKVKyeznQpcAAEBabblvUugSEBgzYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABdeg78HNXYgAAkOmYGQMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAE1KFv+goAAFJr9erVvfPy8hZIGiYmZdpDo6Sa+vr6qSNHjtzZ0g6EMQAAskheXt6CPn36DC4uLt6Tk5PjoevJdI2NjVZXVzektrZ2gaTJLe1DIgYAILsMKy4u3k8Qax85OTleXFy8T9FMZMv7tGM9AAAgvByCWPuKv++jZi7CGAAAQECcMwYAQBbb+/BtFf7B+ynLA1Z4Sn33GQ9VH+392tra3IsvvrhMknbt2pWfk5PjPXv2rJekqqqqdZ07dz7mrN3SpUu7FhQUNF566aXvS9IDDzxQ3KVLl8ZvfvOb76Wqh2Q0r+NEEMYAAMhiqQxiyYzXp0+fhvXr16+VpBkzZpxWVFTUcO+99+5IdvwVK1Z0LSoqamgKQbfffnvdiVV8fJrXcSJYpgQAAEH96U9/6nLeeeeVDR06dPCFF144aOvWrfmSNGfOnN4DBw4cWlpaOuSKK644a8OGDZ2efPLJ4nnz5n26vLx8yLJly4pmzJhx2t133/1pSRo9enTZLbfccvrw4cMHl5SUDFu2bFmRJB04cCDn8ssvP6u0tHTIpEmTzjrnnHPKX3755S7N65g+ffrpTcebNm1aP0l699138yZMmDBw2LBhg4cNGzb4hRdeOKWlOk6kf2bGAABAMO6ub33rW2c899xzb5122mn1P/7xj3vcdtttp//617/e8uijj/bZunXrG4WFhb5r167cXr16Ndxwww11ibNpL7zwwqmJ49XX19sbb7yxbvHixd3uvffe0yZOnLjxwQcfLO7evXvDxo0b1/7lL3/pPGbMmKHN69ixY0fu7373ux5vv/12TU5Ojnbt2pUrSTfddFP/GTNm7JgwYcLBTZs2dZowYcKgt99++83mdZwIwhgAAAjm8OHDOZs2bSocN25cqSQ1NjaquLj4iCSVlZV98KUvfWnA5MmT91533XV7kxnv6quv3iNJ559//vszZ87sJEmrVq0quvXWW3dK0nnnnXeotLT0H80/17Nnz4aCgoLGKVOmnDlp0qR911xzzT5JeuWVV07dtGlTYdN+Bw8ezN2zZ09KVxYJYwAAIBh319lnn/1BVVXV+ubvvfTSS5t+//vfd33mmWe6P/DAA6dt2rSpprXxmi4AyMvLU0NDgzUdozX5+fmqqqpa9+yzz566aNGiHj/60Y96//nPf97o7qqsrFxXVFSUttuBJJXszGyLmb1hZlVmVpmuYgAAQHYpKCho3L17d96LL754iiQdPnzYKisrOzc0NGjz5s2drrzyygNz587dfuDAgdx9+/bldu3ateHAgQO5bTnG+eeff3DRokU9JGn16tWdN27cWNh8n3379uXs3r0795prrtk3b968bevWresiSRdeeOH++++/v3fTfqtWrSqUpOOp42jaMs12ibt/xt1HpeLAAAAgPCs8pT7keDk5OVq0aNHm2bNn9ysrKxsydOjQIX/84x+L6uvr7dprrx1QWlo6ZNiwYUNuuummHb169Wq46qqr9j733HPd23Li/MyZM+vee++9vNLS0iHf+973+pSVlX3Qo0ePhsR99u7dmztx4sRBpaWlQ8aOHVs2Z86cbZI0f/78bWvWrDmltLR0yMCBA4c+9thjxZJ0PHUcjSUzdWdmWySNcvddyQw6atQor6xkAg0AgPZiZquTmTCprq7eUlFRkdR/zzNFfX29PvzwQ+vSpYu/+eabBePHjy/dvHlzTWv3NEul6urqXhUVFSUtvZfsOWMu6QUzc0mPu/v85juY2TRJ0yTpjDPOOM5SAQAAUuvAgQM5Y8eOLTty5Ii5u77//e9vbc8g1ppkw9gF7v6umfWWtNzM1rv7y4k7xAFtvhTNjKW4TgAAgOPSo0ePxpqamnWh6ziapM4Zc/d34587JS2RNDqdRQEAAGSLVsOYmZ1iZl2bnksaL6nVS0sBAADQumSWKT8taYmZNe3/C3dfltaqAAAAskSrYczd35ZU0Q61AAAAZB3uwA8AQBb7H/9zdcX+g/UpywOnFuXV//aJkdVHe7+2tjb34osvLpOkXbt25efk5HjPnj3rJamqqmrdsa5yfPnll7s88cQTn1q4cOG2Y9UwYsSI8tdff/0Td/RPt9mzZ/e57777atv6OcIYAABZLJVBLJnx+vTp07B+/fq1kjRjxozTmv+y7SNHjig/P7/Fz1500UX/uOiiiz7xeyWbCxHEJOnRRx/tezxhLKW/6BIAAKCtrrrqqpKpU6f2++xnP1s6ffr0fi+99FKXESNGlA8ePHjIiBEjyqurqwskaenSpV0vueSSs6UoyF199dUlo0ePLuvXr9/wOXPm/P9fWdSlS5cRTfuPHj26bOLEiWcNGDBg6OTJkwc0NjZKkhYvXtxtwIABQ0eOHFl244039m8aN1FlZWXn4cOHDy4vLx9SWlo65I033iiQpLlz5/Zs2n7ttdeeWV9fr+nTp59++PDhnPLy8iGTJ08e0Jb+mRkDAADBbd68ufMrr7yyMS8vT7t378557bXX1ufn5+uZZ57pevvtt/d7/vnnNzf/zFtvvdV51apVG/bu3Zs7ePDgYTNnzqwrKCj42DLnunXrCquqqt4uKSk5MnLkyPLly5cXjR079v1bb731zJUrV64vLy//8Morr2wxPP3gBz8onj59+o5bbrll96FDh6y+vl5r1qzp/NRTT/WsrKxcX1BQ4Ndff/0Z8+bN+9TcuXPfWbhwYe+mWb+2IIwBAIDgvvzlL+/Jy4tiSfwLuwds2bKls5n5kSNHrKXPjB8/fm9hYaEXFhbW9+zZ88j27dvzBg4ceCRxn+HDh7/ftG3o0KH/2Lx5c6euXbs29O/f/3B5efmHkjRlypTdCxYsKG4+/pgxY95/6KGH+m7fvr3TlClT9gwfPvzwsmXLutbU1HSpqKgYLEmHDh3K6d279wn9fk/CGAAACK6oqKix6fmsWbNO//znP39g+fLlmzds2NBp3LhxZS19JnEWLDc3V/X19Z8IbS3tk8zv5Zakm2++effYsWPfX7JkSbfLLrusdO7cuVvc3a6++ur3fvjDH77TpgaPgXPGAABAh7J///7cfv36fShJjz/+eK9Uj19RUXFo27ZtBRs2bOgkSYsXL+7Z0n5r167tNHjw4MN33XXXzvHjx++tqqoqnDhx4v6lS5f2eOedd/IkaceOHbkbN27sJEl5eXl++PDhFmfxjoUwBgBAFju1KO+EltjSMd6sWbNqv/Od7/Q799xzyxsaGlJR1scUFRX5ww8/vHXixImDRo4cWda7d+8jXbt2/cSBfvazn/UsLS0dWl5ePmTTpk2db7rppvdGjhx56K677nrnC1/4QmlpaemQcePGlW7bti1fkq677rq6wYMHt/kE/qSn6tpi1KhRXllZmfJxAQBAy8xstbuPam2/6urqLRUVFbvao6aObN++fTndunVrbGxs1A033HDGoEGDDt1zzz0703W86urqXhUVFSUtvcfMGAAAyDqPPPJIr/Ly8iGDBg0aun///twZM2YEC6icwA8AALLOPffcszOdM2FtwcwYAADZpbGxsbHNJ5nj+MXfd+PR3ieMAQCQXWrq6uq6EcjaR2Njo9XV1XWTVHO0fVimBAAgi9TX10+tra1dUFtbO0xMyrSHRkk19fX1U4+2A2EMAIAsMnLkyJ2SJoeuAx8hEQMAAATUoWfGbr5/T+gSAABJuHPrUVdg0Ir+c58OXQICY2YMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAIKAOfQf+ebN6hC4BAJAU7iIPHC9mxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACCjpMGZmuWb2upktTWdBAAAA2aQtM2O3SlqXrkIAAACyUVJhzMz6SZokaUF6ywEAAMguyc6MPSLpdkmNR9vBzKaZWaWZVdbV1aWiNgAAgIzXahgzsysk7XT31cfaz93nu/sodx9VXFycsgIBAAAyWTIzYxdImmxmWyQtkjTOzP4jrVUBAABkiVbDmLvf4e793L1E0hRJK9z9+rRXBgAAkAW4zxgAAEBAeW3Z2d1XSlqZlkoAAACyEDNjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAAB5YUu4Fhuvn9P6BIAQHdunRq6hJNW/7lPhy4B6PCYGQMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIqEPfgX/erB6hSwAASdxFHkD6MDMGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACCgVsOYmXU2s9fMrNrM3jSz77ZHYQAAANkgL4l9Dksa5+4HzSxf0n+a2e/d/c9prg0AACDjtRrG3N0lHYxf5scPT2dRAAAA2SKpc8bMLNfMqiTtlLTc3V9tYZ9pZlZpZpV1dXUpLhMAACAzJRXG3L3B3T8jqZ+k0WY2rIV95rv7KHcfVVxcnOIyAQAAMlObrqZ0972SVkqamI5iAAAAsk0yV1MWm1n3+HmhpC9KWp/mugAAALJCMldT9pX0UzPLVRTefuXuS9NbFgAAQHZI5mrKv0oa0Q61AAAAZB3uwA8AABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEFBe6AKO5eb794QuAcgKd26dGrqEk1L/uU+HLgFABmBmDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACCgDn0H/nmzeoQuAcgS3EkeAEJhZgwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECthjEz629mL5nZOjN708xubY/CAAAAskFeEvvUS/o3d19jZl0lrTaz5e6+Ns21AQAAZLxWZ8bc/e/uviZ+fkDSOkmnp7swAACAbNCmc8bMrETSCEmvtvDeNDOrNLPKurq6FJUHAACQ2ZIOY2ZWJOlpSd929/3N33f3+e4+yt1HFRcXp7JGAACAjJVUGDOzfEVB7Ofu/pv0lgQAAJA9krma0iT9RNI6d384/SUBAABkj2Rmxi6Q9C+SxplZVfy4PM11AQAAZIVWb23h7v8pydqhFgAAgKzDHfgBAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAALKC13Asdx8/57QJSAD3Ll1augSTkr95z4dugQAyArMjAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAE1KHvwD9vVo/QJSAjcCd5AEDHxcwYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICAWg1jZvaEme00s5r2KAgAACCbJDMztlDSxDTXAQAAkJVaDWPu/rKk3e1QCwAAQNZJ2TljZjbNzCrNrLKuri5VwwIAAGS0lIUxd5/v7qPcfVRxcXGqhgUAAMhoXE0JAAAQEGEMAAAgoGRubfFLSf8lqczMtpvZ19NfFgAAQHbIa20Hd/9KexQCAACQjVimBAAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIKC90Acdy8/17QpfQ4d25dWroEk5a/ec+HboEAACYGQMAAAiJMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIqEPfgX/erB6hSzgJcBd5AABOZsyMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIyd0/9oGZ1krameNheknaleMyTRbb2Tt/ZJ1t7p+/sk47ez3T34hSPiXaQljCWDmZW6e6jQtcRQrb2Tt/ZJ1t7p+/sk82945NYpgQAAAiIMAYAABDQyRTG5ocuIKBs7Z2+s0+29k7f2Sebe0czJ805YwAAAJnoZJoZAwAAyDiEMQAAgIBOijBmZhPNbIOZvWVms0PXczzM7Akz22lmNQnbeprZcjPbFP/skfDeHXG/G8xsQsL2kWb2Rvzeo2Zm8fYCM1scb3/VzEratcGjMLP+ZvaSma0zszfN7NZ4e0b3bmadzew1M6uO+/5uvD2j+25iZrlm9rqZLY1fZ0vfW+Kaq8ysMt6W8b2bWXcze8rM1sd/18dket9mVhb/c2567Dezb2d630gTd+/QD0m5kjZLOktSJ0nVkoaErus4+rhI0rmSahK2PSBpdvx8tqT74+dD4j4LJA2I+8+N33tN0hhJJun3ki6Lt0+XNC9+PkXS4tA9x7X0lXRu/LyrpI1xfxnde1xjUfw8X9Krkj6X6X0n9D9D0i8kLc2WP+txPVsk9Wq2LeN7l/RTSVPj550kdc+GvhP6z5VUK+nMbOqbRwr/DIUuoNUCoz+gzye8vkPSHaHrOs5eSvTxMLZBUt/4eV9JG1rqUdLz8ffQV9L6hO1fkfR44j7x8zxFd3a20D238B38VtKl2dS7pC6S1kj6bDb0LamfpD9IGqePwljG9x3Xs0WfDGMZ3bukUyX9rXkdmd53s17HS3ol2/rmkbrHybBMebqkbQmvt8fbMsGn3f3vkhT/7B1vP1rPp8fPm2//2GfcvV7SPkmfSlvlxyGeYh+haJYo43uPl+qqJO2UtNzds6JvSY9Iul1SY8K2bOhbklzSC2a22symxdsyvfezJNVJ+j/x0vQCMztFmd93oimSfhk/z6a+kSInQxizFrZl+v04jtbzsb6LDv09mVmRpKclfdvd9x9r1xa2nZS9u3uDu39G0UzRaDMbdozdM6JvM7tC0k53X53sR1rYdtL1neACdz9X0mWSvmFmFx1j30zpPU/RKRg/cvcRkt5XtDx3NJnStyTJzDpJmizp163t2sK2k7ZvpNbJEMa2S+qf8LqfpHcD1ZJqO8ysryTFP3fG24/W8/b4efPtH/uMmeVJ6iZpd9oqbwMzy1cUxH7u7r+JN2dF75Lk7nslrZQ0UZnf9wWSJpvZFkmLJI0zs/9Q5vctSXL3d+OfOyUtkTRamd/7dknb45lfSXpKUTjL9L6bXCZpjbvviF9nS99IoZMhjP1F0iAzGxD/H8gUSc8GrilVnpX01fj5VxWdT9W0fUp8Jc0ASYMkvRZPeR8ws8/FV9vc0OwzTWP9s6QV7h78/6DiOn8iaZ27P5zwVkb3bmbFZtY9fl4o6YuS1ivD+3b3O9y9n7uXKPq7usLdr1eG9y1JZnaKmXVteq7oPKIaZXjv7l4raZuZlcWbviBprTK87wRf0UdLlFL29I1UCn3SWjIPSZcrugpvs6Q7Q9dznD38UtLfJR1R9H87X1e09v8HSZvinz0T9r8z7neD4itr4u2jFP0LfrOkx/TRb1HorGia/C1FV+acFbrnuK4LFU2r/1VSVfy4PNN7l3SOpNfjvmsk3R1vz+i+m30HF+ujE/gzvm9F505Vx483m/5dlSW9f0ZSZfzn/RlJPbKk7y6S3pPULWFbxvfNI/UPfh0SAABAQCfDMiUAAEDGIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgP4fpE4TWoWJOQoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import TimeSeriesSplit\n",
    "from script.RNN.kfoldwindow import plot_cv_indices\n",
    "tscv = TimeSeriesSplit(n_splits = 5)\n",
    "for train_index, test_index in tscv.split(X=df):\n",
    "    cv_train, cv_test = df.iloc[train_index], df.iloc[test_index]\n",
    "    print(cv_train.shape,cv_test.shape)\n",
    "\n",
    "plot_cv_indices(tscv,X=df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7000, 7) (7000, 7)\n",
      "(7000, 7) (7000, 7)\n",
      "(7000, 7) (7000, 7)\n",
      "(7000, 7) (7000, 7)\n",
      "(7000, 7) (7000, 7)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFgCAYAAAD3tH5OAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjaklEQVR4nO3de3hV1ZnH8d+bBAIxyKUEUUGDlCTcighiUfGCLeKN1lpGqo51powXxqd2mII4OmqtTot1rFWLiNbS2iqMWK2DFaUqtdV6CRgKyE0ceEAMhPtFQULe+WPv1OMxlxM4YZlzvp/n2U/O2Wftdd51Es2PtfbeMXcXAAAAwsgJXQAAAEA2I4wBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxtBhmdquZecL2oZktMrMrE9qcEb/Wrxnee1MjbeaZ2ax0vm/c7/Skcde1TU+lxmao7VQzm2tmVWa228xWxrV0S1P/V8TjK0xHfwn9djaz+83sPTPbY2brzex5M/v6AfS12szuSng+3czKE54PMbNb01M5gEyUF7oAoIm2SxoZPz5M0gWSHjSzXe7+WLiyJEnjJO1rhn5/KGlqwvP/ktQhfr9aVZL2SvrfZnj/OpnZqZLmSXpa0nckfSSpt6RLJB0raV0a3uZZSUMlfZiGviRJZtZK0suSCiTdIWmVpG6SRkg6S9F4DsYPJbVNeD5E0i2Sbj3IfgFkKMIYWppqd3894fmLZnaypK9LChrG3P2dZup3laLAIEkysy2ScpI+h1rpCECpukbSUkmj/ZO7R8+VdK+Z2cF0bGa5knLdvUpR0EynMyT1kzTE3d9K2P+bg61b+vv3CwBSxjIlMsFOSa3qe9HMCszsXjOrjJek3jKzEXW0u9DM3jSzj8xss5n9wcyOradPM7P7zGyrmZ0U7/vUMmXtsqGZDTSz1+Nl1bfNbFhSX/lm9oCZbYvf9ydm9j0za9Kfx0hepkxYsj3LzH6fsIw4wsxy4/fZZGbvm9n4Ovo71cz+FNe92cweMrN2CU06SNrodfwZj+R9ZjbWzJaY2V4zW2NmE5Nen25m5Wb2dTNbImmPpJPqWqY0szZmdqeZrY37W2hm5yb1N8rM5sdj3mpmb5jZ6Ql1S1JlQ3UnfP9OMbMF8c9ORTwjWK/EZUozu0LSffHj2iXleQ0dDyD7EMbQ4phZXrwdbmaXSTpd0lMNHPKQpH9StCR1oaS1kp5N/KVqZv8o6XeKZqD+IW6/QlJRHe+fI2mapDGShrv7Gw28d4GkX0l6UNJFipYSnzKzgoQ2d0q6QtIPJF0q6RhJ/95An031oKS/KBr7GkmzJN0vqZ2iJcVZkv7bzL5ce4CZnSLpRUWB5ZuSvifpXEm/TOh3gaQzzew/zey4+t7czCZIekDR8t/58eMfmtm1SU2LFX0WP4rf6//q6XKWos/rvxQtU78l6RkzOz5+v55xm5fi1y+VNFtSp/j4Ckk1kh6JA2dDKwQFkn6jaJl4tKRtkp4zs64NHJPoWUn/HT8eGm/j6m8OICu5Oxtbi9gUnXPjdWw/S2hzRryvX/y8t6JfvN9OaJMjabGk5xOevy/pd4289yZJuZIeVRRS+ia1mSdpVh31Dk/Yd3y8b2T8/AuKzrWakNDGJC1RPFFTRy2zJM2rr8Y6PotbEvb1ife9lPR5VEqanLDvz5JeTup/eNJne7iiwFP7fVivKLSUJBxzuKRdiTXE+2+L3zM3fj497uP4pHZXxPsL4+dnxc9PT2r3iqQn4sfflLS5kZ+l8ZI+jvv6SNIcRcutdf28XZKwr1DSFkk/Tti3WtJdCc+nSypPeH5tfd9LNjY2NndnZgwtznZJJ8bbqZKuk/RtM7ulnvYnKgo3T9TucPea+HntzFippKP06VmfuuRKmqEo5Jzm7ktSqHefopBWq/a8stqrDftLaiPpmYT6XOk9Ef/FhMfvxl9fSni/GknvSTpaipZ1Fc3g/E/CLGSeotm1fZIGxcftUBSOTlY0S7VK0lhJC8zshLj7oYoutHgiqa+XJB2hTz4HSXrf3SsaGctXFIW4V5P6e1HS4LjNIkntzexX8ZLsYcmduPvdknpI+ldFn/VJ8Xh/VMd7PpVw3C5F58UNaaROAEgZYQwtTbW7l8fbq+5+r6Kr1/7DzDrV0f5ISbvcPflqvA2SCswsX9HslCR90Mh7F0g6R9Gs0ooU690Rhx1Jkrt/HD9sE3+tXe5KPkk9nSetb6vj/bcltfk4oaaOioLnFEXhq3bbq+jcvO4J/bm7/9Xdb3T3YYoCUY2k/4ybdI6/Lknq6+V4/9/7UvQ9aUxnRZ/ZvqTt1tq+3H25pK9JOk7SHyRtMrPHzOxTS87u/r67T3H3f1AUCudImmBmX0hotsvdP0qqYaOinysASAuupkQmeEdSa0k963jtA0mFZlaQFMiOkPShu+81s83xvsZ+we6UdLGi880+cPdJB1u4PjmJvEjR8pcSnoeyTdHy3K2Kwkyy9fUd6O4VZjZX0XKo9MmYzlfdYWt54uEp1LZF0ZLy1xtq5O7PKvo+tZd0nqR7FJ1IP6ae9rvNbIqi26Z8UVLtz0ShmbVNCmRd1HhwB4CUEcaQCWpv8LpWUlnSa28p+iX/TUm/lqIrIePnf4nbLFf0C/7bamR50N1fNLPRkn5nZjvd/Y6DrH2RoisHv6bo5PXa+i44yH4PWBxMXpdU6u631dfOzLq4+8akfaYoFNcGr78qOifrqDggHawXFV3csMvdlzXW2N23S3osvpJyaFxjJ0UzltVJzXvFXzcm7b9Q8W1T4qs6v6roAo5UfRwf28bd9zThOABZgjCGliYv4aq/1orOX7pJ0u/dvdLMPhXG3H2pmT0u6X4zO1zROVP/oii0XRO3qYlvtfBbM/utpMcVn3gv6XF3L0/q83/jqy9/a2Y73P2+Ax2Mu282s4ck/cDM9im6b9c/KTrxvUm3tkiziYru4Vaj6IKBnYqu8jxP0o3xMu3D8ZWlTyo6X6yjotoHKLryUO6+zaK7z//MotuEvKLo9IgSSWe6+4VNrGuupOclzTWzyYqWPw9XdGFEG3e/wcyuUhS85iiaxesV1/PruI/hkn5kZr9UFNZrFJ33NknSbHdPvIrzI0l3xCFsvaTvK/q5+1kTaq4NjdeZ2UuKguDyhg4AkF0IY2hp2iuabZGic4XWKLqC7/YGjvkXSZMVncfUQdFs1PnuXjszJnd/zMz2SLpRUfjYLel11XPulrvPiE8MnxbPkE0/iDFNVHQu1q2KgsGjkn6h6HYSQbj7X8zsNEW323hU0TlkaxQFnNpZrymKrna8WdES7zZF4ehsd38hoa87zWy9pH9TNKu1R9FtQ2YeQF1uZt+Q9B+KPp9jFC1dVii+n5ekv0kaJeluRbez+EDR7U1ujl9/Q9LvFd3CZGI8ttWKfoaSQ9aHki6P++6tKFid6+5NWab8s6SfKLrY5EeKAukZTTgeQIaz6MItAJ8nZvZHSa3c/fRGG6NZxDN617p758baAsDBYGYMCMzMzlR0a4UFimbILlZ0y4jRIesCABwahDEgvF2Krg68QdHtJVZKusLdZzV0EAAgM7BMCQAAEBA3fQUAAAioWZYpO3fu7MXFxc3RNQAAqMP8+fM3uXvIG0bjADVLGCsuLlZ5eXnjDQEAQFqY2ZrQNeDAsEwJAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABPS5/tuUxZOeDV0CAADNavWPzwtdAgJjZgwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgoM/1Hfi5KzEAAMh0zIwBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAvpc3/QVAACk1/z587vk5eU9LKmfmJQ5FGokLa6urh47aNCgjXU1IIwBAJBF8vLyHu7atWvvoqKirTk5OR66nkxXU1NjVVVVfSorKx+WNKquNiRiAACyS7+ioqIdBLFDIycnx4uKirYrmomsu80hrAcAAISXQxA7tOLPu97MRRgDAAAIiHPGAADIYtvu/v4A/2h32vKAtT2susP4uxbW93plZWXuGWecUSpJmzZtapWTk+OdOnWqlqSKioqlbdq0aXDWbvbs2e3y8/NrvvrVr+6WpDvvvLOooKCg5tprr92crjGkIrmOg0EYAwAgi6UziKXSX9euXfcvW7bsHUkaP378UYWFhftvu+22Dan2/9JLL7UrLCzcXxuCJk6cWHVwFR+Y5DoOBsuUAAAgqD//+c8FJ554Ymnfvn17n3rqqb3WrFnTSpJuv/32Lj179uxbUlLS5/zzzz9u+fLlrX/9618XTZ069YiysrI+c+bMKRw/fvxRN9988xGSNGTIkNJrrrnm6P79+/cuLi7uN2fOnEJJ2rlzZ8655557XElJSZ/zzjvvuC996Utlr7zySkFyHePGjTu69v2uvPLKbpK0fv36vLPPPrtnv379evfr16/3Cy+8cFhddRzM+JkZAwAAwbi7vvvd7x7z7LPPvnvUUUdVP/TQQx2///3vH/3EE0+svvfee7uuWbNmUdu2bX3Tpk25nTt33n/55ZdXJc6mvfDCC4cn9lddXW2LFi1aOnPmzPa33XbbUSNHjlzxk5/8pKhDhw77V6xY8c5bb73VZujQoX2T69iwYUPuH/7wh47vvffe4pycHG3atClXkq666qru48eP33D22WfvWrlyZeuzzz6713vvvbckuY6DQRgDAADB7N27N2flypVthw8fXiJJNTU1Kioq2idJpaWlH1144YU9Ro0ate3SSy/dlkp/o0eP3ipJJ5988u4JEya0lqTXXnut8LrrrtsoSSeeeOKekpKSD5OP69Sp0/78/PyaMWPGHHveeedtv/jii7dL0quvvnr4ypUr29a227VrV+7WrVvTurJIGAMAAMG4u774xS9+VFFRsSz5tZdffnnlc8891+7pp5/ucOeddx61cuXKxY31V3sBQF5envbv32+179GYVq1aqaKiYukzzzxz+IwZMzo+8MADXV5//fUV7q7y8vKlhYWFzXY7kJSSnZmtNrNFZlZhZuXNVQwAAMgu+fn5NVu2bMn74x//eJgk7d2718rLy9vs379fq1atan3BBRfsnDJlyrqdO3fmbt++Pbddu3b7d+7cmduU9zj55JN3zZgxo6MkzZ8/v82KFSvaJrfZvn17zpYtW3Ivvvji7VOnTl27dOnSAkk69dRTd0yePLlLbbvXXnutrSQdSB31aco025nufry7D07HGwMAgPCs7WHVIfvLycnRjBkzVk2aNKlbaWlpn759+/b505/+VFhdXW2XXHJJj5KSkj79+vXrc9VVV23o3Lnz/osuumjbs88+26EpJ85PmDChavPmzXklJSV97rjjjq6lpaUfdezYcX9im23btuWOHDmyV0lJSZ9hw4aV3n777Wsladq0aWsXLFhwWElJSZ+ePXv2vf/++4sk6UDqqI+lMnVnZqslDXb3Tal0OnjwYC8vZwINAIBDxczmpzJhsnDhwtUDBgxI6fd5pqiurtbHH39sBQUFvmTJkvwRI0aUrFq1anFj9zRLp4ULF3YeMGBAcV2vpXrOmEt6wcxc0oPuPi25gZldKelKSTrmmGMOsFQAAID02rlzZ86wYcNK9+3bZ+6un/70p2sOZRBrTKph7BR3X29mXSTNNbNl7v5KYoM4oE2TopmxNNcJAABwQDp27FizePHipaHrqE9K54y5+/r460ZJT0ka0pxFAQAAZItGw5iZHWZm7WofSxohqdFLSwEAANC4VJYpj5D0lJnVtn/M3ec0a1UAAABZotEw5u7vSRpwCGoBAADIOtyBHwCALPa1f54/YMeu6rTlgcML86p//8ighfW9XllZmXvGGWeUStKmTZta5eTkeKdOnaolqaKiYmlDVzm+8sorBY888sgXpk+fvrahGgYOHFj29ttvf+aO/s1t0qRJXX/84x9XNvU4whgAAFksnUEslf66du26f9myZe9I0vjx449K/mPb+/btU6tWreo89rTTTvvwtNNO+8zflUwWIohJ0r333nvkgYSxtP6hSwAAgKa66KKLiseOHdvtpJNOKhk3bly3l19+uWDgwIFlvXv37jNw4MCyhQsX5kvS7Nmz25155plflKIgN3r06OIhQ4aUduvWrf/tt9/+9z9ZVFBQMLC2/ZAhQ0pHjhx5XI8ePfqOGjWqR01NjSRp5syZ7Xv06NF30KBBpVdccUX32n4TlZeXt+nfv3/vsrKyPiUlJX0WLVqUL0lTpkzpVLv/kksuOba6ulrjxo07eu/evTllZWV9Ro0a1aMp42dmDAAABLdq1ao2r7766oq8vDxt2bIl580331zWqlUrPf300+0mTpzY7fnnn1+VfMy7777b5rXXXlu+bdu23N69e/ebMGFCVX5+/qeWOZcuXdq2oqLiveLi4n2DBg0qmzt3buGwYcN2X3fddcfOmzdvWVlZ2ccXXHBBneHpvvvuKxo3btyGa665ZsuePXusurpaCxYsaDNr1qxO5eXly/Lz8/2yyy47ZurUqV+YMmXK+9OnT+9SO+vXFIQxAAAQ3De+8Y2teXlRLIn/YHeP1atXtzEz37dvn9V1zIgRI7a1bdvW27ZtW92pU6d969aty+vZs+e+xDb9+/ffXbuvb9++H65atap1u3bt9nfv3n1vWVnZx5I0ZsyYLQ8//HBRcv9Dhw7dfddddx25bt261mPGjNnav3//vXPmzGm3ePHiggEDBvSWpD179uR06dLloP6+J2EMAAAEV1hYWFP7+Prrrz/69NNP3zl37txVy5cvbz18+PDSuo5JnAXLzc1VdXX1Z0JbXW1S+bvcknT11VdvGTZs2O6nnnqq/TnnnFMyZcqU1e5uo0eP3vzzn//8/SYNsAGcMwYAAD5XduzYkdutW7ePJenBBx/snO7+BwwYsGft2rX5y5cvby1JM2fO7FRXu3feead179699950000bR4wYsa2ioqLtyJEjd8yePbvj+++/nydJGzZsyF2xYkVrScrLy/O9e/fWOYvXEMIYAABZ7PDCvINaYmuO/q6//vrKW2+9tdsJJ5xQtn///nSU9SmFhYV+9913rxk5cmSvQYMGlXbp0mVfu3btPvNGjz76aKeSkpK+ZWVlfVauXNnmqquu2jxo0KA9N9100/tnnXVWSUlJSZ/hw4eXrF27tpUkXXrppVW9e/du8gn8KU/VNcXgwYO9vLw87f0CAIC6mdl8dx/cWLuFCxeuHjBgwKZDUdPn2fbt23Pat29fU1NTo8svv/yYXr167bnllls2Ntf7LVy4sPOAAQOK63qNmTEAAJB17rnnns5lZWV9evXq1XfHjh2548ePDxZQOYEfAABknVtuuWVjc86ENQUzYwAAZJeampqaJp9kjgMXf9419b1OGAMAILssrqqqak8gOzRqamqsqqqqvaTF9bVhmRIAgCxSXV09trKy8uHKysp+YlLmUKiRtLi6unpsfQ0IYwAAZJFBgwZtlDQqdB34BIkYAAAgIGbGAACSpKsnbw1dQpPcuKbeVZ8WpfuUJ0OXgMCYGQMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDvwAwAkSVOv7xi6hCbizvXIDMyMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQUMphzMxyzextM5vdnAUBAABkk6bMjF0naWlzFQIAAJCNUgpjZtZN0nmSHm7ecgAAALJLqjNj90iaKKmmvgZmdqWZlZtZeVVVVTpqAwAAyHiNhjEzO1/SRnef31A7d5/m7oPdfXBRUVHaCgQAAMhkqcyMnSJplJmtljRD0nAz+02zVgUAAJAlGg1j7n6Du3dz92JJYyS95O6XNXtlAAAAWYD7jAEAAASU15TG7j5P0rxmqQQAACALMTMGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABBQXugCAKAprp68NXQJTXLjmrGhSzho3ac8GboEIKMxMwYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEHfgB9CiTL2+Y+gSmoi71wNoGDNjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACajSMmVkbM3vTzBaa2RIz+8GhKAwAACAb5KXQZq+k4e6+y8xaSfqLmT3n7q83c20AAAAZr9Ew5u4uaVf8tFW8eXMWBQAAkC1SOmfMzHLNrELSRklz3f2NOtpcaWblZlZeVVWV5jIBAAAyU0phzN33u/vxkrpJGmJm/epoM83dB7v74KKiojSXCQAAkJmadDWlu2+TNE/SyOYoBgAAINukcjVlkZl1iB+3lfQVScuauS4AAICskMrVlEdK+pWZ5SoKb//j7rObtywAAIDskMrVlH+TNPAQ1AIAAJB1uAM/AABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAeaELADLR1ZO3hi4hZTeuGRu6hLToPuXJ0CUAwAFhZgwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIO7ADzSDqdd3DF1CE3DnegAIiZkxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABNRrGzKy7mb1sZkvNbImZXXcoCgMAAMgGeSm0qZb07+6+wMzaSZpvZnPd/Z1mrg0AACDjNToz5u4fuPuC+PFOSUslHd3chQEAAGSDJp0zZmbFkgZKeqOO1640s3IzK6+qqkpTeQAAAJkt5TBmZoWSnpT0PXffkfy6u09z98HuPrioqCidNQIAAGSslMKYmbVSFMR+6+6/a96SAAAAskcqV1OapF9IWurudzd/SQAAANkjlZmxUyT9o6ThZlYRb+c2c10AAABZodFbW7j7XyTZIagFAAAg63AHfgAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgPJCF4Bwrp68NXQJKbtxzdjQJaRF9ylPhi4BAPA5w8wYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQEDcgT+LTb2+Y+gSmoA71wMAMhMzYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICACGMAAAABEcYAAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAmo0jJnZI2a20cwWH4qCAAAAskkqM2PTJY1s5joAAACyUqNhzN1fkbTlENQCAACQddJ2zpiZXWlm5WZWXlVVla5uAQAAMlrawpi7T3P3we4+uKioKF3dAgAAZDSupgQAAAiIMAYAABBQKre2eFzSXyWVmtk6M/tO85cFAACQHfIaa+Du3zoUhQAAAGQjlikBAAACIowBAAAERBgDAAAIiDAGAAAQEGEMAAAgIMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICDCGAAAQECEMQAAgIAIYwAAAAERxgAAAALKC11AQ66evDV0CU1y45qxoUtIi+5TngxdAgAAWYOZMQAAgIAIYwAAAAERxgAAAAIijAEAAAREGAMAAAiIMAYAABAQYQwAACAgwhgAAEBAhDEAAICAPtd34J96fcfQJTQRd64HAABNw8wYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACIgwBgAAEBBhDAAAICBz9/R3alYlaU2au+0saVOa+2wpsnXsjDv7ZOvYGXf2aY6xH+vuRWnuE4dAs4Sx5mBm5e4+OHQdIWTr2Bl39snWsTPu7JPNY8dnsUwJAAAQEGEMAAAgoJYUxqaFLiCgbB07484+2Tp2xp19snnsSNJizhkDAADIRC1pZgwAACDjEMYAAAACahFhzMxGmtlyM3vXzCaFrudAmNkjZrbRzBYn7OtkZnPNbGX8tWPCazfE411uZmcn7B9kZovi1+41M4v355vZzHj/G2ZWfEgHWA8z625mL5vZUjNbYmbXxfszeuxm1sbM3jSzhfG4fxDvz+hx1zKzXDN728xmx8+zZdyr45orzKw83pfxYzezDmY2y8yWxf+tD830cZtZafx9rt12mNn3Mn3caCbu/rneJOVKWiXpOEmtJS2U1Cd0XQcwjtMknSBpccK+OyVNih9PkjQ5ftwnHme+pB7x+HPj196UNFSSSXpO0jnx/nGSpsaPx0iaGXrMcS1HSjohftxO0op4fBk99rjGwvhxK0lvSPpypo87YfzjJT0maXa2/KzH9ayW1DlpX8aPXdKvJI2NH7eW1CEbxp0w/lxJlZKOzaZxs6XxZyh0AY0WGP2APp/w/AZJN4Su6wDHUqxPh7Hlko6MHx8paXldY5T0fPw5HClpWcL+b0l6MLFN/DhP0Z2dLfSY6/gMfi/pq9k0dkkFkhZIOikbxi2pm6QXJQ3XJ2Es48cd17Nanw1jGT12SYdL+r/kOjJ93EljHSHp1WwbN1v6tpawTHm0pLUJz9fF+zLBEe7+gSTFX7vE++sb89Hx4+T9nzrG3aslbZf0hWar/ADEU+wDFc0SZfzY46W6CkkbJc1196wYt6R7JE2UVJOwLxvGLUku6QUzm29mV8b7Mn3sx0mqkvTLeGn6YTM7TJk/7kRjJD0eP86mcSNNWkIYszr2Zfr9OOobc0Ofxef6czKzQklPSvqeu+9oqGkd+1rk2N19v7sfr2imaIiZ9WugeUaM28zOl7TR3eenekgd+1rcuBOc4u4nSDpH0r+a2WkNtM2UsecpOgXjAXcfKGm3ouW5+mTKuCVJZtZa0ihJTzTWtI59LXbcSK+WEMbWSeqe8LybpPWBakm3DWZ2pCTFXzfG++sb87r4cfL+Tx1jZnmS2kva0myVN4GZtVIUxH7r7r+Ld2fF2CXJ3bdJmidppDJ/3KdIGmVmqyXNkDTczH6jzB+3JMnd18dfN0p6StIQZf7Y10laF8/8StIsReEs08dd6xxJC9x9Q/w8W8aNNGoJYewtSb3MrEf8L5Axkp4JXFO6PCPp2/Hjbys6n6p2/5j4SpoeknpJejOe8t5pZl+Or7a5POmY2r6+Kekldw/+L6i4zl9IWurudye8lNFjN7MiM+sQP24r6SuSlinDx+3uN7h7N3cvVvTf6kvufpkyfNySZGaHmVm72seKziNarAwfu7tXSlprZqXxrrMkvaMMH3eCb+mTJUope8aNdAp90loqm6RzFV2Ft0rSjaHrOcAxPC7pA0n7FP1r5zuK1v5flLQy/topof2N8XiXK76yJt4/WNH/4FdJul+f/BWFNoqmyd9VdGXOcaHHHNd1qqJp9b9Jqoi3czN97JK+JOnteNyLJd0c78/ocSd9BmfokxP4M37cis6dWhhvS2r/X5UlYz9eUnn88/60pI5ZMu4CSZsltU/Yl/HjZkv/xp9DAgAACKglLFMCAABkLMIYAABAQIQxAACAgAhjAAAAARHGAAAAAiKMAQAABEQYAwAACOj/ARzq9t4VnRCdAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from script.RNN.kfoldwindow import BlockingTimeSeriesSplit, plot_cv_indices\n",
    "tscv = BlockingTimeSeriesSplit(n_splits = 5, X=df)\n",
    "for train_index, test_index in tscv.split(X=df):\n",
    "    cv_train, cv_test = df.iloc[train_index], df.iloc[test_index]\n",
    "    print(cv_train.shape,cv_test.shape)\n",
    "plot_cv_indices(tscv, X=tscv.X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with both a Cross-Validation and a Holdout Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final, out of sample score (RMSE): 0.1659587323665619\n"
     ]
    }
   ],
   "source": [
    "from script.RNN.kfoldprocess import blockkfoldcv\n",
    "from script.RNN.myModel import mymodel\n",
    "mylstm = mymodel()\n",
    "score = blockkfoldcv(df = df, K=3,model=mylstm[0], withholdout=True,holdoutpct=0.1)\n",
    "IPython.display.clear_output()\n",
    "\n",
    "print(f\"Final, out of sample score (RMSE): {score}\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L1 and L2 Regularization to Decrease Overfitting\n",
    "- `kernel_regularizer`: Regularizer to apply a penalty on the layer's kernel\n",
    "- `bias_regularizer`: Regularizer to apply a penalty on the layer's bias\n",
    "- `activity_regularizer`: Regularizer to apply a penalty on the layer's output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "# model design with l2 regulizers\n",
    "lstm_model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.LSTM(units = 20, return_sequences=True,activity_regularizer=regularizers.l2(1e-5),\n",
    "                        activation='tanh', recurrent_activation='sigmoid'),#Recurrent layers\n",
    "    tf.keras.layers.Dense(units=1)#densely-connected NN layer: output with dim-1\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drop Out for Keras to Decrease Overfitting\n",
    "\n",
    "Dropout works by causing hidden neurons of the neural network to be unavailable during part of the training.  Dropping part of the neural network causes the remaining portion to be trained to still achieve a good score even without the dropped neurons.  This decreases coadaption between neurons, which results in less overfitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bootstrapping(other than cross-validation)\n",
    "\n",
    "Use Bootstrapping for hyperparameter benchmarking.  \n",
    "\n",
    "* Number of layers in the neural network\n",
    "* How many neurons in each layer\n",
    "* What activation functions to use on each layer\n",
    "* Dropout percent on each layer\n",
    "* L1 and L2 values on each layer\n",
    "\n",
    "\n",
    "To try out each of these **hyperparameters** you will need to run train neural networks with multiple settings for each hyperparameter.  However, you may have noticed that neural networks often produce somewhat different results when trained multiple times.  This is because the neural networks start with random weights.  Because of this it is necessary to fit and evaluate a neural network times to ensure that one set of hyperparameters are actually better than another.  **Bootstrapping** can be an effective means of benchmarking (comparing) two sets of hyperparameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "from sklearn.model_selection import StratifiedShuffleSplit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For time series we need block bootstrapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayesian Hyperparameter Optimization for Keras  \n",
    "\n",
    "Bayesian Hyperparameter Optimization is a method of finding hyperparameters in a more efficient way than a grid search.  Because each candidate set of hyperparameters requires a retraining of the neural network, it is best to keep the number of candidate sets to a minimum. Bayesian Hyperparameter Optimization achieves this by training a model to predict good candidate sets of hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bayes_opt import BayesianOptimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Update LSTM with new data\n",
    "- Reference: [https://machinelearningmastery.com/update-neural-network-models-with-more-data/](https://machinelearningmastery.com/update-neural-network-models-with-more-data/)\n",
    "- The problem of [*concept drift*](https://machinelearningmastery.com/gentle-introduction-concept-drift-machine-learning/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reload\n",
    "save_path = \"./trainedNN\"\n",
    "lstm_base= tf.keras.models.load_model(os.path.join(save_path,\"lstm-base.h5\"))\n",
    "IPython.display.clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from script.RNN.data_split import mysplit\n",
    "from script.RNN.myModel import myfittedmodel\n",
    "#train, validation sets\n",
    "train_df, val_df, test_df = mysplit(df= df, train_size=0.7,val_size=0.2)\n",
    "train_df, val_df, test_df = mysplit(df= test_df, train_size=0.7,val_size=0.2)\n",
    "\n",
    "train_mean = train_df.mean()\n",
    "train_std = train_df.std()\n",
    "train_df = (train_df - train_mean) / train_std\n",
    "val_df = (val_df - train_mean) / train_std\n",
    "test_df = (test_df - train_mean) / train_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_updated,lstm_updated = myfittedmodel(model = lstm_base, input_width=30, label_width=30, shift=1,\n",
    "                                                train_df=train_df, val_df=val_df, test_df=test_df,\n",
    "                                                label_columns=['Close'], \n",
    "                                                type='update')\n",
    "IPython.display.clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.013663934543728828, 0.08083733171224594, 0.013663934543728828]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm_updated.evaluate(x = window_updated.test,verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-11-20 02:34:01.515508: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n",
      "2021-11-20 02:34:01.543087: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.010519782081246376, 0.07164247334003448, 0.010519778355956078]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "save_path = \"./trainedNN\"\n",
    "lstm_base= tf.keras.models.load_model(os.path.join(save_path,\"lstm-base.h5\"))\n",
    "lstm_base.evaluate(x = window_updated.test,verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-step prediction\n",
    "\n",
    "predict all the features across all output time steps.  \n",
    "\n",
    "For the multi-step model, the training data again consists of hourly samples. However, here, the models will learn to predict 15 steps into the future, given 4 steps of the past."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "__init__() missing 3 required positional arguments: 'train_df', 'val_df', and 'test_df'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/_0/sgv1wmb92l11hndgfl0tyvjr0000gn/T/ipykernel_83673/2562374499.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mOUT_STEPS\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m multi_window = WindowGenerator(input_width=15,\n\u001b[0m\u001b[1;32m      5\u001b[0m                                \u001b[0mlabel_width\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mOUT_STEPS\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m                                shift=OUT_STEPS)\n",
      "\u001b[0;31mTypeError\u001b[0m: __init__() missing 3 required positional arguments: 'train_df', 'val_df', and 'test_df'"
     ]
    }
   ],
   "source": [
    "from script.RNN.window import WindowGenerator\n",
    "\n",
    "OUT_STEPS = 4\n",
    "multi_window = WindowGenerator(input_width=15,\n",
    "                               label_width=OUT_STEPS, shift=OUT_STEPS, train_df, val_df, test_df)\n",
    "\n",
    "multi_window.plot()\n",
    "multi_window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Baseline: repeat the last input time step for the required number of output time steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiStepLastBaseline(tf.keras.Model):\n",
    "  def call(self, inputs):\n",
    "    return tf.tile(inputs[:, -1:, :], [1, OUT_STEPS, 1])\n",
    "\n",
    "last_baseline = MultiStepLastBaseline()\n",
    "last_baseline.compile(loss=tf.losses.MeanSquaredError(),\n",
    "                      metrics=[tf.metrics.MeanAbsoluteError()])\n",
    "\n",
    "multi_val_performance = {}\n",
    "multi_performance = {}\n",
    "\n",
    "multi_val_performance['Last'] = last_baseline.evaluate(multi_window.val)\n",
    "multi_performance['Last'] = last_baseline.evaluate(multi_window.test, verbose=0)\n",
    "multi_window.plot(last_baseline)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c6e4e9f98eb68ad3b7c296f83d20e6de614cb42e90992a65aa266555a3137d0d"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}