beatles/key_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification of key based on chords\n",
    "\n",
    "The task is to classify each chord segment with a key.\n",
    "\n",
    "## Inputs  & outputs\n",
    "\n",
    "Each data point represents a chord segment.\n",
    "\n",
    "Inputs are chord segments with duration and a set of binary pitch classes contained in the chord. Thus the input dimension is 1+12 = 13.\n",
    "\n",
    "Output of each data point is the key within the segment represented by its canonical diatonic key root pitch class. So there are 12 possible class values. With one-hot encoding the output dimension is 12.\n",
    "\n",
    "\n",
    "## Assumptions\n",
    "\n",
    "- assumptions to simplify the problem:\n",
    "  - we assume that each segments has just a single chord and key\n",
    "  - we ignore chord segments with no chord (`N`) or no key (silence)\n",
    "  - we represent the key with its canonical diatonic key root pitch class\n",
    "    - ie. no distinction of modes (major, minor, dorian, etc.) or accidentals (sharp, flat)\n",
    "  - the inputs should be selected such that each input spans a single key, not multiple ones\n",
    "\n",
    "## The strategy of tasks - from simpler to harder \n",
    "\n",
    "First we can try to classify each chord segment independently to establish a baseline performance. In practice this is not too realistic, since the key may be distinguisehd by a set of multiple chords.\n",
    "\n",
    "Then we can try to use as inputs short sequences of adjacent chord segments of fixed size. This might be more realistic and be usable for non-recurrent ML models.\n",
    "\n",
    "Ideally we should handle the fact that the sequence of chords is of a variable length and use recurrent models and not be constrained by a fixed window.\n",
    "\n",
    "\n",
    "## ML algorithms\n",
    "\n",
    "As for the ML models we should start with a plain multi-class logistic regression.\n",
    "\n",
    "Then we could try deep neural networks:\n",
    "- architecture:\n",
    "  - input layer: 13\n",
    "  - hidden layer(s): ???\n",
    "  - output layer: 12\n",
    "    - soft-max classifer\n",
    "  - regularization - dropout, ...\n",
    "- pre-training with an autoencoder, then fine tuning with labels\n",
    "- experiment with:\n",
    "  - convolution layers (http://theanets.readthedocs.org/en/stable/reference.html#module-theanets.layers.convolution)\n",
    "    - eg. we would like to capture several distance metrics on the pitch class space:\n",
    "      - skip by 1 (plain pitch class neighborhood), skip by 7 (circle-of-fifths similarity)\n",
    "    - for pitch spaces we could also use skip by 12 (octave similarity)\n",
    "  - recurrent layers (http://theanets.readthedocs.org/en/stable/reference.html#module-theanets.layers.recurrent)\n",
    "    - we could use the Predict output layer\n",
    "\n",
    "## Evaluation\n",
    "\n",
    "TODO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import glob\n",
    "\n",
    "pylab.rcParams['figure.figsize'] = (16, 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>track_id</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>duration</th>\n",
       "      <th>label</th>\n",
       "      <th>root</th>\n",
       "      <th>bass</th>\n",
       "      <th>C</th>\n",
       "      <th>Db</th>\n",
       "      <th>D</th>\n",
       "      <th>...</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "      <th>Gb</th>\n",
       "      <th>G</th>\n",
       "      <th>Ab</th>\n",
       "      <th>A</th>\n",
       "      <th>Bb</th>\n",
       "      <th>B</th>\n",
       "      <th>key_diatonic_root_label</th>\n",
       "      <th>key_diatonic_root</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0    </th>\n",
       "      <td>   0</td>\n",
       "      <td>   0.000000</td>\n",
       "      <td>   2.612267</td>\n",
       "      <td>  2.612267</td>\n",
       "      <td>        N</td>\n",
       "      <td>  0</td>\n",
       "      <td>  0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1    </th>\n",
       "      <td>   0</td>\n",
       "      <td>   2.612267</td>\n",
       "      <td>  11.459070</td>\n",
       "      <td>  8.846803</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  11.459070</td>\n",
       "      <td>  12.921927</td>\n",
       "      <td>  1.462857</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  12.921927</td>\n",
       "      <td>  17.443474</td>\n",
       "      <td>  4.521547</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  17.443474</td>\n",
       "      <td>  20.410362</td>\n",
       "      <td>  2.966888</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  20.410362</td>\n",
       "      <td>  21.908049</td>\n",
       "      <td>  1.497687</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  21.908049</td>\n",
       "      <td>  23.370907</td>\n",
       "      <td>  1.462858</td>\n",
       "      <td>    E:7/3</td>\n",
       "      <td>  4</td>\n",
       "      <td>  8</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  23.370907</td>\n",
       "      <td>  24.856984</td>\n",
       "      <td>  1.486077</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  24.856984</td>\n",
       "      <td>  26.343061</td>\n",
       "      <td>  1.486077</td>\n",
       "      <td> A:min/b3</td>\n",
       "      <td>  9</td>\n",
       "      <td>  0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9    </th>\n",
       "      <td>   0</td>\n",
       "      <td>  26.343061</td>\n",
       "      <td>  27.840748</td>\n",
       "      <td>  1.497687</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  27.840748</td>\n",
       "      <td>  29.350045</td>\n",
       "      <td>  1.509297</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  29.350045</td>\n",
       "      <td>  35.305963</td>\n",
       "      <td>  5.955918</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  35.305963</td>\n",
       "      <td>  36.803650</td>\n",
       "      <td>  1.497687</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  36.803650</td>\n",
       "      <td>  41.263102</td>\n",
       "      <td>  4.459452</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  41.263102</td>\n",
       "      <td>  44.245646</td>\n",
       "      <td>  2.982544</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  44.245646</td>\n",
       "      <td>  45.720113</td>\n",
       "      <td>  1.474467</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  45.720113</td>\n",
       "      <td>  47.206190</td>\n",
       "      <td>  1.486077</td>\n",
       "      <td>    E:7/3</td>\n",
       "      <td>  4</td>\n",
       "      <td>  8</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  47.206190</td>\n",
       "      <td>  48.692267</td>\n",
       "      <td>  1.486077</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  48.692267</td>\n",
       "      <td>  50.155124</td>\n",
       "      <td>  1.462857</td>\n",
       "      <td> A:min/b3</td>\n",
       "      <td>  9</td>\n",
       "      <td>  0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  50.155124</td>\n",
       "      <td>  51.652811</td>\n",
       "      <td>  1.497687</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  51.652811</td>\n",
       "      <td>  53.138888</td>\n",
       "      <td>  1.486077</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  53.138888</td>\n",
       "      <td>  56.111043</td>\n",
       "      <td>  2.972155</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  56.111043</td>\n",
       "      <td>  65.131995</td>\n",
       "      <td>  9.020952</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  65.131995</td>\n",
       "      <td>  68.150589</td>\n",
       "      <td>  3.018594</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  68.150589</td>\n",
       "      <td>  71.192403</td>\n",
       "      <td>  3.041814</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  71.192403</td>\n",
       "      <td>  74.199387</td>\n",
       "      <td>  3.006984</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  74.199387</td>\n",
       "      <td>  75.697074</td>\n",
       "      <td>  1.497687</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  75.697074</td>\n",
       "      <td>  80.236575</td>\n",
       "      <td>  4.539501</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  80.236575</td>\n",
       "      <td>  83.208730</td>\n",
       "      <td>  2.972155</td>\n",
       "      <td>        B</td>\n",
       "      <td> 11</td>\n",
       "      <td> 11</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29   </th>\n",
       "      <td>   0</td>\n",
       "      <td>  83.208730</td>\n",
       "      <td>  86.221693</td>\n",
       "      <td>  3.012963</td>\n",
       "      <td>        E</td>\n",
       "      <td>  4</td>\n",
       "      <td>  4</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   E</td>\n",
       "      <td>  4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14591</th>\n",
       "      <td> 179</td>\n",
       "      <td> 120.053537</td>\n",
       "      <td> 122.004013</td>\n",
       "      <td>  1.950476</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14592</th>\n",
       "      <td> 179</td>\n",
       "      <td> 122.004013</td>\n",
       "      <td> 122.967641</td>\n",
       "      <td>  0.963628</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14593</th>\n",
       "      <td> 179</td>\n",
       "      <td> 122.967641</td>\n",
       "      <td> 123.942879</td>\n",
       "      <td>  0.975238</td>\n",
       "      <td>      G/2</td>\n",
       "      <td>  7</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14594</th>\n",
       "      <td> 179</td>\n",
       "      <td> 123.942879</td>\n",
       "      <td> 127.843832</td>\n",
       "      <td>  3.900953</td>\n",
       "      <td>      A:7</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14595</th>\n",
       "      <td> 179</td>\n",
       "      <td> 127.843832</td>\n",
       "      <td> 129.805918</td>\n",
       "      <td>  1.962086</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14596</th>\n",
       "      <td> 179</td>\n",
       "      <td> 129.805918</td>\n",
       "      <td> 130.792766</td>\n",
       "      <td>  0.986848</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14597</th>\n",
       "      <td> 179</td>\n",
       "      <td> 130.792766</td>\n",
       "      <td> 131.245555</td>\n",
       "      <td>  0.452789</td>\n",
       "      <td>      G/2</td>\n",
       "      <td>  7</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14598</th>\n",
       "      <td> 179</td>\n",
       "      <td> 131.245555</td>\n",
       "      <td> 131.756394</td>\n",
       "      <td>  0.510839</td>\n",
       "      <td>      D/5</td>\n",
       "      <td>  2</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14599</th>\n",
       "      <td> 179</td>\n",
       "      <td> 131.756394</td>\n",
       "      <td> 135.622517</td>\n",
       "      <td>  3.866123</td>\n",
       "      <td>      A:7</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14600</th>\n",
       "      <td> 179</td>\n",
       "      <td> 135.622517</td>\n",
       "      <td> 137.549773</td>\n",
       "      <td>  1.927256</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14601</th>\n",
       "      <td> 179</td>\n",
       "      <td> 137.549773</td>\n",
       "      <td> 139.477029</td>\n",
       "      <td>  1.927256</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14602</th>\n",
       "      <td> 179</td>\n",
       "      <td> 139.477029</td>\n",
       "      <td> 143.308321</td>\n",
       "      <td>  3.831292</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14603</th>\n",
       "      <td> 179</td>\n",
       "      <td> 143.308321</td>\n",
       "      <td> 145.258798</td>\n",
       "      <td>  1.950477</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14604</th>\n",
       "      <td> 179</td>\n",
       "      <td> 145.258798</td>\n",
       "      <td> 146.245646</td>\n",
       "      <td>  0.986848</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14605</th>\n",
       "      <td> 179</td>\n",
       "      <td> 146.245646</td>\n",
       "      <td> 146.733265</td>\n",
       "      <td>  0.487619</td>\n",
       "      <td>      G/2</td>\n",
       "      <td>  7</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14606</th>\n",
       "      <td> 179</td>\n",
       "      <td> 146.733265</td>\n",
       "      <td> 147.244104</td>\n",
       "      <td>  0.510839</td>\n",
       "      <td>      D/5</td>\n",
       "      <td>  2</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14607</th>\n",
       "      <td> 179</td>\n",
       "      <td> 147.244104</td>\n",
       "      <td> 151.110226</td>\n",
       "      <td>  3.866122</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14608</th>\n",
       "      <td> 179</td>\n",
       "      <td> 151.110226</td>\n",
       "      <td> 153.060702</td>\n",
       "      <td>  1.950476</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14609</th>\n",
       "      <td> 179</td>\n",
       "      <td> 153.060702</td>\n",
       "      <td> 154.024331</td>\n",
       "      <td>  0.963629</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14610</th>\n",
       "      <td> 179</td>\n",
       "      <td> 154.024331</td>\n",
       "      <td> 154.523560</td>\n",
       "      <td>  0.499229</td>\n",
       "      <td>      G/2</td>\n",
       "      <td>  7</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14611</th>\n",
       "      <td> 179</td>\n",
       "      <td> 154.523560</td>\n",
       "      <td> 155.022789</td>\n",
       "      <td>  0.499229</td>\n",
       "      <td>      D/5</td>\n",
       "      <td>  2</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14612</th>\n",
       "      <td> 179</td>\n",
       "      <td> 155.022789</td>\n",
       "      <td> 158.854081</td>\n",
       "      <td>  3.831292</td>\n",
       "      <td>      A:7</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14613</th>\n",
       "      <td> 179</td>\n",
       "      <td> 158.854081</td>\n",
       "      <td> 160.781337</td>\n",
       "      <td>  1.927256</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14614</th>\n",
       "      <td> 179</td>\n",
       "      <td> 160.781337</td>\n",
       "      <td> 161.489546</td>\n",
       "      <td>  0.708209</td>\n",
       "      <td>        A</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14615</th>\n",
       "      <td> 179</td>\n",
       "      <td> 161.489546</td>\n",
       "      <td> 162.174535</td>\n",
       "      <td>  0.684989</td>\n",
       "      <td>      G/2</td>\n",
       "      <td>  7</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14616</th>\n",
       "      <td> 179</td>\n",
       "      <td> 162.174535</td>\n",
       "      <td> 162.696984</td>\n",
       "      <td>  0.522449</td>\n",
       "      <td>      D/5</td>\n",
       "      <td>  2</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14617</th>\n",
       "      <td> 179</td>\n",
       "      <td> 162.696984</td>\n",
       "      <td> 166.423786</td>\n",
       "      <td>  3.726802</td>\n",
       "      <td>      A:7</td>\n",
       "      <td>  9</td>\n",
       "      <td>  9</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14618</th>\n",
       "      <td> 179</td>\n",
       "      <td> 166.423786</td>\n",
       "      <td> 168.192617</td>\n",
       "      <td>  1.768831</td>\n",
       "      <td>        D</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14619</th>\n",
       "      <td> 179</td>\n",
       "      <td> 168.192617</td>\n",
       "      <td> 174.281557</td>\n",
       "      <td>  6.088940</td>\n",
       "      <td>      D:7</td>\n",
       "      <td>  2</td>\n",
       "      <td>  2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14620</th>\n",
       "      <td> 179</td>\n",
       "      <td> 174.281557</td>\n",
       "      <td> 186.984490</td>\n",
       "      <td> 12.702933</td>\n",
       "      <td>        N</td>\n",
       "      <td>  0</td>\n",
       "      <td>  0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>...</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>   D</td>\n",
       "      <td>  2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14621 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       track_id       start         end   duration     label  root  bass  C  \\\n",
       "0             0    0.000000    2.612267   2.612267         N     0     0  0   \n",
       "1             0    2.612267   11.459070   8.846803         E     4     4  0   \n",
       "2             0   11.459070   12.921927   1.462857         A     9     9  0   \n",
       "3             0   12.921927   17.443474   4.521547         E     4     4  0   \n",
       "4             0   17.443474   20.410362   2.966888         B    11    11  0   \n",
       "5             0   20.410362   21.908049   1.497687         E     4     4  0   \n",
       "6             0   21.908049   23.370907   1.462858     E:7/3     4     8  0   \n",
       "7             0   23.370907   24.856984   1.486077         A     9     9  0   \n",
       "8             0   24.856984   26.343061   1.486077  A:min/b3     9     0  1   \n",
       "9             0   26.343061   27.840748   1.497687         E     4     4  0   \n",
       "10            0   27.840748   29.350045   1.509297         B    11    11  0   \n",
       "11            0   29.350045   35.305963   5.955918         E     4     4  0   \n",
       "12            0   35.305963   36.803650   1.497687         A     9     9  0   \n",
       "13            0   36.803650   41.263102   4.459452         E     4     4  0   \n",
       "14            0   41.263102   44.245646   2.982544         B    11    11  0   \n",
       "15            0   44.245646   45.720113   1.474467         E     4     4  0   \n",
       "16            0   45.720113   47.206190   1.486077     E:7/3     4     8  0   \n",
       "17            0   47.206190   48.692267   1.486077         A     9     9  0   \n",
       "18            0   48.692267   50.155124   1.462857  A:min/b3     9     0  1   \n",
       "19            0   50.155124   51.652811   1.497687         E     4     4  0   \n",
       "20            0   51.652811   53.138888   1.486077         B    11    11  0   \n",
       "21            0   53.138888   56.111043   2.972155         E     4     4  0   \n",
       "22            0   56.111043   65.131995   9.020952         A     9     9  0   \n",
       "23            0   65.131995   68.150589   3.018594         B    11    11  0   \n",
       "24            0   68.150589   71.192403   3.041814         A     9     9  0   \n",
       "25            0   71.192403   74.199387   3.006984         E     4     4  0   \n",
       "26            0   74.199387   75.697074   1.497687         A     9     9  0   \n",
       "27            0   75.697074   80.236575   4.539501         E     4     4  0   \n",
       "28            0   80.236575   83.208730   2.972155         B    11    11  0   \n",
       "29            0   83.208730   86.221693   3.012963         E     4     4  0   \n",
       "...         ...         ...         ...        ...       ...   ...   ... ..   \n",
       "14591       179  120.053537  122.004013   1.950476         D     2     2  0   \n",
       "14592       179  122.004013  122.967641   0.963628         A     9     9  0   \n",
       "14593       179  122.967641  123.942879   0.975238       G/2     7     9  0   \n",
       "14594       179  123.942879  127.843832   3.900953       A:7     9     9  0   \n",
       "14595       179  127.843832  129.805918   1.962086         D     2     2  0   \n",
       "14596       179  129.805918  130.792766   0.986848         A     9     9  0   \n",
       "14597       179  130.792766  131.245555   0.452789       G/2     7     9  0   \n",
       "14598       179  131.245555  131.756394   0.510839       D/5     2     9  0   \n",
       "14599       179  131.756394  135.622517   3.866123       A:7     9     9  0   \n",
       "14600       179  135.622517  137.549773   1.927256         D     2     2  0   \n",
       "14601       179  137.549773  139.477029   1.927256         A     9     9  0   \n",
       "14602       179  139.477029  143.308321   3.831292         A     9     9  0   \n",
       "14603       179  143.308321  145.258798   1.950477         D     2     2  0   \n",
       "14604       179  145.258798  146.245646   0.986848         A     9     9  0   \n",
       "14605       179  146.245646  146.733265   0.487619       G/2     7     9  0   \n",
       "14606       179  146.733265  147.244104   0.510839       D/5     2     9  0   \n",
       "14607       179  147.244104  151.110226   3.866122         A     9     9  0   \n",
       "14608       179  151.110226  153.060702   1.950476         D     2     2  0   \n",
       "14609       179  153.060702  154.024331   0.963629         A     9     9  0   \n",
       "14610       179  154.024331  154.523560   0.499229       G/2     7     9  0   \n",
       "14611       179  154.523560  155.022789   0.499229       D/5     2     9  0   \n",
       "14612       179  155.022789  158.854081   3.831292       A:7     9     9  0   \n",
       "14613       179  158.854081  160.781337   1.927256         D     2     2  0   \n",
       "14614       179  160.781337  161.489546   0.708209         A     9     9  0   \n",
       "14615       179  161.489546  162.174535   0.684989       G/2     7     9  0   \n",
       "14616       179  162.174535  162.696984   0.522449       D/5     2     9  0   \n",
       "14617       179  162.696984  166.423786   3.726802       A:7     9     9  0   \n",
       "14618       179  166.423786  168.192617   1.768831         D     2     2  0   \n",
       "14619       179  168.192617  174.281557   6.088940       D:7     2     2  1   \n",
       "14620       179  174.281557  186.984490  12.702933         N     0     0  0   \n",
       "\n",
       "       Db  D        ...         E  F  Gb  G  Ab  A  Bb  B  \\\n",
       "0       0  0        ...         0  0   0  0   0  0   0  0   \n",
       "1       0  0        ...         1  0   0  0   1  0   0  1   \n",
       "2       1  0        ...         1  0   0  0   0  1   0  0   \n",
       "3       0  0        ...         1  0   0  0   1  0   0  1   \n",
       "4       0  0        ...         0  0   1  0   0  0   0  1   \n",
       "5       0  0        ...         1  0   0  0   1  0   0  1   \n",
       "6       0  1        ...         1  0   0  0   1  0   0  1   \n",
       "7       1  0        ...         1  0   0  0   0  1   0  0   \n",
       "8       0  0        ...         1  0   0  0   0  1   0  0   \n",
       "9       0  0        ...         1  0   0  0   1  0   0  1   \n",
       "10      0  0        ...         0  0   1  0   0  0   0  1   \n",
       "11      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "12      1  0        ...         1  0   0  0   0  1   0  0   \n",
       "13      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "14      0  0        ...         0  0   1  0   0  0   0  1   \n",
       "15      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "16      0  1        ...         1  0   0  0   1  0   0  1   \n",
       "17      1  0        ...         1  0   0  0   0  1   0  0   \n",
       "18      0  0        ...         1  0   0  0   0  1   0  0   \n",
       "19      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "20      0  0        ...         0  0   1  0   0  0   0  1   \n",
       "21      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "22      1  0        ...         1  0   0  0   0  1   0  0   \n",
       "23      0  0        ...         0  0   1  0   0  0   0  1   \n",
       "24      1  0        ...         1  0   0  0   0  1   0  0   \n",
       "25      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "26      1  0        ...         1  0   0  0   0  1   0  0   \n",
       "27      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "28      0  0        ...         0  0   1  0   0  0   0  1   \n",
       "29      0  0        ...         1  0   0  0   1  0   0  1   \n",
       "...    .. ..        ...        .. ..  .. ..  .. ..  .. ..   \n",
       "14591   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14592   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14593   0  1        ...         0  0   0  1   0  1   0  1   \n",
       "14594   1  0        ...         1  0   0  1   0  1   0  0   \n",
       "14595   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14596   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14597   0  1        ...         0  0   0  1   0  1   0  1   \n",
       "14598   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14599   1  0        ...         1  0   0  1   0  1   0  0   \n",
       "14600   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14601   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14602   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14603   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14604   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14605   0  1        ...         0  0   0  1   0  1   0  1   \n",
       "14606   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14607   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14608   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14609   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14610   0  1        ...         0  0   0  1   0  1   0  1   \n",
       "14611   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14612   1  0        ...         1  0   0  1   0  1   0  0   \n",
       "14613   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14614   1  0        ...         1  0   0  0   0  1   0  0   \n",
       "14615   0  1        ...         0  0   0  1   0  1   0  1   \n",
       "14616   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14617   1  0        ...         1  0   0  1   0  1   0  0   \n",
       "14618   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14619   0  1        ...         0  0   1  0   0  1   0  0   \n",
       "14620   0  0        ...         0  0   0  0   0  0   0  0   \n",
       "\n",
       "       key_diatonic_root_label key_diatonic_root  \n",
       "0                          NaN               NaN  \n",
       "1                            E                 4  \n",
       "2                            E                 4  \n",
       "3                            E                 4  \n",
       "4                            E                 4  \n",
       "5                            E                 4  \n",
       "6                            E                 4  \n",
       "7                            E                 4  \n",
       "8                            E                 4  \n",
       "9                            E                 4  \n",
       "10                           E                 4  \n",
       "11                           E                 4  \n",
       "12                           E                 4  \n",
       "13                           E                 4  \n",
       "14                           E                 4  \n",
       "15                           E                 4  \n",
       "16                           E                 4  \n",
       "17                           E                 4  \n",
       "18                           E                 4  \n",
       "19                           E                 4  \n",
       "20                           E                 4  \n",
       "21                           E                 4  \n",
       "22                           E                 4  \n",
       "23                           E                 4  \n",
       "24                           E                 4  \n",
       "25                           E                 4  \n",
       "26                           E                 4  \n",
       "27                           E                 4  \n",
       "28                           E                 4  \n",
       "29                           E                 4  \n",
       "...                        ...               ...  \n",
       "14591                        D                 2  \n",
       "14592                        D                 2  \n",
       "14593                        D                 2  \n",
       "14594                        D                 2  \n",
       "14595                        D                 2  \n",
       "14596                        D                 2  \n",
       "14597                        D                 2  \n",
       "14598                        D                 2  \n",
       "14599                        D                 2  \n",
       "14600                        D                 2  \n",
       "14601                        D                 2  \n",
       "14602                        D                 2  \n",
       "14603                        D                 2  \n",
       "14604                        D                 2  \n",
       "14605                        D                 2  \n",
       "14606                        D                 2  \n",
       "14607                        D                 2  \n",
       "14608                        D                 2  \n",
       "14609                        D                 2  \n",
       "14610                        D                 2  \n",
       "14611                        D                 2  \n",
       "14612                        D                 2  \n",
       "14613                        D                 2  \n",
       "14614                        D                 2  \n",
       "14615                        D                 2  \n",
       "14616                        D                 2  \n",
       "14617                        D                 2  \n",
       "14618                        D                 2  \n",
       "14619                        D                 2  \n",
       "14620                        D                 2  \n",
       "\n",
       "[14621 rows x 21 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = pd.read_csv('data/beatles/derived/all_chords_with_keys.tsv', sep='\\t')\n",
    "dataset = dataset.dropna()\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple silly classification of each segment alone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pcs_columns = ['C','Db','D','Eb','E','F','Gb','G','Ab','A','Bb','B']\n",
    "# note: we could also try use features like 'root' or 'bass'\n",
    "input_cols = pcs_columns\n",
    "output_col = 'key_diatonic_root'\n",
    "\n",
    "def select_X_y(df):\n",
    "    X = df[input_cols].astype(np.int32)\n",
    "    y = df[output_col].astype(np.int32)\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = select_X_y(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14441"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>Db</th>\n",
       "      <th>D</th>\n",
       "      <th>Eb</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "      <th>Gb</th>\n",
       "      <th>G</th>\n",
       "      <th>Ab</th>\n",
       "      <th>A</th>\n",
       "      <th>Bb</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   C  Db  D  Eb  E  F  Gb  G  Ab  A  Bb  B\n",
       "1  0   0  0   0  1  0   0  0   1  0   0  1\n",
       "2  0   1  0   0  1  0   0  0   0  1   0  0\n",
       "3  0   0  0   0  1  0   0  0   1  0   0  1\n",
       "4  0   0  0   1  0  0   1  0   0  0   0  1\n",
       "5  0   0  0   0  1  0   0  0   1  0   0  1"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key_diatonic_root</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   key_diatonic_root\n",
       "1                  4\n",
       "2                  4\n",
       "3                  4\n",
       "4                  4\n",
       "5                  4"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "def split_dataset(X, y, test_percentage=0.2, valid_percentage=0.2, random_state=42):\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_percentage, random_state=random_state)\n",
    "    X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=valid_percentage/(1-test_percentage), random_state=random_state)\n",
    "    return (X_train, y_train), (X_valid, y_valid), (X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "((X_train, y_train), (X_valid, y_valid), (X_test, y_test)) = split_dataset(X, y)\n",
    "\n",
    "[d.shape for d in (X_train, y_train, X_valid, y_valid, X_test, y_test)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model = LogisticRegression()\n",
    "lr_model.fit(X_train, y_train.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "def accuracy_report(model, X_train, y_train, X_valid, y_valid):\n",
    "    # mean accuracy on the given test data and labels\n",
    "    accuracy_train, accuracy_valid = model.score(X_train, y_train), model.score(X_valid, y_valid)\n",
    "    print('training set accuracy (%):', accuracy_train)\n",
    "    print('validation set accuracy (%):', accuracy_valid)\n",
    "    print('difference (% points):', accuracy_train - accuracy_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "accuracy_report(lr_model, X_train, y_train, X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: we could better use http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "random_batches = 100\n",
    "y_random_uniform = np.random.random_integers(low=0, high=11, size=(len(X_valid), random_batches))\n",
    "y_pred_lr_valid = lr_model.predict(X_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     0.153393\n",
       "1     0.005194\n",
       "2     0.135042\n",
       "3     0.029778\n",
       "4     0.145776\n",
       "5     0.066136\n",
       "6     0.016620\n",
       "7     0.200485\n",
       "8     0.013850\n",
       "9     0.176939\n",
       "10    0.016620\n",
       "11    0.040166\n",
       "dtype: float64"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def prior_class_probabilities(y):\n",
    "    priors = pd.Series(np.zeros(12))\n",
    "    hist = (pd.Series(y.ravel()).value_counts() / len(y))\n",
    "    priors[hist.index] = hist\n",
    "    return priors\n",
    "\n",
    "priors_valid = prior_class_probabilities(y_valid)\n",
    "priors_valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.008310\n",
       "7    0.006925\n",
       "2    0.005194\n",
       "9    0.004155\n",
       "4    0.003463\n",
       "5    0.002770\n",
       "3    0.002770\n",
       "8    0.000693\n",
       "1    0.000346\n",
       "dtype: float64"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# random samples of classes based on prior probabilities of each class\n",
    "y_random_prior = numpy.random.choice(np.arange(12), size=(len(y_valid), random_batches), p=priors_valid)\n",
    "pd.Series(y_random_prior[0]).value_counts() / len(y_random_prior)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "validation set, mean accuracy of uniform random guess (%): 8.36842105263\n",
      "validation set, mean accuracy of random guess with prior class probabilities (%): 14.1371191136\n",
      "validation set, accuracy of LR model (%): 42.3476454294\n",
      "the model is better than uniform random guess 5.06041046011 times\n",
      "the model is better than prior random guess 2.99549328892 times\n"
     ]
    }
   ],
   "source": [
    "mean_random_uniform_accuracy = ((y_random_uniform == np.outer(np.ones(y_random_uniform.shape[1]), y_valid).T).sum(axis=0) / len(X_valid)).mean() * 100\n",
    "print('validation set, mean accuracy of uniform random guess (%):', mean_random_uniform_accuracy)\n",
    "mean_random_prior_accuracy = ((y_random_prior == np.outer(np.ones(y_random_prior.shape[1]), y_valid).T).sum(axis=0) / len(X_valid)).mean() * 100\n",
    "print('validation set, mean accuracy of random guess with prior class probabilities (%):', mean_random_prior_accuracy)\n",
    "\n",
    "lr_accuracy = sum(y_pred_lr_valid == y_valid.ravel()) / len(X_valid) * 100\n",
    "print('validation set, accuracy of LR model (%):', lr_accuracy)\n",
    "\n",
    "print('the model is better than uniform random guess', lr_accuracy / mean_random_uniform_accuracy, 'times')\n",
    "print('the model is better than prior random guess', lr_accuracy / mean_random_prior_accuracy, 'times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the seemingly good performance of the simple model is caused by the model trained on the skewed class prior probabilities. When we compare the model to random guessing based on the priors, the model is just 3x better than this kind of random guessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.57      0.47      0.52       535\n",
      "          1       0.00      0.00      0.00         0\n",
      "          2       0.37      0.34      0.36       427\n",
      "          3       0.45      0.34      0.39       114\n",
      "          4       0.55      0.47      0.50       491\n",
      "          5       0.29      0.42      0.34       130\n",
      "          6       0.44      0.39      0.41        54\n",
      "          7       0.42      0.46      0.44       532\n",
      "          8       0.25      0.26      0.25        39\n",
      "          9       0.41      0.41      0.41       507\n",
      "         10       0.25      0.29      0.27        41\n",
      "         11       0.03      0.22      0.06        18\n",
      "\n",
      "avg / total       0.45      0.42      0.43      2888\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/bzamecnik/Documents/dev/pyvenv/py3.4/lib/python3.4/site-packages/sklearn/metrics/metrics.py:1773: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.\n",
      "  'recall', 'true', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_pred_lr_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes\n",
    "\n",
    "- an accuracy of pure guess should be theoretically 1/12 ~= 8.333 %, a few rounds of random samples give a very close number\n",
    "- even with this simple ML method and context of a single chord the accuracy both both sets is not to bad (~ 5x better than random guess)\n",
    "- the training accuracy is just slightly better than the validation one\n",
    "  - it means the model is very biased and is not overfitted, more data would likely not help\n",
    "- the classes are very skewed, so we could generate synthetic data points in order to make classes uniform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the actual results of classification on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def classification_results(X, y_true, y_pred):\n",
    "    return pd.DataFrame(\n",
    "        np.hstack([X, y_true.reshape(-1, 1), y_pred.reshape(-1, 1)]),\n",
    "        columns=input_cols+['label_true']+['label_pred'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy on validation set (%) 42.3476454294\n"
     ]
    }
   ],
   "source": [
    "classification_valid = classification_results(X_valid, y_valid, y_pred_lr_valid)\n",
    "\n",
    "classification_valid['is_correct'] = classification_valid['label_true'] == classification_valid['label_pred']\n",
    "\n",
    "print('accuracy on validation set (%)', sum(classification_valid['is_correct']) / len(classification_valid) * 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `score()` from the `LogisticRegression` model seems to really corresponds to manually computed accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def key_classification_error_report(y_true, y_pred):\n",
    "    errors = pd.Series(((y_true - y_pred) + 12) % 12)\n",
    "    errors_hist = errors.value_counts()\n",
    "    stem(errors_hist / len(errors) * 100)\n",
    "    gca().set_xticks(np.arange(len(errors_hist)))\n",
    "    gca().set_xticklabels(errors_hist.index)\n",
    "    xlabel('error (true - predicted key pitch class)')\n",
    "    ylabel('%')\n",
    "    return errors, errors_hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X+wpFd93/nPMz8kiCy0qSwmFqOMKkU4ldgxtqQNNmEl\nQJJtjB2NXU5lsxhbKhvQQhlSa5Y4MgV27MQmMkqMjQklDGITNl6skuWAwy8JLGwqAYR/LSE5wsYj\npEHlZR0QpQlI8+PZP7olXQ1XM2c809PfO/N6VVF09723n3O/M7oz7z5PPzPN8xwAAACoZNu6FwAA\nAABHEqsAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOTtWfYDW2tcn+WSSy5Ock+Q9Se5afvjN\nvfd3rXoNAAAAbC0rjdXW2s4kb0myP8mU5OIkb+i937DK4wIAALC1rfo04OuTvDnJfcv7FyV5QWvt\njtbaW1trX7fi4wMAALAFrSxWW2tXJ/lC7/0DGx7+eJJX9d4vS/LZJK9b1fEBAADYulZ5GvA1SebW\n2hVJviXJO5Jc1Xv/s+XHb03yxoHn+WqSs1ezRAAAANZs2uzBlcXqcvc0SdJa+3CSa5Pc2lp7Re/9\nE1lccOnOgac6O4+zeB5jjjmNMqsx5jTGnMaZ1RhzGmdWY8xpjDmNM6sx5nSCVn414A3mLIL1Ta21\nA1m8j/Ulp/D4AAAAbBHTPM/rXsOxeEVijDmNM6sx5jTGnMaZ1RhzGmdWY8xpjDmNM6sx5nSCVn01\nYAAAADhuYhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIV\nAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAA\nAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADK\nEasAAACUI1YBAAAoR6wCAABQjlgFAACgnPKxOs/zupcAAADAKVY+VrdtmzNND902TdO07rUAAABw\napSP1cUSd16eHLhnmu6/ZN2rAQAAYPWm6qfZTlM2LPDgvmTnBXP1Ra/HnMTu8xizGmNOY8xpnFmN\nMadxZjXGnMaY0zizGmNOJ2gL7KwCAABwptmx7gWMmZMc2pfs32NXFQAA4PS3RWL1kNN/AQAAziDl\nTwPevn2OHVUAAIAzS/lY3bUrmefz7lz3OgAAADh1yseqC2gBAACcebZArAIAAHCmEasAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAA\nUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAc\nsQoAAEA5YhUAAIByxCoAAADliFUAAADK2bHqA7TWvj7JJ5NcnuRwkpuW//+pJC/vvc+rXgMAAABb\ny0p3VltrO5O8Jcn+JFOSG5Jc13u/dHn/qlUeHwAAgK1p1acBX5/kzUnuW96/qPf+keXt9ya5YsXH\nBwAAYAtaWay21q5O8oXe+weWD03L/z3sgSTnrer4AAAAbF2rfM/qNUnm1toVSb4lyTuSPHnDx89N\n8qXB5/K+1jHmNM6sxpjTGHMaZ1ZjzGmcWY0xpzHmNM6sxpjTmGmzB1cWq733yx6+3Vr7cJJrk1zf\nWrus935HkucnuX3w6TZdPI8xx5xGmdUYcxpjTuPMaow5jTOrMeY0xpzGmdUYczpBK78a8AZzkh9P\ncmNr7awkn05y8yk8PgAAAFvENM+1d6YvvDDz3r1ekRjglZtxZjXGnMaY0zizGmNO48xqjDmNMadx\nZjXGnE7Qqq8GDAAAAMdNrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACU\nI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEes\nAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUA\nAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABA\nOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUA\nAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAA\nlCNWAQAAKEesAgAAUM6OVT55a217khuTPD3JnOTaJGcleU+Su5af9ube+7tWuQ4AAAC2lpXGapLv\nSXK49/7s1tplSf5ZkncneUPv/YYVHxsAAIAtaqWnAffefzPJS5d3L0zypSQXJ3lBa+2O1tpbW2tf\nt8o1AAAAsPWs/D2rvfdDrbWbkvxikncm+XiSV/XeL0vy2SSvW/UaAAAA2FpWfRpwkqT3fnVr7SlJ\nPpbkWb33zy8/dGuSNw48xbyyxZ1ezGmcWY0xpzHmNM6sxpjTOLMaY05jzGmcWY0xpzHTZg+u+gJL\nL0qyq/f+c0m+kuRwkltaaz/We/9EksuT3DnwVJsunseYY06jzGqMOY0xp3FmNcacxpnVGHMaY07j\nzGqMOZ2gVe+s3pzkptbaHUl2Jnllks8leVNr7UCS+5K8ZMVrAAAAYIuZ5rn2zvSFF2beu9crEgO8\ncjPOrMaY0xhzGmdWY8xpnFmNMacx5jTOrMaY0wla+QWWAAAA4HiJVQAAAMoRqwAAAJQjVgEAAChH\nrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgF\nAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAA\nQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIBy\nxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhV\nAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAA\nAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAo\nR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAytmxyidvrW1PcmOSpyeZk1yb5MEk\nNyU5nORTSV7ee59XuQ4AAAC2llXvrH5PksO992cneU2Sf57kDUmu671fmmRKctWK1wAAAMAWs9JY\n7b3/ZpKXLu9emOSLSS7uvX9k+dh7k1yxyjUAAACw9az8Pau990OttZuS/GKSd2axm/qwB5Kct+o1\nAAAAsLWs9D2rD+u9X91ae0qSjyd5woYPnZvkSwNP4T2tY8xpnFmNMacx5jTOrMaY0zizGmNOY8xp\nnFmNMacx02YPrvoCSy9Ksqv3/nNJvpLkUJI7W2uX9d7vSPL8JLcPPNWmi+cx5pjTKLMaY05jzGmc\nWY0xp3FmNcacxpjTOLMaY04naNU7qzcnuam1dkeSnUlemeS/JrmxtXZWkk8vPwcAAAAeMc1z7Z3p\nCy/MvHevVyQGeOVmnFmNMacx5jTOrMaY0zizGmNOY8xpnFmNMacTtPILLAEAAMDxEqsAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAA\nUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAc\nsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIV\nAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAA\nAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADK\nEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNW\nAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChnx6qe\nuLW2M8nbkuxOcnaSn01yb5L3JLlr+Wlv7r2/a1VrAAAAYGtaWawmeWGSL/TeX9Ra+8tJ/jDJTyd5\nQ+/9hhUeFwAAgC1ulbH660luXt7eluRAkouTtNbaVUk+k+Qf9d4fWOEaAAAA2IJW9p7V3vv+3vsD\nrbVzswjXn0zy8SSv6r1fluSzSV63quMDAACwda1yZzWttQuS3JLkTb33X2utndd7v3/54VuTvHHw\nqeaVLPD0Y07jzGqMOY0xp3FmNcacxpnVGHMaY07jzGqMOY2ZNntwlRdYekqSDyR5We/9w8uH39da\ne0Xv/RNJLk9y5+DTbbp4HmOOOY0yqzHmNMacxpnVGHMaZ1ZjzGmMOY0zqzHmdIJWubN6XZLzkry2\ntfba5WP/KMm/bK0dSHJfkpes8PgAAABsUdM8196ZvvDCzHv3ekVigFduxpnVGHMaY07jzGqMOY0z\nqzHmNMacxpnVGHM6QSu7wBIAAAD8RYlVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAA\nQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIBy\nxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhV\nAAAAyhGrAAAAlCNWAQAAKEesAgAAUM5xxWpr7YmttXNXtRgAAABIjiNWW2s/kuQ/JfmPrbWfWd2S\nAAAAONM9bqy21r7piIf29N6f0Xv/piTft9plAQAAcCbbcZSPvbS1dlaSf9p735fkD1pr709yIMmn\nTsnqAAAAOCNN8zw/7gdba09P8tokn0vy+iR/NcnZvfc/OjXLSy68MPPevZlO1fG2sDkxp0FmNcac\nxpjTOLMaY07jzGqMOY0xp3FmNcacTtBR37Pae7+r9/6DSd6T5N8k+e4k/+VULAwAAIAz19Hes/qy\n1tqftNbuSnJ+7/3vJbk7yXtaay88ZSsEAADgjHO0ndWXJWlJvjXJdUnSe78lyQuSPGn1SwMAAOBM\ndbQLLN2X5F8leWI2nPrbez+Y5M0rXhcAAABnsKPF6vcm+c4kDyb54KlZDgAAABzjasAVuBrwMFcb\nG2dWY8xpjDmNM6sx5jTOrMaY0xhzGmdWY8zpBB31asAAAACwDmIVAACAcsQqAAAA5YhVAAAAyhGr\nAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEA\nAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQ\njlgFAACgHLEKAABAOWIVAACAcnas6olbazuTvC3J7iRnJ/nZJP8lyU1JDif5VJKX997nVa0BAACA\nrWmVO6svTPKF3vulSb4ryZuSvCHJdcvHpiRXrfD4AAAAbFGrjNVfT/LaDcc5kOSi3vtHlo+9N8kV\nKzw+AAAAW9TKTgPuve9PktbauVmE62uS/MKGT3kgyXmrOj4AAABb18piNUlaaxckuSXJm3rv/661\n9i82fPjcJF8afCrvax1jTuPMaow5jTGncWY1xpzGmdUYcxpjTuPMaow5jZk2e3CVF1h6SpIPJHlZ\n7/3Dy4d/v7V2We/9jiTPT3L74NNtungeY445jTKrMeY0xpzGmdUYcxpnVmPMaYw5jTOrMeZ0gla5\ns3pdFqf5vra19vB7V1+Z5I2ttbOSfDrJzSs8PgAAAFvUNM+1d6YvvDDz3r1ekRjglZtxZjXGnMaY\n0zizGmNO48xqjDmNMadxZjXGnE7QKq8GDAAAAH8hYhUAAIByxCoAAADliFUAAADKEasAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAA\nUI5YBQAAoByxCgAAQDliFQAAgHLE6mlgmqZpnud1LwMAAOCkEatb2DRN0zQ9dFty8NC2bXOm6aHb\npmma1r0uAACAEyVWt7QHP5jsvDzZPi1+KXdenhy4Z5ruv2TdKwMAADgRYnWLWuygbn9esnEjdUqy\n46nJObfaYQUAALYysQoAAEA5YnWLmud5Tg59KNl4YaU5ycF9yf49sysuAQAAW9iOdS+AE3H2lcmB\nexan/ibJoX3JzguEKgAAsNXZWd3CFlG6f08yH9q+fY4dVQAA4HQhVre4eT7vziT37tr1yG0AAIAt\nz2nApwUX/gUAAE4vdlYBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGr\nAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVc4o0zRN8zyvexkAAMAx\niFXOCNM0TdP00G3JwUPbts2Zpodum6ZpWve6AACAzYlVzhAPfjDZeXmyfVr8tt95eXLgnmm6/5J1\nrwwAAPhaYpXT3mIHdfvzko0bqVOSHU9NzrnVDisAANQjVgEAAChHrHLam+d5Tg59KNl4YaU5ycF9\nyf49sysuAQBAOTvWvQA4Nc6+Mjlwz+LU3yQ5tC/ZeYFQBQCAmuysckZYROn+Pcl8aPv2OXZUAQCg\nNrHKGWOez7szyb27dj1yGwAAKMppwJxhXPgXAAC2AjurAAAAlCNWAQAAKEesAgAAUI5YBQAAoByx\nCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUA\nAIByxCoAAADl7Fj1AVprz0zy873357bWvjXJu5N8ZvnhN/fe37XqNQAAALC1rDRWW2uvTvKDSR5Y\nPnRxkht67zes8rgAAABsbas+DfiPk3x/kml5/+IkL2it3dFae2tr7etWfHwAAAC2oJXGau/9liQH\nNzz0sSSv6r1fluSzSV63yuMDAACwNa38PatH+I3e+/3L27cmeePg180rWs9pYffuR26a0zGY1XEz\npzHmNM6sxpjTOLMaY05jzGmcWY0xpzHTZg+e6lh9X2vtFb33TyS5PMmdg1+36eJZuPvu7N29O7tj\nTsdkVsdljjmNMKdxZjXGnMaZ1RhzGmNO48xqjDmdoFMVqw+/onBtkje11g4kuS/JS07R8QEAANhC\nVh6rvfe9SZ61vP2HSZ696mMCAACwta36asAAAABw3MQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEes\nAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUA\nAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABA\nOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUA\nAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAA\nlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChH\nrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADl7Fj1AVprz0zy873357bWnpbkpiSHk3wqyct7\n7/Oq1wAAAMDWstKd1dbaq5PcmOTs5UM3JLmu935pkinJVas8PgAAAFvTqk8D/uMk359FmCbJRb33\njyxvvzfJFSs+PgAAAFvQSmO1935LkoMbHpo23H4gyXmrPD4AAABb08rfs3qEwxtun5vkS4Nf532t\nR7F79yM3zekYzOq4mdMYcxpnVmPMaZxZjTGnMeY0zqzGmNOYabMHT3Ws/n5r7bLe+x1Jnp/k9sGv\n23TxLNx9d/bu3p3dMadjMqvjMsecRpjTOLMaY07jzGqMOY0xp3FmNcacTtCpitWHX1H48SQ3ttbO\nSvLpJDefouMDAACwhaw8Vnvve5M8a3n7M0mes+pjAgAAsLWt+mrAAAAAcNzEKgAAAOWIVQAAAMoR\nqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAA\nUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAc\nsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIV\nAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAA\nAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADK\nEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5exYx0Fba7+X5P7l3c/2\n3n9kHesANjdN03T48OFM07TupQAAcIY65bHaWntCkvTen3uqjw0c3TRNU/LgB5ODz9u2bU5y4Lbk\n7CvneZ7XvbaKRD0AwOqs4zTgZyT5S62197fWbm+tPXMNawA29eAHk52XJ9unxY+HnZcnB+6Zpvsv\nWffKKpmmaZqmh25LDh7atm3OND1026RYAQBOqnXE6v4k1/fevzPJtUne2Vrz3llYs0VsbX9esrG5\npiQ7npqcc6sY20jUAwCs2jres3pXkj9Okt77Z1prf57kG5LsO8rXOAXxKHbvfuSmOR2DWT2+w4cP\nZ3Hq79c6//zpqffee/jwKV5SSYszondks6g///yv+8Q8z04LPjr/7Y0xp3FmNcacxpjTOLMaY05j\nNv3L0zpi9Zok35zk5a2185M8Kcl9x/gaf/M7irvvzt7du7M75nRMZvX4FoF14LbFLuHD45mTHNr3\n+c/v3zNN5925xuWVsW3btik5eCib/B76/Ofnfdu2bbvAe3wf1xz/7Y0wp3FmNcacxpjTOLMaY04n\naB2n3/5qkie11j6S5NeSXNN7t2MDJZx9ZXJow1kOh/YlOy+YZ6H6sEWIHvrQY18onZMc3Jfs3yNU\nAQBOjlO+s9p7P5jkRaf6uMCxzfM8T9P9e5In/aft27P90CHxtbmzr0wO3LN4P2/yaNSbFQDAyeLC\nRsBjLHdR792165HbHGERpfv3JPOh7dvn2FEFADj5xCqwiSneYnF0oh4AYLXWcYElgNOEoAcAWBU7\nqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAA\nUI5YBQAAoByxCsDKTdM0zfO87mUAAFuIWAVgZaZpmqbpoduSg4e2bZszTQ/dNk3TtO51AQD1iVUA\nVujBDyY7L0+2T4s/cnZenhy4Z5ruv2TdKwMAahOrAKzEYgd1+/OSjRupU5IdT03OudUOKwBwNGIV\nAACAcsQqACsxz/OcHPpQsvHCSnOSg/uS/XtmV1z6Gi5EBQCPEqsArNDZVyaH9j16/9C+ZOcF83ze\nnetbUz0uRAUAX0usArAyi93T/XuS+dD27XPsqD4eF6ICgCOJVQBWarmLeu+uXY/cZgMXogKAzYlV\nAE6BKY+NMTgx3t8LcPoTqwCwRi5EdXy8vxfgzCFWAWDtXIhqnPf3ApwpxCoArJkLUY3x/l6AM4tY\nBYACXIgKAB5LrAJAGS5EdTTe3wtwZtmx7gUAAIw7+8rkwD2LU3+TR9/fK1QBTjd2VgGALcP7ewHO\nHGIVANhSvL8X4MzgNGAAYAvy3l6A052dVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAOE1N0zT5\nl32OzZygJrEKAHCamaZpmqaHbksOHtq2bc40PXTbNE0uoXwEc4LaxCoAwGnnwQ8mOy9Ptk+Lv+7t\nvDw5cM803X/JuldWizkdL7vQnEpiFQDgNLLYGdz+vMf+W7RTkh1PTc651c7hgjkdH7vQrINYBQAA\njsEuNKeeWAUAOI3M8zwnhz6UbDxVc05ycF+yf8/sHM4k5nQ87EKzLjvWvQAAAE62s69MDtyziIkk\nObQv2XmBADuSOUFldlYBAE4zi9javyeZD23fPsdO4ebMaYxdaNZFrAIAnIbm+bw7k9y7a9cjt9mE\nOY06+8rFzvPDHt6FNrPNuGryySFWAQBOW1Me+z5DNmdOx2IXeoyrJp9cYhUAADgmu9AjXDX5ZBKr\nAADAILtzc3KSAAAPCklEQVTQj8dVk08+sQoAAEA5YhUAAOAEuWryyeffWQUAADgp/Nu9J5OdVQAA\ngJPAVZNPLrEKAABwkrhq8snjNGAAAICTyoV/TwY7qwAAAJQjVgEAAChHrAIAAFDOKX/PamttW5Jf\nSfLNSR5M8qO99z851esAAACgrnXsrO5Jclbv/VlJfiLJG9awBgAAAApbR6z+3STvS5Le+8eSXLKG\nNQAAAFDYOv7pmicl+fKG+4daa9t674c3++TfvffCZLp776lY2Fb1uezaNd273ZwGmNUYcxpjTuPM\naow5jTOrMeY0xpzGmdWYz2XXrv/wxf8lyfXrXsqWto5Y/XKSczfcf9xQTZJdB/f6R4qO4YJ1L2AL\nMasx5jTGnMaZ1RhzGmdWY8xpjDmNM6sxFyR56boXcRpYx2nAH03y3UnSWvu2JH+0hjUAAABQ2Dp2\nVn8jyZWttY8u71+zhjUAAABQ2DTP87rXAAAAAI+xjtOAAQAA4KjEKgAAAOWIVQAAAMpZxwWWhrTW\ntiX5lSTfnOTBJD/ae/+T9a6qntbaDye5enn3iUmekeQpvfcvP+4XnaFaa7+X5P7l3c/23n9kneup\nprX2zCQ/33t/bmvtaUluSnI4yaeSvLz37g3uG7TWdiZ5W5LdSc5O8rO993evd1X1tNbOSvLWJE9L\nciDJK3rvf7jeVdXUWtue5MYkT08yJ7m29/6f17uqepZ/P3hrFnM6nOTFvfe+3lXVs/Fn+vL+9yX5\ngd77C9e7srqOnBmba619fZJPJrm8937XutdTVWvtnyT53iQ7k/xy7/0da17SllR5Z3VPkrN6789K\n8hNJ3rDm9ZTUe39H7/25yx+sdyb5MaH6tVprT0iSh2clVB+rtfbqLP6SfPbyoRuSXNd7vzTJlOSq\nda2tsBcm+cJyRt+V5JfXvJ6qXpzkvy9/lr84i8Bnc9+T5HDv/dlJXpPkn615PVV9R5JzlnP6pzGn\nr3Hkz/TW2i8m+edZ/DxnE5v8Ocgmli/UviXJ/nWvpbLW2nOSfPvyz77nJPnra13QFlY5Vv9ukvcl\nSe/9Y0kuWe9yamutXZLkG3vvb133Wop6RpK/1Fp7f2vt9uWrpzzqj5N8fx79i8xFvfePLG+/N8kV\na1lVbb+e5LXL29uSHFzjWir7W3n0Z/ldSZ7aWnvSepdUU+/9N/PovyF/YZIvrm81pX0lyXmttSnJ\neUkeWvN6KjryZ/pHk/xvEatHc+TM2Nz1Sd6c5L51L6S470jy/7TWbk3y7iT/fs3r2bIqx+qTkmzc\nITy0PPWHzV2X5KfWvYjC9ie5vvf+nUmuTfJOv58e1Xu/JY+NrY1/WD+QxV8I2aD3vr/3/kBr7dws\nwvUn172mov4gix3DtNa+LcmTk5yz1hUV1ns/1Fq7Kckbk/xfa15OVR9N8oQk/zWLHZ5fWu9y6jny\nZ3rv/V1rXM6WsMmfgxyhtXZ1FmcUfWD5kLB/fE9OcnGSH8jy753rXc7WVfkv619Ocu6G+9t674fX\ntZjKWmv/Q5Kn997vWPdaCrsryx8UvffPJPnzJN+w1hXVtvG/tXOTfGldC6mstXZBkg8l+T9777+2\n7vUU9bYkX26t/U4Wb++4K8l/W++Sauu9X53F+zFvbK09cc3LqejVST7ae29JviXJO5bvjQZW65ok\nV7bWPpxH/9t7yprXVNX/l+QDvfeDy7OKvtpa+x/XvaitqHKsfjTJdyePvBr/R+tdTmmXJrl93Yso\n7pos3/fcWjs/i517p7A8vt9vrV22vP38JB852iefiZZ/QH8gyat77zeteTmV/Z0kH+q9/89Jbk5y\nX+/9wTWvqaTW2ouWF+RIFqe6Hs5jXzhi4Zw8eubVF7O4eMn29S0Hzgy998t6789ZXiflD5L8UO/9\nz9a9rqJ+N4vrWTz8985zstgo4TiVvRpwkt/I4tWbjy7vX7POxRT39CSulHx0v5rk7a21h6PrGjv1\nm3r4ir8/nsWuzllJPp1FZPBY12VxevRrW2sPv3f1+b33r65xTRX1JP93a+26JF/N4iJLbO7mJDe1\n1u7IIsBeKew3dX0WP89/J4s5/ZPe+1fWvKaq5iNuu6r7sZkRJ6z3/luttUtbax/PYnPwZf5Vhb+Y\naZ7NDQAAgFoqnwYMAADAGUqsAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQDKaa39vdba\njy1vf3jd6zkRrbVfbq39cGvtG1prv3WMzz2u77W1dslmX9Na++3W2mXHu9bj1Vp7aWvtJcvbb2+t\nXXCUz726tfb2v+BxTtr301q7qLX2+pPxXACs1o51LwAANmqtnZ3kHye5dPnQyqNrxeYk6b3fl+QF\nx/jck/W9zg8fd5V672/ZcPc5OfqL4CeynpP2/fTef6+19qrW2jf13j91Mp4TgNUQqwAMaa39RJK/\nn2R7kvf33v9xa+3CJO9L8oUkX03yb5NcneSvJPn3SX4pya8muSDJwSTX9d7f31r7qSTftnz8l3rv\n/3rDoV6Y5I7e+6HW2huXx/6Pvfdvb619IcmdSf5qkv8jyU/23p+7/Jybkny49/6O1toPJXllFvH0\nySQv770/OPh9PifJa5Z3dyX5eJIfTXL+hu/1K0m+K8kvZBGY25Pc1Hv/V621Kcn1Sb43yZ8leSjJ\nJ5az+u3e+4Wttd1J3p7kyUn++/L5X3zE9/pdSX46yc4kf5rkxb33/9ZauzLJDUkeTPKfj/G9fH2S\n27OY+7sf59fwZ5Js673/5PJr3p7kvb33d214npuW38e3JnlSkp/pvf/b5a/jnMWv/flJfqu1dmmS\ni5az2Zbk7iT/a5IpydOWO8F/LcntvfeXbLLm1yfZk8Xvl7f03t+44WPbk/zrJN+Y5ClJepLvT3JW\nkn+3fCxJfnr5/f7vSX4oyeEkH++9X7v8+DuTvCqL36sAFOU0YACOaRlOFyX5n5b/v6u19sLlh5+e\n5IW99yuzCJKnJvmW3vtrsojV23rvz0jyA0netgyoJDmr9/6NR4Rqsoi8jyRJ7/0Vy///9uXH/kqS\nn+u9f2sWMbPRnGRurX1jFvH37cvP+0IWYXI8vi3JS5P8zSRPSPLyI77X70jykiRz7/3iJM9MclVr\n7dlZxNMlSf5WkquSPO2INSbJryT59d77307yU1lE9yPfa2vtyUl+Lsl39N4vSvKBJK9vrZ2V5B1J\n/kHv/ZIkXz7K9/CXk/xWktctw+3xfg3fluQfJklr7Zwkz0vyG0c815xFjD5z+fFfaK09Zfn43Ht/\nfZLPJ/nuJPuzeNHih3rv35zkj5L88PJz/1qS71vO9fmttb+58SCttb+f5FlJvinJ30lyzfI4yeL3\n1rOSfLX3/qzlXJ+4POaeJH+6nMkPJnn2Mmx/IsnFy/8dbq2dv3yu38ni9xkAhdlZBWDEFVmEyieX\n95+QZG+S303y//beP7fhc3+v9354efu5SX4kSXrvf9pa+9jyeeYkH3ucY/2NJPceZS2P93XJImie\nu3yOj7XWksWu2yeP8jWbua33/idJ0lr7N1mE6S157Pd6RZJntNaet7x/TpK/nUWk3tx7P5Tki621\nWzd5/kuT/IMk6b2/N8l7j/j4M7MIu99efg/bk/z58vnv671/evl5v5rkX27y/FMWO5D39d5v2bDe\nr/k17L2/s7W2d7kjujvJe3rvBzZ5zhuXv677WmsfTfLsTT4n/397dxNaRxmFcfx/Y1Swtki1tahg\nKMjjxkVAIip+tSsXilaQqgiiIKVKF4J0I6QLteJGN0XrB2pBxBDBkiCCEDTWgEaNxgY91K6kglLb\nIi4K/bguzjvJ9HZujYs2V3l+kMXcO/N+zATCmXPeN2WMByJitsyvytg+AkxGxJFyvB+4rOG+vF/6\nP0Zmcin3oB0Rn0s6JOkJ4FryOS8DpoDnJV1JBujPlsz8FJmJ3w3siIhfy5j+lNSStDIiDnWZh5mZ\nLTFnVs3MbDH6gJcjYrBkK28iM38tsiS20u447ivnVFosvCg92qWvk5yeNZ1XK+dtd7R9fq3PkdpY\nbwC21NuQdJekmfKzraGbev/n1Y475/Z0rZ+bydLeNqf+fT3R0P6x+tg7M4ylzz21toeA+xvm3NQ2\n5bwXgIOSNtfG2/QMIbOrD5EZ1re7tFnvq4/uz+iUQFfSCklXlTHVr+mcS3Vt/b4MlGwvQEvS3WTW\n9q8y5kmgFRE/k8Hru8AtZOk2EXEPsKm0+XEJyOt9ncTMzHqWg1UzM1uMCeBhScsk9ZNZxg0N53UG\nHxOUzKqktWRAN9VwXt1+YKB2fKKUdHY6CKyVdKGklWSQ0gY+Be6VtKqsH32FXL86LyLGqqAtIrY1\ntH2HpDWS+sg1jx91mdvjkvolLSdLS4eAT4CNki6QtILmTZUmgY0AZQ3qax1z/RK4UdI15fNngBfJ\nktrVkgbL5w82tF2ZATYDw6X89UzPcBRYD1weEdMNbbVYKBW+mnwBMFk+r+7LcfKFQQCragH4VrKk\nejEmgQ3lnl5ErhG+ovb9evJFxDvkeuBbgX5Jm8h1qqNkyfZqSZdK+hHYGxHDZCn1dWUOy8kg98gi\nx2VmZkvAwaqZmf2jiBgHPiCDqB+AmYjYVb6u79LauWvrFmCdpFlyHeRjEfFbw3l1Y2Qpb2U38F3Z\nJXj+moiYI0s+54ARFta5zpIbE00A1W6v2/l3DpBZujngF+CN2vwqrwL7yKDwK+DNiJiMiDEyYN1L\nlvf+VLu2uv5J4D5JM8AwZXOlaq7AYeBRYKTcu0HgqYg4TpYPvyXpG3JdatddckvGcQe5iVXXZxgR\nR8mXCO91aaoNXCzpa2Cc3OzpcMecxsmgfg25bnSXpO/JjOf2WjtdRcSHwBfAt+Q9fSki9tWufR14\nQNI0sLPcrwHyWancq8/Idbp/lHOmy7gvYSFrfBv5e2ZmZj2s1W6f9Z3tzczMFq0EpXvIDZK6lgOf\nxf5vB7ZGxJ3nuu+lUjLAU8C6iPi94fvTdgj+L5M0Sga0Z9xN2czMlpYzq2Zm1lPKmtTnyBLWpXBO\n/kdpr5A0RP5rnJ1Nger/jaTryY2lHKiamfU4Z1bNzMzMzMys5zizamZmZmZmZj3HwaqZmZmZmZn1\nHAerZmZmZmZm1nMcrJqZmZmZmVnPcbBqZmZmZmZmPcfBqpmZmZmZmfWcvwFWlBGKHDJ7qwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116acfb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "key_classification_error_report(classification['label_true'], classification['label_pred']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic data - all transpositions of the original dataset\n",
    "\n",
    "This dataset show have all classes with uniform prior probabilities.\n",
    "\n",
    "It is expected that the model would perform worse since it cannot overfit on the priors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dataset_synth = pd.read_csv('data/beatles/derived/all_chords_with_keys_synth.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset_synth = dataset_synth.dropna()\n",
    "X, y = select_X_y(dataset_synth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(173292, 173292, 173292)"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset_synth), len(X), len(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "((X_train, y_train), (X_valid, y_valid), (X_test, y_test)) = split_dataset(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
      ]
     },
     "execution_count": 253,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model_synth = LogisticRegression()\n",
    "lr_model_synth.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training set accuracy (%): 0.381585781061\n",
      "validation set accuracy (%): 0.379122305895\n",
      "difference (% points): 0.00246347516608\n",
      "accuracy of model withou synth data 42.3476454294\n",
      "the model is better than uniform random guess 4.53039233459 times\n"
     ]
    }
   ],
   "source": [
    "accuracy_report(lr_model_synth, X_train, y_train, X_valid, y_valid)\n",
    "lr_synth_accuracy = 100 * lr_model_synth.score(X_valid, y_valid)\n",
    "print('accuracy of model without synth data', lr_accuracy)\n",
    "print('the model is better than uniform random guess', lr_synth_accuracy / mean_random_uniform_accuracy, 'times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can se the model is worse than model trained on data with skewed classes but not dramatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred_lr_synth_valid = lr_model_synth.predict(X_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(34659, 12) (34659,) (34659,)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xu0nXdd5/HPk5M0YC0dRxGlqemaxfAbryh0BCq2QKiK\nN6JLx5mpaLuQy9AFuEZErSzAuwyCiiK6qlBGGRW7ahEdboXaAjpgAWWQ8VcutrSxy0GBMo3QJifP\n/LF30pP0JHnSdGd/T/J6rdXVvfc5Zz+//c1pet77t8+zh3EcAwAAAJVsWvYCAAAA4FBiFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChn86IP0Fr74iTvTbIjyb4kV8z//cEkl/bevXcOAAAAB1no\nzmprbUuS30qyO8mQ5GVJLuu9nz+//qRFHh8AAICNadEvA35JklcmuW1+/eG99+vnl9+Y5AkLPj4A\nAAAb0MJitbV2cZJP9N7fMr9pmP+z3x1JzlzU8QEAANi4Fvk7q5ckGVtrT0jytUlek+SBaz5+RpJP\nT7ifzyXZet8vDwAAgAKG9W5cWKz23i/Yf7m1dm2SZyR5SWvtgt77dUmemORtE+5qaw6zeA4yxpym\nMqtpzGkac5rOrKYxp+nMahpzmsacpjOraczpOC38bMBrjEl+JMnlrbXTknwoyZUn8PgAAABsEMM4\nln/nGM9ITGNO05nVNOY0jTlNZ1bTmNN0ZjWNOU1jTtOZ1TTmdJwWfTZgAAAAOGZiFQAAgHLEKgAA\nAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADK\nEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNW\nAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWUj9VxHJe9BAAAAE6w8rG6adOYYbjr\nmmEYhmWvBQAAgBOjfKzOlrhlR7LnlmG4/dxlrwYAAIDFG6q/zHYYsmaBe3clW84eqy96OcYkdp+n\nMatpzGkac5rOrKYxp+nMahpzmsacpjOraczpOG2AnVUAAABONZuXvYBpxiSru5LdO+2qAgAAnPw2\nSKyuevkvAADAKaT8y4BXVsbYUQUAADi1lI/VbduScTzzhmWvAwAAgBOnfKw6gRYAAMCpZwPEKgAA\nAKcasQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABA\nOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUA\nAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAA\nlCNWAQAAKGfzou64tbaS5PIkD00yJnlGktOS/GmSG+ef9sre++sWtQYAAAA2poXFapJvT7Kv9/6Y\n1toFSX4uyRuSvLT3/rIFHhcAAIANbmEvA+69vz7J0+dXz0ny6SSPSPJtrbXrWmu/3Vr7/EUdHwAA\ngI1rob+z2ntfba1dkeRXk7w2yXuSPLf3fkGSjyV54SKPDwAAwMa0yJcBJ0l67xe31h6U5N1Jzuu9\n/8P8Q1cnefnEuxkXsriTjzlNZ1bTmNM05jSdWU1jTtOZ1TTmNI05TWdW05jTNMN6Ny7yBEtPTrKt\n9/4LST6bZF+Sq1prz+q9/1WSHUlumHh36y6eg4wxp6nMahpzmsacpjOracxpOrOaxpymMafpzGoa\nczpOi9xZvTLJFa2165JsSfKcJB9P8orW2p4ktyV52gKPDwAAwAY1jGPtnelzzsl4002ekZjAMzfT\nmdU05jSNOU1nVtOY03RmNY05TWNO05nVNOZ0nBZ6giUAAAC4N8QqAAAA5YhVAAAAyhGrAAAAlCNW\nAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIA\nAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACg\nHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDli\nFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoA\nAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAA\nyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQj\nVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wC\nAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlLN5kXfeWltJcnmShyYZkzwj\nyZ1JrkiyL8kHk1zaex8XuQ4AAAA2lkXvrH57kn2998ckeX6Sn0/y0iSX9d7PTzIkedKC1wAAAMAG\ns9BY7b2/PsnT51fPSfKpJI/ovV8/v+2NSZ6wyDUAAACw8Sz8d1Z776uttSuS/GqS12a2m7rfHUnO\nXPQaAAAA2FgW+jur+/XeL26tPSjJe5Lcb82Hzkjy6Ql34XdapzGn6cxqGnOaxpymM6tpzGk6s5rG\nnKYxp+nMahpzmmZY78ZFn2DpyUm29d5/Iclnk6wmuaG1dkHv/bokT0zytgl3te7iOcgYc5rKrKYx\np2nMaTqzmsacpjOracxpGnOazqymMafjtOid1SuTXNFauy7JliTPSfJ3SS5vrZ2W5EPzzwEAAIAD\nhnGsvTN9zjkZb7rJMxITeOZmOrOaxpymMafpzGoac5rOrKYxp2nMaTqzmsacjtPCT7AEAAAAx0qs\nAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUA\nAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABA\nOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUA\nAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAA\nlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChH\nrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgF\nAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoJzNi7rj\n1tqWJK9Ksj3J1iQ/m+TWJH+a5Mb5p72y9/66Ra0BAACAjWlhsZrkoiSf6L0/ubX2BUn+JslPJXlp\n7/1lCzwuAAAAG9wiY/WPklw5v7wpyZ4kj0jSWmtPSvLhJD/ce79jgWsAAABgA1rY76z23nf33u9o\nrZ2RWbj+ZJL3JHlu7/2CJB9L8sJFHR8AAICNa5E7q2mtnZ3kqiSv6L3/QWvtzN777fMPX53k5RPv\nalzIAk8+5jSdWU1jTtOY03RmNY05TWdW05jTNOY0nVlNY07TDOvduMgTLD0oyVuSPLP3fu385je1\n1p7de/+rJDuS3DDx7tZdPAcZY05TmdU05jSNOU1nVtOY03RmNY05TWNO05nVNOZ0nBa5s3pZkjOT\nvKC19oL5bT+c5Jdba3uS3JbkaQs8PgAAABvUMI61d6bPOSfjTTd5RmICz9xMZ1bTmNM05jSdWU1j\nTtOZ1TTmNI05TWdW05jTcVrYCZYAAADg3hKrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDli\nFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoA\nAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAA\nyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQj\nVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6ye\nBIZhGMZxXPYyAAAA7jNidQMbhmEYhruuSfaubto0ZhjuumYYhmHZ6wIAADheYnVDu/OtyZYdycow\n+6PcsiPZc8sw3H7uslcGAABwPMTqBjXbQV15fLJ2I3VIsvms5PSr7bACAAAbmVgFAACgHLG6QY3j\nOCarb0/WnlhpTLJ3V7J75+iMSwAAwAa2edkL4HhsvTDZc8vspb9Jsror2XK2UAUAADY6O6sb2CxK\nd+9MxtWVlTF2VAEAgJOFWN3gxvHMG5Lcum3bgcsAAAAbnpcBnxSc+BcAADi52FkFAACgHLEKAABA\nOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLE\nKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUA\nAADKEasAAACUI1YBAAAoZ/OxfHJr7f5JNvfe/9+Ez92S5FVJtifZmuRnk/yfJFck2Zfkg0ku7b2P\nx7hmAAAATnKTd1Zba09J8r+S/GVr7WcmfMlFST7Rez8/ybckeUWSlya5bH7bkORJx75kAAAATnaH\njdXW2lcdctPO3vvDeu9fleS7Jtz3HyV5wZrj7Eny8N779fPb3pjkCce4XgAAAE4BR3oZ8NNba6cl\n+ene+64kf91ae3Nm0fnBo91x7313krTWzsgsXJ+f5JfWfModSc68twsHAADg5HXYWO29P6u19tAk\nL26tfTzJi5N8SZKtvfcPTLnz1trZSa5K8ore+++31v7bmg+fkeTTE9fp91qPYPv2AxfNaTqzmsac\npjGn6cxqGnOazqymMadpzGk6s5rGnKYZ1rvxiCdY6r3fmOT7W2vnJfndJNcm+fUpR2utPSjJW5I8\ns/d+7fzm97fWLui9X5fkiUnedjyLZ+bmm3PT9u3ZHnOaaoxZTWFO05jTdGY1jTlNZ1bTmNM05jSd\nWU1jTsfpSL+z+szW2kdbazcmeXDv/TuT3JzkT1trF02478sye5nvC1pr17bWrs3spcA/1Vr7i8xC\n+crjfwgAAACcbI60s/rMJC2zt515R5Ire+9Xtdb+JMlTj3bHvffnJHnOOh967L1YJwAAAKeQI8Xq\nbUl+Jcn9M3t/1CRJ731vklcueF0AAACcwo4Uq9+R5JuT3JnkrSdmOQAAAHDkswF/LsnrT+BaAAAA\nIMkRTrAEAAAAyyJWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAA\nAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAo\nR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5Y\nBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVU4pwzAM4zguexkAAMBRiFVOCcMwDMNw1zXJ3tVNm8YM\nw13XDMMwLHtdAADA+sQqp4g735ps2ZGsDLNv+y07kj23DMPt5y57ZQAAwD2JVU56sx3UlccnazdS\nhySbz0pOv9oOKwAA1CNWAQAAKEesctIbx3FMVt+erD2x0phk765k987RGZcAAKCczcteAJwYWy9M\n9twye+lvkqzuSracLVQBAKAmO6ucEmZRuntnMq6urIyxowoAALWJVU4Z43jmDUlu3bbtwGUAAKAo\nLwPmFOPEvwAAsBHYWQUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wC\nAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAA\noByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5\nYhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQzuZFH6C19sgkv9h7f1xr7euSvCHJ\nh+cffmXv/XWLXgMAAAAby0JjtbX2vCTfn+SO+U2PSPKy3vvLFnlcAAAANrZFvwz4I0m+O8kwv/6I\nJN/WWruutfbbrbXPX/DxAQAA2IAWGqu996uS7F1z07uTPLf3fkGSjyV54SKPDwAAwMa08N9ZPcQf\n995vn1++OsnLJ37duKD1nBS2bz9w0ZyOwqyOmTlNY07TmdU05jSdWU1jTtOY03RmNY05TTOsd+OJ\njtU3tdae3Xv/qyQ7ktww8evWXTwzN9+cm7Zvz/aY01GZ1TEZY05TmNN0ZjWNOU1nVtOY0zTmNJ1Z\nTWNOx+lExer+ZxSekeQVrbU9SW5L8rQTdHwAAAA2kIXHau/9piTnzS//TZLHLPqYAAAAbGyLPhsw\nAAAAHDOxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoA\nAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACA\ncsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWI\nVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasA\nAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAA\nKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCO\nWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEK\nAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAA\ngHI2L/oArbVHJvnF3vvjWmsPSXJFkn1JPpjk0t77uOg1AAAAsLEsdGe1tfa8JJcn2Tq/6WVJLuu9\nn59kSPKkRR4fAACAjWnRLwP+SJLvzixMk+Thvffr55ffmOQJCz4+AAAAG9BCY7X3flWSvWtuGtZc\nviPJmYs8PgAAABvTwn9n9RD71lw+I8mnJ36d32s9gu3bD1w0p6Mwq2NmTtOY03RmNY05TWdW05jT\nNOY0nVlNY07TDOvdeKJj9f2ttQt679cleWKSt038unUXz8zNN+em7duzPeZ0VGZ1TMaY0xTmNJ1Z\nTWNO05nVNOY0jTlNZ1bTmNNxOlGxuv8ZhR9Jcnlr7bQkH0py5Qk6PgAAABvIwmO1935TkvPmlz+c\n5LGLPiYAAAAb26LPBgwAAADHTKwCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGr\nAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEA\nAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQ\njlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByx\nCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUA\nAIByxCoAAADliFUAAADKEasAAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA\n5YhVAAAAyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoR\nqwAAAJQjVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoAAADliFUAAADKEasAAACUI1YB\nAAAoR6wCAABQjlgFAACgHLEKAABAOZuXcdDW2vuS3D6/+rHe+1OWsQ4AAABqOuGx2lq7X5L03h93\noo8NAADAxrCMndWHJfm81tqb58e/rPf+7iWsAwAAgKKW8Turu5O8pPf+zUmekeS1rTW/OwsAAMAB\ny9hZvTHJR5Kk9/7h1to/J/nSJLuO8DXjiVjYRrV9+4GL5nQUZnXMzGkac5rOrKYxp+nMahpzmsac\npjOracxpmmG9G5cRq5ck+Zokl7bWHpzkAUluO8rXrLt4Zm6+OTdt357tMaejMqtjMsacpjCn6cxq\nGnOazqymMadpzGk6s5rGnI7TMmL1d5K8urV2/fz6Jb33fUtYBwAAAEWd8Fjtve9N8uQTfVxgumEY\nhn379mUYPBkIAMByOLERcMAwDMMw3HVNsnd106Yxw3DXNYNiBQBgCcQqsMadb0227EhWhtlfD1t2\nJHtuGYbbz132yioahmEYR+dNAABYBLEKJJmFV7Ly+IPPAzAk2XxWcvrVdljvZgcaAGDxxCrAMbMD\nDQCwaGIVSJKM4zgmq28/+O3AxiR7dyW7d45e75rEDjQAwImyjLeuAcraemGy55ZZeCXJ6q5ky9lC\nFQCAE83OKnDALEp370zG1ZWVMXZU78kONADAiSFWgYOM45k3JLl127YDl7mHrRfOdp33278DbV4A\nAPcVsQqsY8jBv5PJWnagAQAWT6wC3At2oAEAFssJlgDuNbvPAACLYmcVAACAcsQqAAAA5YhVAAAA\nyhGrAAAAlCNWAQAAKEesAgAAUI5YBQAAoByxCgAAQDliFQAAgHLEKgAAAOWIVQAAAMoRqwAAAJQj\nVgEAAChHrAIAAFCOWAUAAKAcsQoAAEA5YhUAAIByxCoACzcMwzCO47KXAQBsIGIVgIUZhmEYhruu\nSfaubto0ZhjuumYYhmHZ6wIA6hOrACzQnW9NtuxIVobZ/3K27Ej23DIMt5+77JUBALWJVQAWYraD\nuvL4ZO1G6pBk81nJ6VfbYQUAjkSsAgAAUI5YBWAhxnEck9W3J2tPrDQm2bsr2b1zdMYlAOAINi97\nAQCczLZemOy5ZfbS3yRZ3ZVsOVuoAgBHY2cVgIWZRenuncm4urIyxo7qkXmLHwC4m1gFYKHG8cwb\nkty6bduByxzCW/wcO2EPcPITqwCcAEMOPiswB/MWP1MJ+2Mj6oGNTKwCwBJ5i59jJeynEPXAyUCs\nAgAbgrA/FqIe2PjEKgAskbf44b4m6oGThVgFgKXbeuHsbX322/8WP05ItZawBzi1iFUAWDJv8XMs\nhP3RiHrgZCFWAaAAb/EzjbCfStQDG59YBYAyvMXPFML+6EQ9cDIQqwDABiTsj0bUAxudWAUAOGmJ\n+imGYRhsPEM9YhUAgFPSMAzDMNx1TbJ3ddOmMcNw1zXe2gfqEKsAAJyi7nxrsmVHsjLMfizesiPZ\nc8sw3H7uslcGiFUAAE5Bsx3Ulccf/DLpIcnms5LTr7bDCssnVgEAAChHrAIAcMqZvZXP6tuTtSdW\nGpPs3eWtfg7Pyag4kcQqAACnqK0XJqu77r6+uivZcra3+rknJ6NiGcQqAACnpNnu6e6dybi6sjLG\njuqROBkVJ55YBQDglDXfRb1127YDlzmEk1GxLJuXvQAAAFgurQUV2VkFAAAOy8moWBY7qwAAwFFs\nvTDZc8vspb/J3SejEqosjp1VAADgiJyMimUQqwAAwFE5GRUnmpcBAwAAEzkZFSeOnVUAAADKEasA\nAACUI1YBAAAoR6wCAABQjlgFAACgHLEKAABAOWIVAACAcsQqAAAA5YhVAAAAyhGrAAAA96FhGIZx\nHJe9jA1PrAIAANwHhmEYhuGua5K9q5s2jRmGu64ZhmFY9ro2KrEKAABwn7jzrcmWHcnKMEutLTuS\nPbcMw+3nLntlG5FYBQAAOE6zHdSVxydrN1KHJJvPSk6/2g7rsROrAAAAlCNWAQAAjtM4jmOy+vZk\n7YmVxiR7dyW7d47OuHTMNp/oA7bWNiX5jSRfk+TOJD/Ue//oiV4HAADAfWvrhcmeW2Yv/U2S1V3J\nlrOF6r2zjJ3VnUlO672fl+THk7x0CWsAAAC4T82idPfOZFxdWRljR/X4LCNWvyHJm5Kk9/7uJM6M\nBQAAnBTG8cwbkty6bduBy9xLJ/xlwEkekOQza66vttY29d73rffJ77z1nGS4+aYTsbCN6uPZtm24\ndcWcJjCracxpGnOazqymMafpzGoac5rGnKYzq2k+nm3b/uen/mOSlyx7KRvaMmL1M0nOWHP9sKGa\nJNv23uQUz0dx9rIXsIGY1TTmNI05TWdW05jTdGY1jTlNY07TmdU0Zyd5+rIXcRJYxsuA35XkW5Ok\ntfaoJB9YwhoAAAAobBk7q3+c5MLW2rvm1y9ZwhoAAAAobHByKgAAAKpZxsuAAQAA4IjEKgAAAOWI\nVQAAAMoqo7d6AAANrklEQVRZxgmWjqq1tinJbyT5miR3Jvmh3vtHl7uqmlprP5jk4vnV+yd5WJIH\n9d4/c9gvOkW11t6X5Pb51Y/13p+yzPVU1VrbkuRVSbYn2ZrkZ3vvb1juqmpprT0yyS/23h/XWntI\nkiuS7EvywSSX9t6dDGBu/vf5byd5aGYzemrvvS93VTW11laSXJ7ZrMYkz+i9/+1yV1VXa+2Lk7w3\nyY7e+43LXk9FrbXTMvvv7yFJ9iR5du/9b5a7qrrW/t2+7LVU1Vr7iSTfkWRLkl/vvb9myUsq59Dv\no9badyX5nt77Rctd2cZUdWd1Z5LTeu/nJfnxJC9d8nrK6r2/pvf+uPl/EDckeZZQvafW2v2SZP+s\nhOoRXZTkE73385N8S5JfX/J6SmmtPS+zoNg6v+llSS6bz2tI8qRlra2ob0pyeu/9MUl+OsnPLXk9\nlX17kn3zWT0/ZnVY8yfVfivJ7mWvpbinJvmX+c9TT83siUjWsc7f7RyitfbYJI+efz89Nsm/WeqC\nCjr0+6i19qtJfj6znw+4F6rG6jckeVOS9N7fneTc5S6nvtbauUm+svf+28teS1EPS/J5rbU3t9be\nNn/Wi/X9UZIXzC9vSrJ3iWup6CNJvjt3/4/n4b336+eX35jkCUtZVV2fTXJma21IcmaSu5a8nrJ6\n76/P3e8hf06STy1vNeW9JMkrk9y27IUU9xW5++epG5Oc1Vp7wHKXVNahf7dzT9+U5H+31q5O8oYk\nf7Lk9VR06PfRu5L8l/i+uteqxuoDkqzdHVydv5SMw7ssyYuWvYjCdid5Se/9m5M8I8lrfU+tr/e+\nu/d+R2vtjMzC9SeXvaZKeu9X5eCAX/s/oDsyCzLu9q4k90vyd5nthP3acpdTW+99tbV2RZKXJ/kf\nS15OSa21izN79cdb5jf5IfDw/jqzHfu01h6V5IFJTl/qiopa5+927umBSR6R5Hsy/1lqucup59Dv\no97765a4nJNC1R/WP5PkjDXXN/Xe9y1rMdW11v5Vkof23q9b9loKuzHzv1R77x9O8s9JvnSpKyqs\ntXZ2krcn+e+99z9Y9nqKW/t30xlJPr2shRT1vCTv6r23JF+b5DXz36PjMHrvF2f2e6uXt9buv+Tl\nVHRJkgtba9fm7u+pBy15TVW9KslnWmvvyOxXrG5M8snlLokN7J+SvKX3vne+U/+51toXLXtRnNyq\nxuq7knxrcuCZwA8sdznlnZ/kbcteRHGXZP67z621B2e2e+/lY+uY/9D3liTP671fseTlbATvb61d\nML/8xCTXH+mTT0Gn5+5Xynwqs5NyrCxvOXW11p48P3lJMnv59L4c/GQISXrvF/TeHzs/V8NfJ/mB\n3vs/LntdRX19krf33r8xyZVJbuu937nkNbFxvTOzc1ns/1nq9Mye/IeFKXk24CR/nNmzpu+aX79k\nmYvZAB6axNmSj+x3kry6tbY/JC6xW39Yl2X2UtYXtNb2/+7qE3vvn1vimiraf8bfH8lsB+y0JB/K\n7AdC7vaSzP7be0dmofoTvffPLnlNVV2Z5IrW2nWZzeo5woLj1JP8YWvtsiSfy+wkSxyZs7kfRu/9\nz1pr57fW3pPZhtcznf3+sMZDLpvTvTSMo9kBAABQS9WXAQMAAHAKE6sAAACUI1YBAAAoR6wCAABQ\njlgFAACgHLEKAABAOWIVgHJaa9/ZWnvW/PK1y17P8Wit/Xpr7Qdba1/aWvuzo3zuMT3W1tq5631N\na+3PW2sXHOtaj1Vr7emttafNL7+6tXb2ET734tbaq+/lce6zx9Nae3hr7cX3xX0BsFibl70AAFir\ntbY1yY8lOX9+08Kja8HGJOm935bk247yuffVYz0hb0Lfe/+tNVcfmyM/CX4867nPHk/v/X2ttee2\n1r6q9/7B++I+AVgMsQrAJK21H0/yvUlWkry59/5jrbVzkrwpySeSfC7J7yW5OMkXJvmTJL+W5HeS\nnJ1kb5LLeu9vbq29KMmj5rf/Wu/9N9cc6qIk1/XeV1trL58f+y97749urX0iyQ1JviTJjyb5yd77\n4+afc0WSa3vvr2mt/UCS52QWT+9Ncmnv/c6Jj/OxSZ4/v7otyXuS/FCSB695rJ9N8i1JfimzwFxJ\nckXv/Vdaa0OSlyT5jiT/mOSuJH81n9Wf997Paa1tT/LqJA9M8i/z+3/qIY/1W5L8VJItSf4+yVN7\n759srV2Y5GVJ7kzyt0d5LF+c5G2Zzf0Nh/kz/Jkkm3rvPzn/mlcneWPv/XVr7ueK+eP4uiQPSPIz\nvfffm/85jpn92T84yZ+11s5P8vD5bDYluTnJf04yJHnIfCf4y5K8rff+tHXW/OIkOzP7fvmt3vvL\n13xsJclvJvnKJA9K0pN8d5LTkvz+/LYk+an54/2vSX4gyb4k7+m9P2P+8dcmeW5m36sAFOVlwAAc\n1TycHp7k38//va21dtH8ww9NclHv/cLMguSsJF/be39+ZrF6Te/9YUm+J8mr5gGVJKf13r/ykFBN\nZpF3fZL03p89//ej5x/7wiS/0Hv/usxiZq0xydha+8rM4u/R88/7RGZhciweleTpSb48yf2SXHrI\nY/2mJE9LMvbeH5HkkUme1Fp7TGbxdG6Sr0jypCQPOWSNSfIbSf6o9/7VSV6UWXQfeKyttQcm+YUk\n39R7f3iStyR5cWvttCSvSfJ9vfdzk3zmCI/hC5L8WZIXzsPtcH+Gr0ryn5KktXZ6kscn+eND7mvM\nLEYfOf/4L7XWHjS/fey9vzjJPyT51iS7M3vS4gd671+T5ANJfnD+uV+W5Lvmc31ia+3L1x6ktfa9\nSc5L8lVJvj7JJfPjJLPvrfOSfK73ft58rvefH3Nnkr+fz+T7kzxmHrY/nuQR83/2tdYePL+vd2T2\nfQZAYXZWAZjiCZmFynvn1++X5KYk70zyf3vvH1/zue/rve+bX35ckqckSe/971tr757fz5jk3Yc5\n1r9NcusR1nK4r0tmQfO4+X28u7WWzHbd3nuEr1nPNb33jyZJa+13MwvTq3LwY31Ckoe11h4/v356\nkq/OLFKv7L2vJvlUa+3qde7//CTflyS99zcmeeMhH39kZmH35/PHsJLkn+f3f1vv/UPzz/udJL+8\nzv0Pme1A3tZ7v2rNeu/xZ9h7f21r7ab5juj2JH/ae9+zzn1ePv9z3dVae1eSx6zzOZmvcVfv/QPz\nx7d/x/biJNf33j89v/7RJF+0zlz+cH78PZnt5GY+g7H3/o7W2idba5cm+XeZ/TmfnuQvkvx8a+2s\nzAL9Z+c783+R2U7865O8ovf+D/M1faa1NrTW/nXv/ZOHeRwALJmdVQCm2JTkV3rvXzffrTwvs52/\nIbOXxO43HnJ90/xz9hty9xOlnzvMsfblnrumB6x5Oe94yH1vWXPM161Z6yOTPHvtfbTWvqO19v75\nPy9a5zBrj7+y5vqhj+1H1xznGzJ7ae+Yg///urrO/e9Zu/ZDdxjnx3znmvv++iT/YZ3HvN59Z/55\nv5jkn1prz1yz3vX+DJPZ7upFme2wXnGY+1x7rE05/J/RQaHbWntAa23bfE1rv+bQx7L/a9fO5Zz5\nbm+SDK2178xs1/aO+ZqvTzL03j+SWby+Nsk3ZvbS7fTed+b/t3f3IHZUUQDH/5FFwRgRJR/YGAJy\nKouABGz8SCpLYxOFNAoSJKSwSSOsjQTSpBKNGsQ0QlhB2UUEYdElCcRVV2MWPQQrsVCUpEyRZC3O\nfdnhOS9uimRH+f+6fTNz590725y555wHB9qYn7eAvHuv60iSBstgVZK0FvPA/ojYGBFT1C7j3p7z\nxoOPedrOakTsoAK6sz3ndf0CbO/8fa2ldI77E9gREfdExINUkLICfAk8FxGbW/3o21T96g2ZOTsK\n2jLzjZ6xn4mIbRFxF1Xz+NmEub0SEVMRsYlKLd0FfAHsi4i7I+J++psqLQD7AFoN6rtjcz0HPBER\nj7bPXweOUim1WyJiZ/v8xZ6xR5aAV4Hplv56s2c4A+wBtmbmYs9YG1hNFX6EegGw0D4frctV6oVB\nAps7AfhhKqV6LRaAvW1N76VqhB/uHN9DvYj4kKoHfhKYiogDVJ3qDJWyvSUiHoqIn4ALmTlNpVI/\n1uawiQpyL6/xe0mS1oHBqiTpX2XmHPAxFUT9CCxl5sl2uNuldbxr6yFgd0Scp+ogX87M33vO65ql\nUnlHPgW+b12Cb1yTmctUyucycIrVOtfzVGOieWDU7fUIt+Y3apduGfgVeL8zv5F3gItUUPg1cCIz\nFzJzlgpYL1DpvT93rh1dfxB4PiKWgGlac6XRXIFLwEvAqbZ2O4HXMvMqlT78QUR8S9WlTuyS23Yc\n36KaWE18hpl5hXqJ8NGEoVaA+yLiG2COavZ0aWxOc1RQv42qGz0ZET9QO55HOuNMlJmfAGeA76g1\nPZaZFzvXvge8EBGLwPG2XtupZxVtrb6i6nT/aucstu/9AKu7xk9R/2eSpAHbsLJy2zvbS5K0Zi0o\nPU01SJqYDnwb7/80cDgzn73T914vbQf4LLA7M//oOf6PDsH/ZRExQwW0N+2mLElaX+6sSpIGpdWk\nvkmlsK6HO/IbpUMREbuon8Y53heo/t9ExONUYykDVUkaOHdWJUmSJEmD486qJEmSJGlwDFYlSZIk\nSYNjsCpJkiRJGhyDVUmSJEnS4BisSpIkSZIGx2BVkiRJkjQ4fwOLW4PissOE4QAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112b14668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "key_classification_error_report(classification_results(X_valid, y_valid, y_pred_lr_synth_valid));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there's almost no gap between training and validation accuracy the model seems to be highly biased, just too simple. We can try a few combinations of regularization parameters, but we expect regularization do not help here.\n",
    "\n",
    "As for the kind of errors, we can see the correct key is often quite close to the true key on the circle of fifths. Moreover, the distribution of errors nicely follows the two-way distance on the circle of fifths. This is an important thing and should be taken into account when choosing/designing the cost function or validation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV(cv=None,\n",
      "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001),\n",
      "       fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
      "       param_grid={'C': [0.1, 0.5, 1], 'penalty': ['l2', 'l1']},\n",
      "       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,\n",
      "       verbose=0)\n",
      "0.380508248397\n",
      "1\n",
      "GridSearchCV(cv=None,\n",
      "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001),\n",
      "       fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
      "       param_grid={'C': [0.1, 0.5, 1], 'penalty': ['l2', 'l1']},\n",
      "       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,\n",
      "       verbose=0)\n",
      "best mean accuracy: 0.380508248397\n",
      "1 l2\n",
      "[mean: 0.38034, std: 0.00161, params: {'C': 0.1, 'penalty': 'l2'}, mean: 0.38041, std: 0.00153, params: {'C': 0.1, 'penalty': 'l1'}, mean: 0.38049, std: 0.00164, params: {'C': 0.5, 'penalty': 'l2'}, mean: 0.38048, std: 0.00162, params: {'C': 0.5, 'penalty': 'l1'}, mean: 0.38051, std: 0.00166, params: {'C': 1, 'penalty': 'l2'}, mean: 0.38049, std: 0.00164, params: {'C': 1, 'penalty': 'l1'}]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "def tune_lr_model_on_grid():\n",
    "    model = LogisticRegression()\n",
    "    grid = GridSearchCV(estimator=model, param_grid=dict(C=[0.1, 0.5, 1], penalty=['l2', 'l1']))\n",
    "    # let the CV do multiple train-valid splits\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "    grid.fit(X_train, y_train)\n",
    "    return grid\n",
    "\n",
    "lr_model_synth_grid = tune_lr_model_on_grid()\n",
    "print(lr_model_synth_grid)\n",
    "print('best mean accuracy:', lr_model_synth_grid.best_score_)\n",
    "print(lr_model_synth_grid.best_estimator_.C, lr_model_synth_grid.best_estimator_.penalty)\n",
    "print(lr_model_synth_grid.grid_scores_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see regularization didn't help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear model trained with stochastic gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter: 0 validation score: 0.3545110938\n",
      "iter: 1 validation score: 0.362532098445\n",
      "iter: 2 validation score: 0.373409504025\n",
      "iter: 3 validation score: 0.378545255201\n",
      "iter: 4 validation score: 0.378689517874\n",
      "iter: 5 validation score: 0.378660665339\n",
      "iter: 6 validation score: 0.378776075478\n",
      "iter: 7 validation score: 0.378804928013\n",
      "iter: 8 validation score: 0.378776075478\n",
      "iter: 9 validation score: 0.378458697597\n",
      "iter: 10 validation score: 0.378458697597\n",
      "iter: 11 validation score: 0.378718370409\n",
      "iter: 12 validation score: 0.378660665339\n",
      "iter: 13 validation score: 0.378718370409\n",
      "iter: 14 validation score: 0.378718370409\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA68AAALACAYAAAB8TUTSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xmc61V9//HX3I31sooi4oaVgxsuoCKIKIsKmBtLbVJ+\nbd2A1rigNoqKVtGK2NJAwUqqoFUrv5rQiiFsioiiKBSLC4oeLP7qjrJvF7iXO/P745uR8TI3ycx8\nk3yT7+v5eHwfmSQn557cz+Teec853/OdmpmZQZIkSZKkLFs26gFIkiRJktSL4VWSJEmSlHmGV0mS\nJElS5hleJUmSJEmZZ3iVJEmSJGWe4VWSJEmSlHkruj0ZQlgGnAHsCdwPHB1jvGHO838CvAOYAc6O\nMZ7eec1ZwO7ANHBMjDGGEJ4JtIGfdF5ejzE2035DkiRJkqTJ0zW8Ai8HVsUY9w0hPBeodR4jhLAc\nOAnYC7gHuC6EcDbwbGCrGOPzQwgHAycCr+i0OyXGeMpg3ookSZIkaVL1Wja8H3AxQIzxKmDv2Sdi\njBuAPWKMdwE7ActJZmfvBbYNIUwB2wLrOi/ZCzg8hPC1EMJZIYStU30nkiRJkqSJ1Su8bgPcOef+\nhs6yYABijNMhhCOA7wCXAWuBK4DNgR8DHwdO7zS/CnhbjPEA4KfA+1J5B5IkSZKkiddr2fCdwOo5\n95fFGKfnNogxfj6EcC7wKeCVwC7AFTHGd4cQdgW+EkJ4KnBujPGOzsu+wIOhtpv7gM36aCdJkiRJ\nGi9TC2ncK7xeARSAc0II+wDfn30ihLANyQZMh8QY14UQ7iHZoGkrHpytva3zZ6wALg4hHBtjvBo4\nCPh2H+PbjAW+IU2MGax9nln//LL2+Wb988va55v1V1+mZmZmNvlk57zV2d2GAV5Dcu7q1jHGM0MI\nxwBHAeuB7wFvIjnP9V+BhwErgX+KMX4uhPB04KOdtr8B/irGeHeP8fmNnF/WPt+sf35Z+3yz/vll\n7fPN+qsvXcNrBviNnF/WPt+sf35Z+3yz/vll7fPN+qsvvTZskiRJkiRp5AyvkiRJkqTMM7xKkiRJ\nkjLP8CpJkiRJyjzDqyRJkiQp8wyvkiRJkqTMM7xKkiRJ0hgJIWwWQjiqz7avCiEUBj2mYfA6r8oq\na59v1j+/rH2+Wf/8svb5Zv0XKITwOODfY4zPG/VYhmnFqAcgSZIkSeOqUG2dDPxpyt2e064V397l\n+XcDTw4hbAC+DGwNHAW8CtgL2BH4XozxtSGEE4DfAD8G3gncD+wGfC7G+KGUxz1QLhuWJEmSpPHy\nQeA64P3AdTHG/YBfAbfGGF8MPBvYJ4SwC8nM9qzHAEcA+wDHDXfIS+fMqyRJkiQtUmeGtNss6SBM\nzbm9vvP1fcDDQwj/F7ibZDZ25UavuzbGOA2sDSHcO5SRpsiZV0mSJEkaLxt4MMtNd24PBR4dY/w/\nJMuKt+Ch5xJnesOjXgyvkiRJkjRefgesAjbnwUB6FbBbCOErwGmd+7t0npvZ6Hbjr8eCuw0rq6x9\nvln//LL2+Wb988va55v1V1+ceZUkSZIkZZ7hVZIkSZKUeYZXSZIkSVLmGV4lSZIkSZlneJUkSZIk\nZZ7hVZIkSZKUeStGPQBJkiRJUvpCCJcBrwP2AW6NMbY3ev7aGOPTurz+j4ErgWngvTHGNwxyvL0Y\nXiVJkiRpcs3EGD+9yNceC1wXY4zASIMrGF4lSZIkadFKjcrJwJ+m3O05zXL97Zt6MoTwn8BpMcbL\nQwh7AycDvwO2A3YBPhpj/JdO86kQwgnAb4CPAx8D9gR+AWzT6e+pQA1YDjwMqAA7AM8APh1C+Evg\nMzHG54UQDgH+DrgPuAV4LfBM4B3A/cBuwOdijB9K6e/i9zznVZIkSZLGy5nAqzpfvwa4lCQwvgR4\nCfA3G7Wf6dz+MbBljHEfkoC6befxJwPVGOPBwN8Dr4kxXgB8F3glsH5OXx8D/jjG+ELga8B7Ov0/\nBjiCZInycem8zT/kzKskSZIkLVJnhnSTs6QD8iXg5BDC9sDzgUOBk0IIRwB3sumcF4CrAWKMN4cQ\nftR5/NfA34YQ7gVWA3fM++IQHgbcGWP8TeehrwMnAucD18YYp4G1nX5S58yrJEmSJI2RTkg8B/gX\n4FygCnwrxviXwH+w6Zx3HbAvQCf47t55/DTgfTHGVwPXznn9NMlS4tk/92ZgmxDCzp2HDgBi5+vZ\n2d2BceZVkiRJksbPvwL/QzLruxvwkc7uwD8E7gohrNqo/UyMsRVCeFEI4SqS2dYbO899FjgnhPAL\n4NvAIzuPfxP4NPDXPBhOjwE+H0KYBm4FXg08jT8MrwMJslMzMwMPyEsxA0yNehAaCWufb9Y/v6x9\nvln//LL2+Wb91ReXDUuSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmS\nJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqS\nJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+S\nJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOr\nJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/w\nKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwz\nvEqSJEmSMs/wKkmSJEnKPMOrJEmSJCnzDK+SJEmSpMwzvEqSJEmSMm/FqAcgSRqNQrU1RfL/wKo+\nj5WDbvuYnVfz8xvv+gZwW79Hu1a8dwB/PZIkKWOmZmZmRj2GbmaAqVEPQiNh7fNtYutfqLaWA1sB\nW8+5nXtsxdKC4ELarxzw2+3HDLBu9th6i5U73n3v+mkWtjLofhYQdjc67m3Xipn+jzBnJvazr56s\nfb5Zf/XF8Kqssvb5NvL6d2Ylt+APQ+V8QbOfx+Y+vsUQhr+BOYFwnmN9j+cX23bB7du14oaNxj7T\nCfirge0XcWwHLF/A39U6Fh981xp8Uzfyz75Gxtrnm/VXXwyvyiprn28Lqn+h2lrF4kJlr8eW+j34\nAHAXcDdwT+d27jHfY2tJZhIXGxjXzxMIx8mSPvudXzosJvhu17ldyOk061l88L3H4Dsv/+3PL2uf\nb9ZffTG8KqtyW/tCtXUA8PRRj2OEpl552JP+6TMX/uhE+p/RXOry1xn6C5YLeqxdK65b4rjyaGSf\n/U7w3YrFzfhuz8K+Dx9g/lB7N0ub1e535nt9RsNzbv/tVz5rX6i2lgGbk/z7MQNM97idyehnd6mG\nUv/Ov/PLF3gsW8RrFtr/1EYHXe53e27c2tKuFY9mAQyvyqpc1r5QbQXgB7iZWjf3kn7Q9LzH7BjL\nz37nB6ItWXzwXTX8UfMAKYbhNNqd9e5D4tEnXvKojduN+WoC9Sczn/3O53kVyWd6i41u035s80UO\n8/dhlt6Bt9/bkfW195MeUfj2j357MYMPk5n4HtOD2rXigmpieFVW5a72nf8sLwReChwPxNGOaHRO\nOGaf/zzhzCv356Fhc60/xE68vH72tyAJsXM37Br47s59ts3Cxl7TjOa87MX0/YC/DFuUnp/9QrW1\ngvTD46aeS/tykutJTgtZS/JL2I1v1/PgbNSyBd4u5jXD6mspZkj2cJjvmO7yXNrHoP+s6c4x+55n\nj273J6Ztu1b8HxbA8Kqsyl3tC9XWy4A28GXgxTn/4Sd39dfvWfuM6YTrfgLvUgP0Zgfu/eijv/Lt\nX/z7EvodxQz2fBYTjNfz4A+wufO8pz3yFd+69jcX0z1Qpv2LlGn+METOFyzTeOzedq34QMpjHwud\nfz96Bd1ljRMPu7387gu3Yk6oy/nPQdoEw6uyKle1L1RbmwE/BB4H7NmuFa8b7YhGLlf11x+w9vmW\nxoZdy8nGjHWvNpst5b1OuHsZTIjc+LGsnvudR/7br754Xp2UDW8BngCcZnCVpMXpBJEHOsfaEQ+n\np85loeaG3dz+8H72Bw793Z+/96KtgPvatWJuZ6AldefMq7IqN7UvVFu7ANeT/Db4ie1a8fYRDykL\nclN/PYS1zzfrn1/WPt+sv/rizKs0en9PsknLWw2ukiRJ0vzS3klN0gIUqq19gb8ArgE+OeLhSJIk\nSZlleJVGpHNh8tM7d4/1EjCSJEnSphlepdF5DbAXcHa7Vrxi1IORJEmSsqzrOa8hhGXAGcCewP3A\n0THGG+Y8/yfAO0hOsj47xnh65zVnAbuTXD/rmBhjDCH8EfCpzmM/AN4QY8z0blHSoBSqre2Ak4B7\nSD5DkiRJkrroNfP6cmBVjHFf4J1AbfaJEMJykh++DwKeB7w+hLAj8GJgqxjj84EPACd2XnIKcHyM\n8QUku4kV03wj0ph5H7ATcGK7VvzVqAcjSZIkZV2v8LofcDFAjPEqYO/ZJ2KMG4A9Yox3kfwQvpxk\ndvZeYNsQwhSwLbCu85JnxRgv73x9EXBwWm9CGieFauvJwBuBG4BTRzwcSZIkaSz0Cq/bAHfOub+h\nsywYgBjjdAjhCOA7wGUkFwS/Atgc+DHwcR7ckGbutZvuJgm2Uq4Uqq0p4DSSJftvbdeK9414SJIk\nSdJY6HWd1zuB1XPuL4sxTs9tEGP8fAjhXJLzWV8J7AJcEWN8dwhhV+ArIYSnkZzrOms10O/1LD0v\nNr8mrvbHv/o5fOhT/8WzwsM54Zh9zhv1eDJu4uqvvln7fLP++WXt883659NU7yYP6hVerwAKwDkh\nhH2A788+EULYBmgDh8QY14UQ7iEJqFvx4GztbcBKkiXF3wkhHBBj/BpwKHBpn2Nc0BvSxJhhwmpf\nqLY2B64DHn1N/N3TpqamfjzqMWXYxNVffbP2+Wb988va55v1V196hddzgUNCCLOX8XhNCOFIYOsY\n45khhM8Cl4cQ1gPfA/6NZDnwv4YQvk4SXN8VY1wbQqgCZ4YQVpH8AP8fg3hDUoZVgccDp7RrRYOr\nJEmStABTMzOZnqH3tzD5NVG1L1RbuwKR5Hzv3du14h0jHlLWTVT9tSDWPt+sf35Z+3yz/upLr5lX\nSen4B2BL4I0GV0mSJGnheu02LGmJCtXW/sCRwNXAp0c8HEmSJGksGV6lASpUW8t58HJRx7Zrxelu\n7SVJkiTNz/AqDdbRwDOAz7RrxStHPRhJkiRpXBlepQEpVFvbAyeSbNL0zhEPR5IkSRprbtgkDc77\ngR2B49q14m9GPRhJkiRpnDnzKg1Aodp6KvB64CfAaSMejiRJkjT2DK9SygrV1hRJYF0OvKVdK64b\n8ZAkSZKksWd4ldJ3BHAgcEG7Vrxw1IORJEmSJoHhVUpRodraAqgB64G3jng4kiRJ0sQwvErpejvw\nWODUdq34k1EPRpIkSZoUhlcpJYVq6zEkl8S5EfjgiIcjSZIkTRQvlSOl52RgC+B17VrxrlEPRpIk\nSZokzrxKKShUWy8ESsCVwGdHOxpJkiRp8kzNzMyMegzdzABTox6ERmJsal+otlYA1wBPA57TrhWv\nHvGQJsHY1F+ps/b5Zv3zy9rnm/VXX5x5lZbur0iC6ycNrpIkSdJgGF6lJShUWzuSbM50J3D8iIcj\nSZIkTSw3bJKW5gPA9kC1XSv+dtSDkTQapUZlGckvhGeP5Rvdn++x+dpMARuA6TlHr/sPeaxZrk8P\n+C1LkjR0hldpkQrV1tOB1wE/Bv55xMORRqrUqEwBq0h23N4C2HzO1xsfm3zuwN324ys/veIT9A56\niw2HabSZ7zWZUmpUoEfAned+Wm0W3e+aPQ7hvB9f8jfAb0guO3Zj5+s7muV6pjfpkCQNnhs2Kasy\nXftCtTUFfBV4AfDSdq34xdGOaOJkuv5Z1wmSK1likFzEa7NUs6GErQG8BoYTtgfR7yDdz4NBdu7t\nxl//tlmurxvwWDQYE//vfmeFxrbAw+Y5tgQeANb3uF3scw9p0yzXNwz4LS9EJurf+f9zOckE38pN\n3C72uW5tcqtZrp+wkPaGV2VVpmtfqLZKQAM4r10rFkc9nlHo/AO/ObDVIo6tuzy35RYrN9/23vX3\n3U56oSLNgDLovmbYdGBcSJAcVJi4H7gXuK9zO9+x6Of+6bATvv+WC0/4I5YQBp2hG405P/QtNiSv\n+OBBb//xey49+QhgZ+CRnduNv+71g94tbDrczv36dr9XMiXT/+9vrPP9vpr5g+imjh1Ivt+zYoYB\nB+R+nzt6rz/7l7P++3PHMphguNDwqSFqlusL+twbXpVVma19odraCvgR8Ajgye1a8YYRD2mTOv+5\nbsHiQmQ/RxoBaRq4G7inc6x93Ha77vm/t//yWnrPBC30h+NMfk8twToGGCTne2wI51Jm9rOvoeha\n/87M1Q50D7ezX2/X48/aeDZ3U187mzscI/vsd/6v3JLuwXPHeR5b2Uf3M8CtwM2d45Y5X8891rLp\nYJXm7F8az2UpgHcztPC9xNc/QPJ9kkvNcv2rC2lveFVWZbb2hWrrA8DfAie1a8XUdxguNSrbAE8m\npVlM0vl7fIAHw2Wv4+4FtL0HuH+e2Y+B1L/zA8oU6S6lTLuvZWwiOG58P2NLvtKS2c++hiK1+pca\nlc3pHm7nft3PbG4/QdfZ3MVLu/a9gufGx+Z9dn873UPoxsdtk/ZvdeeXSGkurV355ue99uzTvvXJ\nV5BeeHQVzoQyvCqrMln7QrX1eJJZ11uA0K4V706z/1Kj8nDguyQ/UC3UetILlH/wmhHMOmSy/hoK\na59vQ69/5wfx7XkwyHYLuv3O5m4q3Dqbu2nz1r7UqKykv/A599iqzz/zbvoLoLPHrc1yff2i3p16\n8d9+9cV13dLC/COwGXBc2sG14zSSH5AaJCG57yDqf6iStHCdpfC3dI4fdGvbmdF7BL2D7rPosaS0\n1KjMzubOhtrfkfwSMpcO3/0gLrj+0k/z0CC6TZ9d3AfcBFxPf0H0lma5fn+a70HS4DnzqqzKXO0L\n1dZBwJeBbwLPb9eKqX54So3KGqAFXAk8f9KWGS1Q5uqvobH2+TYR9Z9nNrdb0N1+RMPMsnX0Pxt6\nC3Bzs1xfO5qhKiUT8dnX4BlelVWZqn2h2loJfIfkXNS927XiNWn2X2pUtgWuI/kt8zOb5fp1afY/\nhjJVfw2Vtc+33NV/zmzuwxmfjXBSd9Ih7/zWuy758G4kgfRuz1fMndx99rU4LhuW+lMBngKcmXZw\n7fgHYBfgvQZXScqPZrl+H/CzzpFrzXL9/416DJKybdAXFZfGXqHa2gl4P3AH8O60+y81Ki8C/gq4\nFvj7tPuXJEmSJoEzr1JvHyTZYfIt7VrxpjQ7LjUqWwJnklzr9Ch3n5QkSZLm58yr1EWh2nomcAzJ\n+ahnDOCP+ADwBOCUZrl+9QD6lyRJkiaC4VXahEK1NQV8hGQDgTe3a8VUL2FQalSeA7wV+B/gfWn2\nLUmSJE0aw6u0aUcC+wHntmvFL6fZcalRWQV8guQzeIxb/EuSJEndGV6leRSqra1JdgC+H6gO4I94\nJ/BU4GPNcv2rA+hfkiRJmiiGV2l+7wIeBZzcrhVT3bq/1Kg8BXgP8CvgHWn2LUmSJE0qw6u0kUK1\n9QTgbcAvgQ+n2XepUVlOslx4JVBplut3pNm/JEmSNKkMr9JD1YBVwNvateI9Kfd9LPBc4N+b5Xo7\n5b4lSZKkiWV4leYoVFsvBorA5UAzzb5LjcpuwInAzcCb0+xbkiRJmnQrRj0AKSsK1dZK4DRgmuTS\nODNp9V1qVKaAM4EtgKOa5fpNafUtSZIk5YEzr9KD3gjsAXy8XSt+N+W+XwscCJwPfC7lviVJkqSJ\nNzUzk9rk0iDMAFOjHoRGYqi1L1RbjwCuBzYAu7drxZvT6rvUqOwCXEfyy6InN8v1X6bV9wTzs59f\n1j7frH9+Wft8s/7qi8uGpcSJwDbAG1MOrlPAGcC2wOsMrpIkSdLiuGxYuVeotvYmWdZ7LfCxlLv/\nU5INoL5Gcs6rJEmSpEVw2bCyaii1L1Rby4BvAM8DDmzXipel1XepUdkR+BGwGtizWa7/JK2+c8DP\nfn5Z+3yz/vll7fPN+qsvLhtW3v05SXA9J83g2nEqsBNwnMFVkiRJWhqXDSu3CtXWauAfgPuAt6fZ\nd6lRORT4S+DbJCFWkiRJ0hIYXpVn7wZ2Bj7crhV/llanpUZlNcm5sw+QXNP1gbT6liRJkvLK8Kpc\nKlRbTwT+Bvg5cHLK3Z8EPBo4qVmufz/lviVJkqRcMrwqr04FVgLVdq24Nq1OS43K/sAbSDZqOjGt\nfiVJkqS8M7wqdwrV1mHA4cBlwH+m1W+pUdkC+ATJjnlHNcv1+9PqW5IkSco7w6typVBtrSKZdZ0G\n3tyuFdO8VtR7gScCpzfL9W+l2K8kSZKUe4ZX5c2xwO7AGe1a8dq0Oi01Ks8i2bH4f0k2gpIkSZKU\nIsOrcqNQbT0SeB9wS+c2FaVGZSXJcuHlwDHNcv2etPqWJEmSlFgx6gFIQ3QSsDXw9nateGuK/b4N\neAbwyWa5/uUU+5UkSZLU4cyrcqFQbT0XeBXwPeDMtPotNSp7kMzi3kgSYiVJkiQNgOFVE69QbS0D\nPtK5+6Z2rbghjX5LjcoykuXCmwGvb5brt6XRryRJkqSHMrwqD14JPBv4XLtW/HqK/b4e2Bf4j2a5\nfm6K/UqSJEnaiOFVE61QbW0LfBhYCxyXVr+lRuWxnX5vA96YVr+SJEmS5ueGTZp0fws8Avjbdq34\nizQ6LDUqU8DHgK1Ilgv/No1+JUmSJG2aM6+aWIVqaw/gzcD/A/4xxa7/EngJ8EXg31LsV5IkSdIm\nTM3MzIx6DN3MAFOjHoRGYkm1L1RbU8BFJCHziHatmMo5qaVGZWfgOmAV8JRmuf6zNPrVQ/jZzy9r\nn2/WP7+sfb5Zf/XFZcOaVC8jCa5fBr6QYr8fAbYH3mRwlSRJkobHZcOaOIVqazPgVGAD8OZ2rZjK\n8oJSo3IE8ArgCuCMNPqUJEmS1B/DqybRW4EnAP/crhWvS6PDUqOyPfBR4H7gqGa5Pp1Gv5IkSZL6\n47JhTZRCtbUL8B7gZuCEFLv+R2Bn4PhmuR5T7FeSJElSHwyvmjR/T3IJm7e2a8Xb0+iw1KgcDLwW\n+C7p7losSZIkqU8uG9bEKFRb+wJ/AVwDfDKNPkuNytbAmSTnzx7VLNfXp9GvJEmSpIUxvGoiFKqt\n5cDpnbvHtmvFDSl1/UHgccDJzXL9mpT6lCRJkrRAhldNitcAewFnt2vFK9LosNSoPA84Frge+EAa\nfUqSJElaHMOrxl6h2toO+BBwD/CONPosNSqbAZ8guWD20c1y/d40+pUkSZK0OG7YpEnwPmAn4Ph2\nrfirlPp8N/Ak4KPNcv3rKfUpSZIkaZGcedVYK1RbTwbeBNwAnJpGn6VGZU/gXcAvOreSJEmSRszw\nqrFVqLamgNOA5SSXxrlvqX2WGpUVJDsVrwD+ulmu37XUPiVJkiQtneFV46wIHAx8ETg/pT7fSrLx\n0781y/WLUupTkiRJ0hIZXjWWCtXW5sApwAPAW9q14sxS+yw1Kk8k2VX4JpIQK0mSJCkj3LBJ46oK\nPB6otWvFHy+1s1Kjsgw4E9gceFWzXL9lqX1KkiRJSo8zrxo7hWprV+B44HfA36XU7THAAcAXgHNS\n6lOSJElSSpx51Tj6B2BL4I3tWvGOpXZWalR2BU4G7gDe0CzXl7wEWZIkSVK6DK8aK4Vqa3/gSOBq\n4NNL7a/UqEwB/wKsBo5uluu/XmqfkiRJktLnsmGNjUK1tRz4SOfuse1acTqFbo8EDge+QnKJHEmS\nJEkZZHjVODkaeDrw6XateOVSOys1KjsBpwP3Ase4XFiSJEnKLpcNaywUqq3tgROBu4F3pdTtacCO\nwN80y/WfptSnJEmSpAEwvGpcvJ8kaB7XrhV/s9TOSo1KgWTJ8FUks6+SJEmSMsxlw8q8QrX1NOD1\nwE9IZkuXpNSobAvUgfXAUc1yfcNS+5QkSZI0WIZXZVqh2poiCazLgbe0a8V1KXT798CjgA82y/Uf\nptCfJEmSpAEzvCrrjgBeBFzQrhUvXGpnpUblhcBfAz8APrzU/iRJkiQNh+FVmVWotrYETiFZ3vvW\npfZXalS2BM4CpkmWC6cxiytJkiRpCAyvyrK3A48BTm3Xij9Job/3A08ATm2W6/+VQn+SJEmShqTr\nbsMhhGXAGcCewP3A0THGG+Y8/yfAO4AZ4OwY4+khhFcDr+o02YLkupyPIAkNbZJNdwDqMcZmem9F\nk+R3t64FeCdwI/DBpfZXalSeDfwNcAPw3qX2J0mSJGm4el0q5+XAqhjjviGE5wK1zmOEEJYDJwF7\nAfcA14UQPhtj/BTwqU6bfwbOijHeGULYCzglxnjKQN6JJsonz/8hwObAO9q14l1L6avUqKwCPkGy\n0uCYZrm+dukjlCRJkjRMvZYN7wdcDBBjvArYe/aJGOMGYI8Y413ATiS7wf7+HMIQwt7AU2KMZ3Ue\nehZweAjhayGEs0IIW6f3NjRJCtXWC6/43q8BrgQ+m0KX7wCeBny8Wa5flkJ/kiRJkoasV3jdBrhz\nzv0NnaXEAMQYp0MIRwDfAS4D5s5oHQ+cMOf+fwFvizEeAPwUeN8Sxq3JNrus99h2rTi9lI5KjcpT\ngL8Ffg0ct9SBSZIkSRqNXsuG7wRWz7m/LMb4B2Eixvj5EMK5JEuFXwl8KoSwHbB7jPFrc5qeG2O8\no/P1F4DT+xzjTJ/tNAHuvGcdy5ZN8cRHb8c/HvuCJW2qND09zRN3eBw/ufV/Oe75lV32ftSet6c1\nTg2Fn/38svb5Zv3zy9rnm/XPp6mFNO4VXq8ACsA5IYR9gO/PPhFC2IZkA6ZDYozrQgj3ABs6T78A\nuHSjvi4OIRwbY7waOAj4dp9jXNAb0nj78/de9BfAvz33KTvDEmv/Z+e84S3AqcDn9n7UnkemMDwN\nzwx+9vNv/etmAAAgAElEQVTK2ueb9c8va59v1l996RVezwUOCSFc0bn/mhDCkcDWMcYzQwifBS4P\nIawHvseD5yfuTrKr61yvAz7aafsb4K9SeQeaNGsAnpOE10UrNSq7AScCtwDHLn1YkiRJkkZpamYm\n0zP0/hYmRwrV1mbATcDvzvvHNU+YmppaVO1LjcoUcAnJDP+fN8v1/5viMDUcfvbzy9rnm/XPL2uf\nb9Zffem1YZM0TAeQnGN93iJz66zXkATXC4B/T2FckiRJkkbM8KosWdO5PW+xHZQalV2AU4C7gEqz\nXM/00gJJkiRJ/el1zqs0FIVqa4okvN5GslHYgnWWC58BbEsSXH+R3gglSZIkjZIzr8qKZwCPBi5s\n14rrF9nHK4AicDnw8bQGJkmSJGn0DK/KiiUtGS41KjsC/wzcBxzdLNene7xEkiRJ0hhx2bCyYg2w\nHvjiIl9/CvBw4Lhmuf6T1EYlSZIkKROcedXIFaqtXYFnAV9t14p3LPT1pUblpcArgf8GTk15eJIk\nSZIywPCqLCh0bhe8ZLjUqKwGPgY8ABzVLNcfSHNgkiRJkrLB8KosmD3ftb2I154EPAb4cLNc/156\nQ5IkSZKUJYZXjVSh2loNHAh8t10r/mwhry01KvsDbwB+DHxwAMOTJEmSlBGGV43ai4FVLHDJcKlR\n2Rw4C5gBXtss1+8fwNgkSZIkZYThVaO22EvkvBfYHTi9Wa5/K90hSZIkScoaw6tGplBtrQBeBvwa\nuKbf15UalWcCxwH/C7xnIIOTJEmSlCmGV43SvsAOwHntWnGmnxeUGpWVwCeB5cBfNcv1uwc4PkmS\nJEkZYXjVKC1myfDbgGcA/9os1y9Jf0iSJEmSssjwqpEoVFtTQBG4B7isn9eUGpU9gPcBNwLVwY1O\nkiRJUtYYXjUqAfgj4IvtWvG+Xo1Ljcoykt2FNwPe0CzXbxvw+CRJkiRliOFVo7LQJcMVYD/gP5vl\n+ucHMyRJkiRJWWV41aisAaaBC3o1LDUqjwU+DNwGvHHA45IkSZKUQStGPQDlT6Ha2olkp+FvtGvF\nm7u1LTUqU8C/AFsDr26W6zcOYYiSJEmSMsaZV43C4cAU/S0Z/kvgpcCXgM8MclCSJEmSssvwqlHo\n63zXUqPyCOCfSHYk/utmud7XtWAlSZIkTR6XDWuoCtXWFsBLgNiuFa/v0fwjwPbAsc1y/X8HPTZJ\nkiRJ2eXMq4btQGBLesy6/tcvvwvwp8A3gY8OfliSJEmSssyZVw1bzyXDpUZl++023wZgHXBUs1yf\nHsbAJEmSJGWXM68amkK1tQwoALcA3+rS9O9uv+9OgPc3y/UfD2NskiRJkrLN8Kph2gt4JHB+u1bc\nMF+DUqOyAjhy+y22BTh5iGOTJEmSlGEuG9Yw9bPL8H7ADs/e5ekcvfeR64cwJkmSJEljwJlXDdMa\n4H6Sa7Z2a8Pej3r6UAYkSZIkaTwYXjUUhWrrccCewKXtWvHu+dqUGpUpoAjc/ZSHP3GIo5MkSZKU\ndYZXDUuhc9ttyfCTgCcAF69cvnLwI5IkSZI0NgyvGpZi5/b8Ptq0BjwWSZIkSWPG8KqBK1Rb2wEH\nAN9u14q/6tJ0DbABuHAoA5MkSZI0NgyvGoaXkuxsvcklw6VGZWfgucDXm+X6rcMamCRJkqTxYHjV\nMPRziZyXAVM92kiSJEnKKcOrBqpQba0EDgN+Dny/S9PZ810Nr5IkSZIewvCqQdsf2BY4r10rzszX\noNSobAUcDPywWa7fMMzBSZIkSRoPhlcN2uyS4W47CB8MbN6jjSRJkqQcM7xqYArV1hRJeL0TuLxL\nU5cMS5IkSerK8KpBegrweOCidq24br4GpUZlOclmTTcCVw9xbJIkSZLGiOFVg9TPLsP7ADsB7Wa5\nPj34IUmSJEkaR4ZXDVIR2ABc1KVNPwFXkiRJUs4ZXjUQhWrrkcBzgMvbteJtXZoWgbXApUMZmCRJ\nkqSxZHjVoLysc7vJGdVSoxKAAHypWa7fO5RRSZIkSRpLhlcNyuxy4HYfbbxEjiRJkqSuDK9KXaHa\n2ork2q0/bNeKN3RpugaYBi4YysAkSZIkjS3DqwbhYGBzui8Z3gnYF/hms1y/aVgDkyRJkjSeDK8a\nhH6WAx9O8v3nLsOSJEmSejK8KlWFams5UAB+C1zdpannu0qSJEnqm+FVaXsOsBPQbteK0/M1KDUq\nWwAvAWKzXL9+mIOTJEmSNJ4Mr0rb7Ixqt+XABwJb9mgjSZIkSb9neFXaisC9wKU92oBLhiVJkiT1\nyfCq1BSqrScCTwIuadeKa+drU2pUlpGcE3sTcOUQhydJkiRpjBlelaZC57bbcuBnAzsD5zfL9Q2D\nH5IkSZKkSWB4VZrWADPA+T3agOe7SpIkSVoAw6tSUai2dgSeD1zVrhV/26XpGuA+4JKhDEySJEnS\nRDC8Ki2HAsvpsglTqVHZDXgq8OVmuX7PsAYmSZIkafwZXpWWfpYDz7Zxl2FJkiRJC2J41ZIVqq3N\ngJcCNwA/6tJ09hI53c6JlSRJkqSHMLwqDQcAq4Hz2rXizHwNSo3KDsD+wFXNcv3GYQ5OkiRJ0vgz\nvCoNszOq3ZYMH0aPc2IlSZIkaVMMr1qSQrU1RXIu623AFV2aeokcSZIkSYtmeNVSPQPYFbiwXSuu\nn69BqVHZjGQ34huA64Y4NkmSJEkTwvCqpepnRvWFwNbAec1yfd5zYiVJkiSpG8OrlmoNsB74Ypc2\ns+fEer6rJEmSpEUxvGrRCtXWrsCzgK+2a8U75mtTalRmz4m9le7nxEqSJEnSJhletRSFzm23GdVn\nAo8CLmiW6w8MfkiSJEmSJpHhVUsxe75ru0ubfi6jI0mSJEldGV61KIVqazVwIPDddq348y5N1wDr\n6H5OrCRJkiR1ZXjVYr0YWEWXGdVSo/JYkkvpfKVZrt81rIFJkiRJmjyGVy1WP8uBC320kSRJkqSe\nDK9asEK1tQI4HPg1cE2XprMBt9s5sZIkSZLUk+FVi7EvsANwXrtWnJmvQalR2RZ4IfDfzXL9l0Mc\nmyRJkqQJZHjVYszuMtxtOfChwAq6X0ZHkiRJkvpieNWCFKqtKZLlwPcAl3Vp2k/AlSRJkqS+GF61\nUAH4I+Didq1433wNSo3KSuAw4GfA94c4NkmSJEkTyvCqhepnRvUFwLbAec1yfd5zYiVJkiRpIQyv\nWqg1wDRwYY824PmukiRJklJieFXfCtXWTiQ7DV/RrhVvnq9NqVGZPSf2DuDyIQ5PkiRJ0gQzvGoh\nDgem6L5k+GnAY4GLmuX6+qGMSpIkSdLEM7xqIYqd227hdbaNS4YlSZIkpcbwqr4Uqq0tgBcDsV0r\nXt+l6RrgAeDioQxMkiRJUi4YXtWvA4Et6TLrWmpUHgXsDXy1Wa7fPqyBSZIkSZp8hlf1q59L5BT6\naCNJkiRJC2Z4VU+FamsZSTC9GfhWl6b9nBMrSZIkSQtmeFU/9gIeCZzfrhU3zNeg1KisJlla/L1m\nuf6zYQ5OkiRJ0uQzvKof/SwZfjGwCncZliRJkjQAK7o9GUJYBpwB7AncDxwdY7xhzvN/ArwDmAHO\njjGeHkJ4NfCqTpMtgKcDjwAeDnwKmAZ+ALwhxjiT5pvRwKwhqf8lXdq4ZFiSJEnSwPSaeX05sCrG\nuC/wTqA2+0QIYTlwEnAQ8Dzg9SGEHWKMn4oxvijG+CLg28CbYox3AqcAx8cYXwBM8WDYUYYVqq3H\nkfzy4tJ2rXj3fG1KjcoK4HDgV8A1wxudJEmSpLzoFV73o3O9zhjjVSSXQaFzfwOwR4zxLmAnYDmw\nbvb5EMLewFNijGd1HnpWjPHyztcXAQen8g40aP0sGd4P2AE4r1muO5suSZIkKXW9wus2wJ1z7m/o\nLCUGIMY4HUI4AvgOcBmwdk7b44ET5tyfmvP13cC2ixmwhm42vJ7fRxuXDEuSJEkaiK7nvJIE19Vz\n7i+LMU7PbRBj/HwI4VyS81lfCXwqhLAdsHuM8Wtzms593Wrg9j7H6EzeiNx973qWL5vi8Y/allPf\ncsAv52szMzPDI7beiTvuu5NPvPzki1IegrXPN+ufX9Y+36x/fln7fLP++TTVu8mDeoXXK0iu73lO\nCGEf4PuzT4QQtgHawCExxnUhhHuA2cuovAC4dKO+vhNCOKATaA+d5/lNWdAbUnqOfM+Ffwb8+//8\n4vb3An83X5ty8/VPBn4I/MfK5Sv/NMU/fgZrn2fWP7+sfb5Z//yy9vlm/dWXXuH1XOCQEMIVnfuv\nCSEcCWwdYzwzhPBZ4PIQwnrge8BnO+12B27YqK8qcGYIYRVwHfAfqbwDDVI/y4FnN97yEjmSJEmS\nBmZqZibTM/T+FmZECtXWSuAmkuXdj2/XivN+o5QalW8BzwYe3izXb01xCNY+36x/fln7fLP++WXt\n8836qy+9NmxSfu1PsqnWeV2C687Ac4GvpxxcJUmSJOkPGF61Kf0sGX4ZyW/J3GVYkiRJ0kAZXvUQ\nhWpriiS83glc3qWpl8iRJEmSNBSGV83nKcDjgYvateK6+RqUGpWtgEOAHzbL9Y0355IkSZKkVBle\nNZ/ZHYS7zageDGzeo40kSZIkpcLwqvmsIblm70Vd2niJHEmSJElDY3jVHyhUW48EngNc3q4Vb5uv\nTalRWU6yWdONwNVDHJ4kSZKknDK8amMv69x2Ww68D7AT0G6W69ODH5IkSZKkvDO8amP97CDsLsOS\nJEmShsrwqt8rVFtbkWzE9IN2rfjTLk3XAGuBS4cyMEmSJEm5Z3jVXD13EC41KrsDewBfapbr9w5r\nYJIkSZLyzfCquRayZNhdhiVJkiQNjeFVABSqreVAAfgt3XcQLgIzwAXDGJckSZIkgeFVD3oOnR2E\n27XivDsIlxqVnYB9gW82y/Wbhjk4SZIkSflmeNWsYue225Lhw0m+Z1wyLEmSJGmoDK+atQa4l+47\nCHuJHEmSJEkjYXgVhWrricCTgEvateLa+dqUGpUtgJcAsVmux2GOT5IkSZIMr4JkoyboPqN6ILBl\njzaSJEmSNBCGV0GyHHgGOL9Lm9lzYj3fVZIkSdLQGV5zrlBt7Qg8H7iyXSv+dr42pUZlGcns7E3A\nlUMcniRJkiQBhlfBocByui8H3hvYGTi/Wa5vGMqoJEmSJGkOw6v62UG4n8voSJIkSdLAGF5zrFBt\nbQa8FLgB+FGXpmuA+4BLhjEuSZIkSdqY4TXfXgisBs5r14oz8zUoNSq7AU8Fvtws1+8Z4tgkSZIk\n6fcMr/nWz5LhftpIkiRJ0kAZXnOqUG1NkQTT24ArujSdPd+1PfBBSZIkSdImGF7z6xnArsCF7Vpx\n/XwNSo3KDsD+wFXNcv3GYQ5OkiRJkuYyvObX7HLgVpc2h5FcRqdbG0mSJEkaOMNrfq0B1gNf7NEG\nPN9VkiRJ0ogZXnOoUG3tCjwLuKxdK945X5tSozL3MjrXDXF4kiRJkvQQhtd8KnRuu82ovpDOZXSa\n5fq8l9GRJEmSpGExvObT7HLgbjsIu2RYkiRJUmYYXnOmUG2tBg4EvtuuFX8+X5tSozL3MjrfGOLw\nJEmSJGlehtf8eQmwiu4zqs8kuYzOBc1y/YGhjEqSJEmSujC85k8/y4GLnVsvkSNJkiQpEwyvOVKo\ntlYAhwO/Bq7p0nQNsI7ul9GRJEmSpKExvObLvsAOwHntWnHeHYRLjcpjgWcAX2mW63cNc3CSJEmS\ntCmG13yZXTLcbTlwP5fRkSRJkqShMrzmRKHamiI5l/Vu4LIuTWfPd+12GR1JkiRJGirDa34E4I+A\nL7Zrxfvna1BqVLYFXgj8d7Nc/+UQxyZJkiRJXRle86OfXYZfCqzAXYYlSZIkZYzhNT/WANPAhV3a\nzC4Z9nxXSZIkSZlieM2BQrW1E8lOw1e0a8Wb52tTalRWAocBPwO+P8ThSZIkSVJPhtd8eBkwRfcZ\n1RcA2wLnNcv1eS+jI0mSJEmjYnjNh37Od+2njSRJkiSNhOF1whWqrS2AFwOxXSteP1+bUqMyexmd\nO4CvDXF4kiRJktQXw+vkOxDYku47CD8NeCxwUbNcXz+UUUmSJEnSAhheJ18/y4Fndxn2EjmSJEmS\nMsnwOsEK1dYyoADcDFzZpeka4AHg4mGMS5IkSZIWyvA62fYCHgmc364VN8zXoNSoPArYG/hqs1y/\nfZiDkyRJkqR+GV4nWz9Lhgt9tJEkSZKkkTK8TrY1wP3AJT3agOFVkiRJUoYZXidUodp6PLAncGm7\nVrx7vjalRmU1cBDwvWa5/rNhjk+SJEmSFsLwOrn6WQ78YmBVjzaSJEmSNHKG18k1uxz4/C5tvESO\nJEmSpLFgeJ1AhWprO+AA4NvtWvFX87UpNSorgMOBXwHXDHF4kiRJkrRghtfJ9FJgBd1nVPcDdgDO\na5brM0MZlSRJkiQtkuF1MvWzg7C7DEuSJEkaG4bXCVOotlYChwE/A66dr02pUZkiOd/1buCy4Y1O\nkiRJkhbH8Dp59ge2Bc5r14qbWg78JOAJwMXNcv3+oY1MkiRJkhbJ8Dp5XDIsSZIkaeIYXidIodqa\nIgmmdwKXd2laBDYAFwxjXJIkSZK0VIbXyfJU4PHARe1acd18DUqNys7Ac4FvNMv1W4c5OEmSJEla\nLMPrZOlnOfDLgCm6X0ZHkiRJkjLF8DpZ1pAsB76oRxvwfFdJkiRJY2RqZmZTG9JmwgzJLKF6KFRb\njwR+DXylXSseNF+bUqOyFXAzcEOzXH/qMMe3CNY+36x/fln7fLP++WXt8836qy/OvE6Ol3Vuu82o\nHgxs3qONJEmSJGWO4XVyzC4HbndpU+zcer6rJEmSpLFieJ0AhWprK5JZ1R+0a8Wfztem1KgsJ5md\nvRG4eojDkyRJkqQlM7xOhn6WAz8X2AloN8v16aGMSpIkSZJSYnidDP3sIFzso40kSZIkZZLhdcwV\nqq3lQAH4Ld2XA68B1gKXDmNckiRJkpQmw+v4+/1y4HatOO9y4FKjsjuwB/ClZrl+7zAHJ0mSJElp\nMLyOv36WDPfTRpIkSZIyy/A6/tYA99J9OXCR5OLP5w9lRJIkSZKUMsPrGCtUW08EngR8qV0rrp2v\nTalR2QnYF/hms1y/aZjjkyRJkqS0GF7HW6Fz22058OEkdW4NfjiSJEmSNBiG1/G2hmQ58AU92oDn\nu0qSJEkaY1MzMzOjHkM3M8DUqAeRRYVqa0eSy+P8V7tW3He+NqVGZXPgFuAXzXJ9j2GOLwXWPt+s\nf35Z+3yz/vll7fPN+qsvzryOr0OB5XSfUT0I2LJHG0mSJEnKPMPr+PISOZIkSZJyw/A6hgrV1mYk\nM683AD+ar02pUVlGEl5vBr41vNFJkiRJUvoMr+PphcDWwHntWnFTJy3vDewMnN8s1zcMa2CSJEmS\nNAiG1/HUz3LgYufWS+RIkiRJGnuG1zFTqLamSMLrbcA3ujRdA9wHXDKMcUmSJEnSIBlex88zgF2B\nC9q14gPzNSg1KrsBTwW+3CzX7xnm4CRJkiRpEAyv48ddhiVJkiTljuF1/KwB1gNf7NJm9nzX9uCH\nI0mSJEmDt6LbkyGEZcAZwJ7A/cDRMcYb5jz/J8A7gBng7Bjj6Z3H3wUUgJXAP8cYPx1CeCZJmPpJ\n5+X1GGMz5fcz0QrV1q7As4AvtWvFO+drU2pUdgD2B65qlus3DnN8kiRJkjQoXcMr8HJgVYxx3xDC\nc4Fa5zFCCMuBk4C9gHuA60IInyUJus/rvGYr4LhOX3sBp8QYTxnA+8iLQue223LgQ4HlPdpIkiRJ\n0ljptWx4P+BigBjjVSTXDqVzfwOwR4zxLmAnksC0HngJcG0I4QskM62zIWov4PAQwtdCCGeFELZO\n9Z3kw+y5rN2WA3uJHEmSJEkTp1d43QaYuzx1Q2cpMQAxxukQwhHAd4DLSGZgH0YSVF8BvA44u9P8\nKuBtMcYDgJ8C70vlHeREodpaDRwIfLddK/58vjalRmUz4KUkf7/XDXF4kiRJkjRQvZYN3wmsnnN/\nWYxxem6DGOPnQwjnAp8CXgncDPwoxvgAcH0I4b4QwsOAc2OMd3Re9gXg9D7HONNnu4n2zlc+mw9/\n5mr+7JDwDDbxd3L8C97Ehy7/CIfvftDqVz3zFdPztRkz1j7frH9+Wft8s/75Ze3zzfrn09RCGvea\neb0COAwghLAP8P3ZJ0II23SWAK+KMc6QzLpuAL5BMvtHCGEXYEvgVuDiEMKzOy8/CPh2n2Oc8mDq\nw5+5+t8APndJ3HtTbT50+UfOALjg+ktfNOrxpnBY+3wfZGAMHtbew/p7WHsP6+8x+Nr3rdfM67n/\nv717D5KsrNM8/i36glwavAAjIorXF0VRAUVAAUUQhKRY0UwYFEGbbrJ7d2djakNHN2J2Y2MnnFij\n2F1XOoEGAQUlEwXSlOHiAIL0qCgwXMR55TKDF0BF7o3Q0F37xzm1lC2VmdVd5+TJPN9PBHGqKt/s\neIqX7uapc87vAIeGENakn58cQjge2DbGuDod0HRDCOE54DbgghjjVAjhwBDCTSTleGV6efGpwOnp\n2geBZXMNW1aVifZC4EjgN8AtL7am2qyPkdwT+yjJDxAkSZIkaWSMTU0V+gz9FJvQyEdNZaJ9IHA9\n0OhMjq94sTXVZn0v4Gbgglat8ck882XEvS8397+83Ptyc//Ly70vN/dffel12bCKYXrKcLfH3zhl\nWJIkSdLIsrwWXGWiPUZSTJ8imeg8m6OBdcBVeeSSJEmSpDxZXosvAG8ErupMjj/7YguqzfprgHcC\n17ZqjSfzDCdJkiRJebC8Fl8/lwz3s0aSJEmShpbltfiOBjYA/9BjDUAn+ziSJEmSlD/La4FVJto7\nAfsDazqT4w+/2Jpqs749cDBwc6vW+HWO8SRJkiQpN5bXYjuSZGx4t8uBDwcW9VgjSZIkSUPN8lps\n05cDd3v8jY/IkSRJkjTyLK8FVZlobwUcBvxLZ3L87hdbU23WFwEfAe4Hbs8xniRJkiTlyvJaXB8E\ntqb75cAHAtsD32nVGlO5pJIkSZKkAbC8FpePyJEkSZKklOW1gCoT7S2ACvAw8KMXW1Nt1sdIyuvj\nwPX5pZMkSZKk/Flei2lvYGfgu53J8fWzrHk7sBtwRavWeC6vYJIkSZI0CJbXYvpYevSSYUmSJEnC\n8lo4lYn2YuAk4BHgii5Lx4Hne6yRJEmSpJFgeS2eo4GdgPM7k+PPvNiCarO+C7APcH2r1ngsz3CS\nJEmSNAiW1+JZnh5Xd1lTSY/tjLNIkiRJUiFYXgukMtF+A/Ah4IbO5PjPuyz1fldJkiRJpWJ5LZal\n6fGs2RZUm/UlwCHAba1a4/5cUkmSJEnSgFleCyId1PRp4FHg212WHgYsxrOukiRJkkrE8locFXoM\nakqNp0fvd5UkSZJUGpbX4liWHmcd1FRt1hcCRwIPALfkEUqSJEmSisDyWgCVifbrSC4HvrEzOX5X\nl6X7Ay8HvtOqNaZyCSdJkiRJBWB5LYaeg5pSXjIsSZIkqZQsrwNWmWgvIhnU9BjwrdnWVZv1MZLy\n+hRwXT7pJEmSJKkYLK+DdxTwSuBrncnxP3ZZ9xbgDcCVrVrj2VySSZIkSVJBWF4Hb3l67HXJ8NHp\n0UfkSJIkSSody+sAzRjUtKYzOf6zHsvHgfXA5ZkHkyRJkqSCsbwO1meAMXqcda02668E9gVubNUa\nj+QRTJIkSZKKxPI6IBsNarq4x/KjSEquU4YlSZIklZLldXCOBHYGvt5jUBN4v6skSZKkkrO8Ds6y\n9Li626Jqs741cCjws1atcW/mqSRJkiSpgCyvA1CZaL8WOBz4YWdy/I4eyw8FXoJnXSVJkiSVmOV1\nMPoa1JTykmFJkiRJpWd5zVllor2QpLw+DrS6ra026wuACvBb4Kbs00mSJElSMVle8/cR4FXABZ3J\n8ad7rN0X2BHotGqNDZknkyRJkqSCsrzmb3l67OeS4fH06CNyJEmSJJWa5TVHlYn2a4AjgB91Jsdv\n7+MtRwNPA9dkGkySJEmSCs7ymq++BzVVm/U3A7sDV7dqjV7PgZUkSZKkkWZ5zcmMQU1P0GNQU8op\nw5IkSZKUsrzm5whgF5JBTWv7WD8OTAHfzTSVJEmSJA0By2t+lqXHfi4Z3hHYH/inVq3x+0xTSZIk\nSdIQsLzmoDLR3pXkETk3dSbHb+vjLUeS7I2XDEuSJEkSlte8fJrk33U/j8eB5N5YgG9nE0eSJEmS\nhovlNWOVifYCYCnwJNDstb7arO8JvA+4qlVr3JtxPEmSJEkaCpbX7B0OvBq4sDM5/lQf61ekx1XZ\nRZIkSZKk4WJ5zd7y9NjPoKbtgU8AvwQuzzKUJEmSJA0Ty2uGKhPtV5MMX/pJZ3L81j7e8klgG+CM\nVq2xPtNwkiRJkjRELK/Z6ntQU7VZHyO5ZHgdcE7GuSRJkiRpqFheMzJjUNNTwEV9vOVg4C3Axa1a\n43cZRpMkSZKkoWN5zc6HgV3pf1DTyvTooCZJkiRJ2ojlNTvL0mM/lwzvAhwD3Ab8MMtQkiRJkjSM\nLK8ZqEy0dwGOAm7uTI7f0sdbTgEWAKe3ao2pTMNJkiRJ0hCyvGbjZJIy2s9Z10UkZ2mfAL6RcS5J\nkiRJGkqW13k2Y1DTWuCbfbzlGGBn4LxWrbE2y2ySJEmSNKwsr/PvUOC1wDc6k+NP9rHeQU2SJEmS\n1IPldf7NZVDTHsBBwDWtWiNmmkqSJEmShpjldR5VJtqvAo4GbulMjv+0j7fU0+Pp2aWSJEmSpOFn\neZ1fcxnUtAQ4EfgN0Mk4lyRJkiQNNcvrPKlMtLcgeeRNv4OaPgEsAc5s1RrPZ5lNkiRJkoad5XX+\nTA9q+mZncvyJbgurzfoYyaCm54HVOWSTJEmSpKFmeZ0/fQ9qAt4P7AF8u1VrPJRdJEmSJEkaDZbX\neVCZaO9MMqjpn4F+BjWtSI8OapIkSZKkPlhe58dJwELgrM7k+FS3hdVmfWfgWOBO4Mbso0mSJEnS\n8IAoZk4AABofSURBVLO8bqYZg5qeBr7Rx1uWkhTdVa1ao2vRlSRJkiQlLK+b7xDgdcBFncnxx7st\nrDbrC4HlwJPABTlkkyRJkqSRYHndfHMZ1HQ0sAvwtVat8WR2kSRJkiRptFheN0Nlov1K4BjgduCm\nPt4yPahpVWahJEmSJGkEWV43z0kk96+e2cegpt1JLjH+fqvWuCuHbJIkSZI0Miyvm2jGoKY/Ahf2\n8ZZ6evSsqyRJkiTNkeV1030QeD39DWrahuQs7YPAZdlHkyRJkqTRYnnddHMZ1HQCsB1wVqvWeC67\nSJIkSZI0miyvm6Ay0f4L4N8BdwA/7ra22qyPASuB9fRXdCVJkiRJG7G8bppPkQxqOqvXoCZgf2BP\n4NJWrfFA5skkSZIkaQRZXudoxqCmZ4AL+niLj8eRJEmSpM1keZ27g4E3As3O5Phj3RZWm/WdgI8D\nPwe+n3kySZIkSRpRlte5m8ugpqXAImBVq9bodXmxJEmSJGkWltc5qEy0dwI+CvwM+GG3tdVmfQFw\nKrAW+Hr26SRJkiRpdFle5+ZTJGdSz+xjUNNRwK7A11u1RtfnwEqSJEmSurO89qky0R4juWTYQU2S\nJEmSlDPLa/8OJhnU1OpMjj/abWG1WX8TcBjwg1atcUcO2SRJkiRppFle+zeXQU319OhZV0mSJEma\nB5bXPlQm2juSDGq6C/inbmurzfrWwMnAb4FLsk8nSZIkSaPP8tqfE4HFwFl9DGo6DngpsLpVa6zL\nPJkkSZIklYDltYcZg5qepccjb6rN+hiwEthAf5cXS5IkSZL6YHnt7UDgzcDFncnxR3qsfQ+wF9Bu\n1Rq/yjyZJEmSJJWE5bW3uQxqWpkeHdQkSZIkSfNoYbcXQwhbkBSxPUkum10aY7x3xuvHAp8DpoAL\nY4xfTr/+eaACLAK+EmM8P4TwRuA8kktq7wRWxhh73T86UJWJ9g7Ax4CfAzd2W1tt1ncAasAvgGuz\nTydJkiRJ5dHrzOsxwOIY4/7A3wCT0y+EEBYAXwQOAfYDVoQQXh5COBjYL33PB4DXp285DfhCjPFA\nYAwYn89vJCNzGdT06XTtqlatsSHzZJIkSZJUIr3K6wHAlQAxxh8D+0y/EGNcD+weY3wS2BFYADwH\nfBi4I4RwGfCd9B+AvWKMN6QfXwF8aL6+iSzMcVDTApJnuz4NnJ99OkmSJEkql17ldTvgiRmfr08v\nJQYgxrghhPBR4FbgOmAtsAOwN8nltqcCF6bLx2b8Ok8B229e9My9HwjAtzqT43/osfZwYDfgwlat\n8VjWwSRJkiSpbLre80pSXJfM+HyLGOOfXBIbY7wkhHApyf2sJwIPAz+PMT4P/CKE8EwIYUeSe12n\nLQH6LXkDuS/24L1ezfdv+TVfXHHACcAJ3da+a+c9uPXBn/E/D/vCKcAp+SQshULfE63Muf/l5d6X\nm/tfXu59ubn/5TTWe8kLepXXNSSDly4OIbwXuH36hRDCdkAHODTGuC6EsBZYTzLY6K+A00IIrwK2\nBv4A3BpCOCjGeD1wBHBNFt/QfKhMtF8B/Ab4t8+vWvOWbve7Vpv1NwB3Az/a7WW77p9XxhKYYgB7\nr8Jw/8vLvS8397+83Ptyc//Vl17l9VLg0BDCmvTzk0MIxwPbxhhXhxAuAG4IITwH3AZcEGOcCiEc\nGEK4ieSy5JXp5cUTwOoQwmLgLuBb2XxL8+KTwJb0N6hpOclvttMzTyVJkiRJJTU2NVXoM/S5/xQm\nHdT0M+ANwC6dyfGHZ1tbbda3An5NcsZ511at8Ww+KUvBn8CVm/tfXu59ubn/5eXel5v7r770GthU\nRgcAbwG+3a24pqrAy4GzLa6SJEmSlB3L659blh7P6mPtSpKfFJ2ZXRxJkiRJkuV1hspE++UkZ1N/\nAVzfbW21WX838G7gu61a4/4c4kmSJElSaVle/9RcBjXV06ODmiRJkiQpY5bXVDqoaRmwDji/29pq\ns/4K4HjgXuB72aeTJEmSpHKzvL5gf+CtwCV9DGo6CXgJ0GjVGhuyDiZJkiRJZWd5fUFfg5qqzfoW\nJJcMPwOcm3UoSZIkSZLlFYDKRPtlJIOa7gG+32P5YSTPgP1mq9Z4JONokiRJkiQsr9M+QXIZcD+D\nmlakRwc1SZIkSVJOSl9eZwxqeo7eg5p2A44CbmrVGjdnn06SJEmSBJZXgPcCbwMu7UyO/67H2uXA\nGLAq81SSJEmSpP/P8tr/oKYtgaXAI0Az61CSJEmSpBeUurxWJtovBWokz2u9rsfyjwM7AOe0ao1n\nss4mSZIkSXpBqcsryaCmrUgGNfV6XusKYAo4I/NUkiRJkqQ/UdryutGgpvO6ra026+8C9gOuaNUa\n92WfTpIkSZI0U2nLK7Av8Hbgsj4GNU0/HsdBTZIkSZI0AGUur/0OanopcALwr8CVWYeSJEmSJP25\nUpbXykR7e+A44D7g2h7LTyK5L/aMVq2xPuNokiRJkqQXUcrySnImdStgdbdBTdVmfQuSS4afBb6a\nUzZJkiRJ0kZKV17TQU3LgefpMagJOAR4E9Bs1RoPZxxNkiRJkjSL0pVX4N3AnkC7Mzn+UI+1DmqS\nJEmSpAIoY3ntd1DTrsDRwM3ATVmHkiRJkiTNrlTltTLR3g44nmRy8D/2WL6c5N/PqlatMZV1NkmS\nJEnS7EpVXkkGNW1N70FNi4FTgEeBi3LKJkmSJEmaRWnK60aDms7tsfyjwE7Aua1a4+mss0mSJEmS\nuitNeQX2Ad4BfKePQU0r0+MZ2UaSJEmSJPWjTOW130FNewLvA65q1Rp3Z55KkiRJktRTKcrrjEFN\n9wPf67Hcx+NIkiRJUsGUorySFNdt6D2oaXvgE8AvgctzyiZJkiRJ6qEs5XUZsJ7eg5o+SVJyz2jV\nGuszTyVJkiRJ6svIl9fKRHtvYC+g05kcf2C2ddVmfYzkkuF1wDk5xZMkSZIk9WHkyyt9DmoCDgbe\nAlzcqjV+l2kiSZIkSdKcjHR5rUy0lwB/SXIP69U9lk8/HsdBTZIkSZJUMCNdXkkGNW1LMqhp1ntY\nq836LsAxwG3AD3PKJkmSJEnq06iX134HNZ0CLABOb9UaU5mnkiRJkiTNyciW13RQ097AdzuT47+Z\nbV21WV9EUnIfB76RUzxJkiRJ0hyMbHklOZsKvQc1HQPsDJzXqjXWZhtJkiRJkrQpRrK8Viba2wIn\nAL8CruqxfHpQUyPTUJIkSZKkTTaS5RU4jmRQ09k9BjXtARwEXNOqNWJe4SRJkiRJczOq5XUZsAH4\nao919fR4erZxJEmSJEmbY+TKa2Wi/S7g3cDlncnxX8+2rtqsLwFOBH4NdHKKJ0mSJEnaBCNXXul/\nUNMngCXAma1a4/lsI0mSJEmSNsdIldfKRHsbklL6a+DK2dZVm/UxYAXwPHB2PukkSZIkSZtqpMor\nyaCmJcA5ncnxbmdT3w+8Dfh2q9Z4KJdkkiRJkqRNNmrldXpQ0zk91q1Ijw5qkiRJkqQhMDLltTLR\nfifwHuAfOpPjv5ptXbVZ3xk4FrgTuDGneJIkSZKkzTAy5ZX+BzUtBRYCq1q1xlS2kSRJkiRJ82Ek\nyuuMQU2/Aa6YbV21WV8ILAeeBC7IJ50kSZIkaXONRHkFqsB29B7UdDSwC/C1Vq3xZC7JJEmSJEmb\nbVTK6zJgiv4HNa3KNo4kSZIkaT4NfXmtTLT3BN4LXNGZHP/lbOuqzfruwCHA91u1xl155ZMkSZIk\nbb6hL6/0P6ipnh496ypJkiRJQ2aoy2tlor018EngQeDy2dZVm/VtgJPSdZflEk6SJEmSNG+GurwC\nHwe2p/egphNIBjqd1ao1nsslmSRJkiRp3gx7eV1OMqjp7NkWVJv1MWAlsJ7elxZLkiRJkgpoaMtr\nZaL9dmA/4MrO5Pj9XZbuB+wJXNqqNR7IJZwkSZIkaV4NbXml/0FNK9Ojg5okSZIkaUgNZXmdw6Cm\nnUjui/058P1cwkmSJEmS5t1QllfgY8BLga92Jse7DWBaCiwCVrVqjalckkmSJEmS5t2wltdlJIOa\nzpltQbVZXwCcCqwFvpZTLkmSJElSBoauvFYm2nsABwBXdybH/7XL0iOBXYGvt2qNJ3IJJ0mSJEnK\nxNCVVxzUJEmSJEmlM1TltTLR3go4Efgt0JltXbVZfxNwGPCDVq1xR07xJEmSJEkZGaryChwLvIze\ng5rq6dGzrpIkSZI0AoatvC5Pj6tnW1Bt1rcGTiY5O3tJHqEkSZIkSdkamvJamWi/FXgfvQc1HUfy\nGJ3VrVpjXS7hJEmSJEmZGprySh+DmqrN+hjJoKYN3dZJkiRJkobLUJTXykT7JcCnSC4F/k6Xpe8B\n9gLarVrjV3lkkyRJkiRlbyjKKy8Majq3x6AmH48jSZIkSSNoWMrrsvR49mwLqs36DkAN+AVwbR6h\nJEmSJEn5KHx5rUy0dwcOBP6xMzl+b5elnwYWA6tatcaGXMJJkiRJknJR+PJKf4OaFpA82/Vp4Pw8\nQkmSJEmS8lPo8rruufWQDGr6PdDusvRwYDfgwlat8Vj2ySRJkiRJeSp0ef3hHQ8CvIJkUFO3Z7au\nSI8OapIkSZKkEVTo8nrlj/5t+sPVs62pNuuvB44AftiqNf45h1iSJEmSpJwVurzeee8fAK7pTI7f\n02XZqcAYcHouoSRJkiRJuSt0eU11G9S0FfAZkntiv5VbIkmSJElSrgpdXrffdjHAZV2WVIGXA2e3\nao1ncwklSZIkScrdwkEH6OZD734NJx21R69BTVPAmTlFkiRJkiQNwNjU1NSgM8xq3XPrpxYvWjD2\nYq9Vm/V9gJ8AnVatcXS+yZSDKZJ7mVVO7n95uffl5v6Xl3tfbu6/+lLoy4YXL1rQ7eXpx+M4qEmS\nJEmSRlyhy+tsqs36K4DjgXuA7w04jiRJkiQpY0NZXoGTgJcAjVatsWHAWSRJkiRJGRu68lpt1rcA\n6sAzwHmDTSNJkiRJysPQlVfgMOANwDdbtcYjgw4jSZIkScreMJZXBzVJkiRJUskU+lE5bDQ2u9qs\n7wbcB/ykVWvsO6hQyoUj08vN/S8v977c3P/ycu/Lzf1XX4btzOtykv+wPesqSZIkSSWysNuLIYQt\ngFXAnsCzwNIY470zXj8W+BzJT0sujDF+Of36LcDj6bL7YoyfCSG8C+gAd6dfb8QYW/0GrTbrWwJL\ngUeAvt8nSZIkSRp+XcsrcAywOMa4fwhhX2Ay/RohhAXAF4G9gbXAXSGEC4CnAWKMH9jo19obOC3G\neNomZv04sAPwpVat8cwm/hqSJEmSpCHU67LhA4ArAWKMPwb2mX4hxrge2D3G+CSwI7AAWAe8A9g6\nhHBVCOGatPRCUl6PDCFcH0I4O4Sw7RyzriA5w3vGHN8nSZIkSRpyvcrrdsATMz5fn15KDECMcUMI\n4aPArcB1JGdd1wJfijF+GDgVuDA9S/tj4D/HGA8iGbr0X/sNWW3W3wXsB1zRqjXu6/d9kiRJkqTR\n0Ouy4SeAJTM+3yLGuGHmghjjJSGES4HzgBOBbwD3pK/dHUL4A/BK4NIY4/R9sJcBX+4z49QHX38A\n1963hr95/4qPkJx9VTm41+Xm/peXe19u7n95uffl5v6X05ymTPcqr2uACnBxCOG9wO3TL4QQtiMZ\nwHRojHFdCGEtsB44mWTA08oQwqtIyu9DwI0hhP8YY/wJcAjw034CVpv1lwEPAA/9/Q9WvalVa6yf\nyzeooeXI9HJz/8vLvS8397+83Ptyc//Vl17l9VLg0BDCmvTzk0MIxwPbxhhXpwOabgghPAfcBlxA\ncu/ruSGEG9L3fDrGuD6EcCpwerr2QWBZnxlPArYCzrC4SpIkSVI5jU1NFfcM/YapDVPHtVbeDbwG\neHWr1nh40JmUG38CV27uf3m59+Xm/peXe19u7r/60mtg00Dd+dsI8CagaXGVJEmSpPIqdHm96p7r\npz9cNcgckiRJkqTBKnR5/ekDtwPcDNw04CiSJEmSpAEqdHlN78dd1ao1intjriRJkiQpc4Uur9ss\n3hrgokHnkCRJkiQNVq9H5QzUB1+3P59857FPDzqHJEmSJGmwCv2onHXrn5tavGCRY7PLyZHp5eb+\nl5d7X27uf3m59+Xm/qsvhb5sePGCRYOOIEmSJEkqgEKXV0mSJEmSwPIqSZIkSRoClldJkiRJUuFZ\nXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4\nlldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQV\nnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJ\nhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJ\nUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJ\nklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJ\nkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJ\nkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJ\nkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVV\nkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5\nlSRJkiQVnuVVkiRJklR4lldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4lldJkiRJUuFZ\nXiVJkiRJhbew24shhC2AVcCewLPA0hjjvTNePxb4HDAFXBhj/HL69VuAx9Nl98UYPxNCeCNwHrAB\nuBNYGWOcmt9vR5IkSZI0irqWV+AYYHGMcf8Qwr7AZPo1QggLgC8CewNrgbtCCBcATwPEGD+w0a91\nGvCFGOMNIYQGMA5cNm/fiSRJkiRpZPW6bPgA4EqAGOOPgX2mX4gxrgd2jzE+CewILADWAe8Atg4h\nXBVCuCYtvQB7xRhvSD++AvjQ/H0bkiRJkqRR1qu8bgc8MePz9emlxADEGDeEED4K3ApcR3LWdS3w\npRjjh4FTgQvTs7RjM36dp4Dt5yG/JEmSJKkEepXXJ4AlM9fHGDfMXBBjvATYBdgSOBH4BXBh+trd\nwB+AnUnudZ22BHisj3xjvZdoRLn35eb+l5d7X27uf3m59+Xm/qsvvcrrGuAjACGE9wK3T78QQtgu\nhHB9CGFxOnhpLbAeOJnk3lhCCK8iKaoPAreGEA5K334EcAOSJEmSJPVhbGpq9oG/IYQxXpg2DEkx\n3RvYNsa4OoRwCvAZ4DngNuA/kNz7ei7w2vQ9n40x/iiE8CZgNbAYuAs4xWnDkiRJkqR+dC2vkiRJ\nkiQVQa/LhiVJkiRJGjjLqyRJkiSp8CyvkiRJkqTCWzjoABtLnyM7PSTqWWBpjPHewaZSXkIIi4Cv\nkgz82hL4HzHGzmBTKU8hhJ2Am4FDYoy/GHQe5SeE8HmgAiwCvhJjPH/AkZSD9O/9s4E3kzxW75QY\nYxxsKuUhhLAv8Pcxxg+EEN4InEfy38CdwEoHe46ujfb+ncCXSZ5a8ixwYozxdwMNqEzN3P8ZX/tL\n4N/HGPfv9t4innk9BlicBv8b0sfuqDROAH4fYzwQOBz4yoDzKEfpDy/OJHn0lkokhHAwsF/6Z//B\nwOsHGkh5OgzYJsb4PuC/A3834DzKQQjhsyRPodgy/dJpwBfSv//HgPFBZVO2XmTv/zdJafkAcAnw\nuUFlU/ZeZP8JIbwL+HQ/7y9ieT0AuBIgxvhjYJ/BxlHOLgb+Nv14C+D5AWZR/r4ENEieDa1yOQy4\nI4RwGdABvjPgPMrPH4Ht08fzbQ+sG3Ae5eMe4KMkRRVgrxjjDenHVwAfGkgq5WHjvT8uxnh7+vEi\nkj8TNLr+ZP9DCK8g+aHlf+KF/yZmVcTyuh3wxIzP16eXFKkEYoxrY4xPhRCWkBTZ/zLoTMpHCOEk\nkrPuV6df6vkHmEbKjiTPEf8YcCpw4WDjKEdrgJcA/0Jy5cX/HWwc5SHGeAl/+gPqmX/mP0XygwyN\noI33Psb4EEAIYX9gJfC/BhRNOZi5/2nHOwf4a5Lf9z0VsRQ+ASyZ8fkWMcYNgwqj/IUQdgWuBb4W\nY7xo0HmUm5OBQ0MI1wHvBM4PIfzFgDMpPw8DV8cYn0/vdX4mhLDDoEMpF58F1sQYAy/83l884EzK\n38z/11sCPDaoIMpfCKFGcuXVR2KMfxh0HuVmb+CNJHv/TeCtIYTTur2hcAObSH4CWwEuDiG8F7i9\nx3qNkLSsXA2siDFeN+g8yk+M8aDpj9MCuzzG+NsBRlK+bgT+CjgthPAqYBvA/4Eph2144YqrR0ku\nG1wwuDgakFtDCAfFGK8HjgCuGXQg5SOE8AlgGXBwjPHRQedRfmKMPwHeBhBCeC1wUYzxr7u9p4jl\n9VKSsy9r0s9PHmQY5e4LJJcK/W0IYfre1yNijM8MMJOkjMUYLw8hHBhCuInkqqAVThotjS8B54YQ\nfkBSXD8fY/Set/KY/n0+AaxOz7rfBXxrcJGUk6n0stH/A9wPXBJCALg+xvjfBhlMudj47/ixF/na\nnxmbmvL/DSRJkiRJxVbEe14lSZIkSfoTlldJkiRJUuFZXiVJkiRJhWd5lSRJkiQVnuVVkiRJklR4\nlldJkiRJUuFZXiVJkiRJhWd5lSRJkiQV3v8DX0uMt7HT8bQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11258e940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "lsgd_model_synth = SGDClassifier(loss='log')\n",
    "lsgd_scores = {'train':[], 'valid':[]}\n",
    "for i in range(15):\n",
    "    lsgd_model_synth.partial_fit(X_train, y_train, classes=np.arange(12))\n",
    "    lsgd_scores['train'].append(lsgd_model_synth.score(X_train, y_train))\n",
    "    lsgd_scores['valid'].append(lsgd_model_synth.score(X_valid, y_valid))\n",
    "    print('iter:', i, 'validation score:', lsgd_scores['valid'][-1])\n",
    "plot(lsgd_scores['train'], label='train')\n",
    "plot(lsgd_scores['valid'], label='validation')\n",
    "legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that the logistic regression trained with SGD (full-batch) obtains a very similar result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training score 0.39647411853\n",
      "validation score 0.392221356646\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "tree_model = DecisionTreeClassifier()\n",
    "tree_model = tree_model.fit(X_train, y_train)\n",
    "print('training score', tree_model.score(X_train, y_train))\n",
    "print('validation score', tree_model.score(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also a simple decision tree gives a similar result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence of multiple chords as an input feature\n",
    "\n",
    "As we could see trying to classify the key based on a single chord gives not completely bad results, it is very limited. Naturally even humans perceive the key in a context of multiple chords. So we should now move to providing input features that consist of a sequence of multiple chords. We expect much better results.\n",
    "\n",
    "In order to keep things simple we restrict our problem such that each sequence of adjacent chords lies within a single key, ie. no sequence spans multiple keys. We then classify the key for for the whole sequence.\n",
    "\n",
    "A question is how the context size affects the accuracy. What is a good tradeoff between accuracy and redundancy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "- prepare chord blocks\n",
    "  - put several adjacent chords within a single key into each data point\n",
    "  - chord block are expected to carry enough information to enable more reasonable key classification\n",
    "  - use the synthetic unskewed data\n",
    "- try the same model training on the chord block data\n",
    "  - LogisticRegression [with cross-validated grid search]\n",
    "  - try some tree-based model - eg. RandomForestClassifier\n",
    "  - try deep neural networks\n",
    "- explore the effect of context size\n",
    "- try RNNs without the need on the full sequences (no need for explicit windowing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(103965, 192), (103965,), (34656, 192), (34656,), (34656, 192), (34656,)]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = pd.read_csv('data/beatles/derived/all_chords_with_keys_synth_rolling_16.tsv', sep='\\t')\n",
    "X, y = dataset.iloc[:,:-1].astype(int32), dataset.iloc[:,-1].astype(int32)\n",
    "((X_train, y_train), (X_valid, y_valid), (X_test, y_test)) = split_dataset(X, y)\n",
    "\n",
    "[d.shape for d in (X_train, y_train, X_valid, y_valid, X_test, y_test)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model = LogisticRegression()\n",
    "lr_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.72085798105131538, 0.71592220683287167)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.score(X_train, y_train), lr_model.score(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2U1nWd//HXNcO9SMZpipNWGIdjdueRGhZOSh1+VFPb\nHasWCGQbbbqt1YLZDimOd60onXIzSsrq2GTi0kHStpsN9CwkrbAnZFOym12xMhDQrQZEGZjr94en\nyTvki3oxn4HH46+Za5jv9329GZCn3+9cU6vX6/UAAABAQZr6egAAAAB4IrEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHEGNOrAPT09Oe+887Jp06Y0NTXlkksuSXNzc9rb29PU1JSxY8emo6Mj\ntVqtUSMAAADQTzUsVn/84x9n165duf7667NmzZp87nOfy549ezJ37ty0tramo6MjK1euzJQpUxo1\nAgAAAP1Uw24DHjJkSLq6ulKv19PV1ZWBAwfmrrvuSmtra5Jk0qRJWbNmTaNODwAAQD/WsCur48aN\ny+7du9PW1pY//OEPufrqq7Nu3brejw8bNixdXV2NOj0AAAD9WMNi9Zprrsm4ceMyZ86cbNmyJe9/\n//uzZ8+e3o/v3LkzI0aM2O9x6vW672sFAAA4zDQsVnft2pUjjjgiSTJixIjs2bMnr3zlK7N27dqM\nHz8+q1atysSJE/d7nFqtlm3bXIHdn5aWI+2pIruqxp6qsafq7Koae6rOrqqxp2rsqTq7qsaeqmtp\nOfIpH29YrM6ePTvz5s3L6aefnj179uScc87Jq171qsyfPz/d3d0ZM2ZM2traGnV6AAAA+rGGxeqI\nESOyaNGiJz3e2dnZqFMCAABwiGjYqwEDAADAMyVWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDjFx2q9Xu/rEQAAADjIio/VpqZ6TjlloGgFAAA4jBQfq0lTVq8enBNO\nGJr16wUrAADA4aAfxGqS1LJly8CcccZQV1gBAAAOA/0kVgEAADicDOjrAaqpZ9SoPbn22l2p1Wp9\nPQwAAAAN1i9iddSoPdmwQagCAAAcLoq/Dbi5ue6KKgAAwGGm+Fg95pjkxBOFKgAAwOGk+FhNhCoA\nAMDhph/EKgAAAIcbsQoAAEBxxCoAAADFaeiPrrnxxhuzbNmyJMkjjzySu+++O9/61rfy6U9/Ok1N\nTRk7dmw6Ojq80i8AAACP09Arq1OnTk1nZ2c6Ozvz6le/OvPnz8+iRYsyd+7cXHfddanX61m5cmUj\nRwAAAKAfOii3Af/sZz/Lr3/965x22mm566670tramiSZNGlS1qxZczBGAAAAoB85KLG6ePHinH32\n2UmSer3e+/iwYcPS1dV1MEYAAACgH2no96wmyZ/+9Kds2rQp48ePT5I0Nf2lj3fu3JkRI0bs9xgt\nLUc2bL5DiT1VZ1fV2FM19lSdXVVjT9XZVTX2VI09VWdX1djTs9PwWF23bl0mTJjQ+/7xxx+ftWvX\nZvz48Vm1alUmTpy432Ns2+bq6/60tBxpTxXZVTX2VI09VWdX1dhTdXZVjT1VY0/V2VU19lTdvqK+\n4bG6adOmvPSlL+19v729PfPnz093d3fGjBmTtra2Ro8AAABAP9PwWJ09e/bj3h89enQ6OzsbfVoA\nAAD6sYPyAksAAABwIMQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUJwBjTz44sWLc+utt6a7uzszZ87MuHHj0t7enqampowd\nOzYdHR2p1WqNHAEAAIB+qGFXVm+//fasX78+S5YsSWdnZ377299mwYIFmTt3bq677rrU6/WsXLmy\nUacHAACgH2tYrN5222057rjj8pGPfCRnnXVWJk+enLvuuiutra1JkkmTJmXNmjWNOj0AAAD9WMNu\nA37wwQezefPmLF68OL/97W9z1llnpV6v93582LBh6erqatTpAQAA6McaFqvPf/7zM2bMmAwYMCDH\nHntsBg8enK1bt/Z+fOfOnRkxYkSlY7W0HNmoMQ8p9lSdXVVjT9XYU3V2VY09VWdX1dhTNfZUnV1V\nY0/PTsNi9XWve12+8Y1v5G//9m9z//335+GHH86ECROydu3ajB8/PqtWrcrEiRMrHWvbNldg96el\n5Uh7qsiuqrGnauypOruqxp6qs6tq7Kkae6rOrqqxp+r2FfUNi9U3velNWbduXU499dT09PSko6Mj\nRx99dObPn5/u7u6MGTMmbW1tjTo9AAAA/VhDf3TNueee+6THOjs7G3lKAAAADgENezVgAAAAeKbE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRnQKNPMHXq1AwfPjxJ8pKXvCRnnnlm2tvb09TUlLFjx6ajoyO1Wq3RYwAAANCPNDRWH3nk\nkSRJZ2dn72NnnXVW5s6dm9bW1nR0dGTlypWZMmVKI8cAAACgn2nobcB33313du3aldmzZ+eMM87I\nHXfckY0bN6a1tTVJMmnSpKxZs6aRIwAAANAPNfTK6tChQzN79uycdtpp2bRpUz70oQ897uPDhg1L\nV1dXI0cAAACgH2porI4ePTove9nLet8+6qij8vOf/7z34zt37syIESP2e5yWliMbNuOhxJ6qs6tq\n7Kkae6rOrqqxp+rsqhp7qsaeqrOrauzp2WlorC5btiy/+MUv0tHRkfvvvz87d+7MG97whqxduzbj\nx4/PqlWrMnHixP0eZ9s2V1/3p6XlSHuqyK6qsadq7Kk6u6rGnqqzq2rsqRp7qs6uqrGn6vYV9Q2N\n1VNPPTXz5s3LjBkzkiSXXXZZjjrqqMyfPz/d3d0ZM2ZM2traGjkCAAAA/VBDY3XAgAFZuHDhkx5/\n7KsDAwAAwBM19NWAAQAA4JkQqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByx\nCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxDihWH3744ezYseOATvDAAw/kjW98Y+65557c\ne++9mT59embMmJELL7ww9Xr9gI4FAADA4aFyrC5dujTvfe97M23atFx55ZWVPqe7uzsXXHBBhg4d\nmnq9nssuuyxz587Nddddl3q9npUrVz7jwQEAADh07TNWf/nLXz7u/ZUrV+amm27Kd7/73axYsaLS\nwa+44opMnz49LS0tSZKNGzemtbU1STJp0qSsWbPmmc4NAADAIWzAvj5www03pLu7O//wD/+QF73o\nRXnFK16R2bNnZ8CAARk7dux+D7xs2bKMHDkyJ510UhYvXpx6vf64236HDRuWrq6u5+ZZAAAAcEip\n1Z/mG0fvueeeLFq0KC9+8Yvzd3/3d9m2bVt2796dV7ziFfs98MyZM1Or1ZIkd999d0aPHp2f//zn\nufPOO5MkK1asyE9+8pPMnz//aY8zenSyaVP1JwQAAED/97Sx+mc//elP8+UvfzkTJkzIjBkzMnDg\nwAM6yaxZs3LRRRfliiuuyAc/+MGMHz8+F1xwQSZOnJi3ve1tT/u5o0cn69a5Ars/LS1HZts2e6rC\nrqqxp2rsqTq7qsaeqrOrauypGnuqzq6qsafqWlqOfMrH9/k9q9ddd12mTJmSt771rdm6dWuuvvrq\nvPjFL85ZZ52Vm2666YAHqNVqaW9vz1VXXZVp06Zl7969aWtrO+DjAAAAcOjb5/esXn/99fnBD36Q\n3bt3Z8aMGWlra8tb3vKWTJ48Of/6r/96QCfp7Ox8yrcBAADgqewzVltaWvLP//zPefjhh/Pyl7/8\nL58wYEBOP/30gzIcAAAAh6d9xurVV1+d1atXZ9CgQXnDG95wMGcCAADgMLfPWB08eHCmTJlyMGcB\nAACAJE/zAksAAADQV8QqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAcQY08uB79+7N+eefn02bNqVWq+Wiiy7KoEGD0t7enqampowdOzYdHR2p1WqNHAMAAIB+\npqGxeuutt6apqSnXX3991q5dm89+9rNJkrlz56a1tTUdHR1ZuXJlpkyZ0sgxAAAA6GcaehvwlClT\ncvHFFydJ7rvvvjzvec/LXXfdldbW1iTJpEmTsmbNmkaOAAAAQD/U8O9ZbW5uTnt7ez796U/nne98\nZ+r1eu/Hhg0blq6urkaPAAAAQD9Tqz+2Hhto+/btOe200/LQQw/l9ttvT5KsWLEiP/nJTzJ//vx9\nft7o0cmmTQdjQgAAAErR0O9ZXb58ee6///6ceeaZGTJkSJqamvLqV786a9euzfjx47Nq1apMnDhx\nv8fZts3V1/1paTnSniqyq2rsqRp7qs6uqrGn6uyqGnuqxp6qs6tq7Km6lpYjn/LxhsZqW1tb2tvb\nM3PmzOzZsyfnnXdeXv7yl2f+/Pnp7u7OmDFj0tbW1sgRAAAA6IcaGqtDhgzJlVde+aTHOzs7G3la\nAAAA+rmGv8ASAAAAHCixCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMUZ0KgDd3d351Of+lR+//vfZ/fu3fn7v//7jBkzJu3t7WlqasrY\nsWPT0dGRWq3WqBEAAADopxoWqzfffHNGjhyZhQsX5o9//GPe/e535/jjj8/cuXPT2tqajo6OrFy5\nMlOmTGnUCAAAAPRTDbsNuK2tLR/72MeSJD09PRkwYEA2btyY1tbWJMmkSZOyZs2aRp0eAACAfqxh\nsTps2LAcccQR2bFjRz7+8Y/nH//xH9PT0/O4j3d1dTXq9AAAAPRjDbsNOEk2b96cs88+OzNmzMg7\n3vGOLFy4sPdjO3fuzIgRIyodp6XlyEaNeEixp+rsqhp7qsaeqrOrauypOruqxp6qsafq7Koae3p2\nGhar27dvzwc/+MF0dHRkwoQJSZLjjz8+a9euzfjx47Nq1apMnDix0rG2bXMFdn9aWo60p4rsqhp7\nqsaeqrOrauypOruqxp6qsafq7Koae6puX1HfsFi9+uqr09XVlUWLFmXRokVJkvPOOy+f/vSn093d\nnTFjxqStra1RpwcAAKAfa1isnn/++Tn//POf9HhnZ2ejTgkAAMAhomEvsAQAAADPlFgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YvUQUK/XU6/X+3oMAACA54xY7cfq9XpOOWVgXvSiI9LU\n9OjbohUAADgUiNV+7NRTB2X16sFJmpM0ZfXqwTnhhKFZv16wAgAA/ZtY7afq9XpWrx6YpPaYR2vZ\nsmVgzjhjqCusAABAvyZWAQAAKI5Y7adqtVpOPrk7yWOvoNYzalR3rr12V2q12r4+FQAAoHgNj9UN\nGzZk1qxZSZJ7770306dPz4wZM3LhhRe6VfVZ+va3d2fUqD29748atScbNuzKiScKVQAAoH9raKx+\n5Stfyfnnn5/u7u4kyWWXXZa5c+fmuuuuS71ez8qVKxt5+kNerVbLtdfuSlNTPc3NdVdUAQCAQ0ZD\nY/VlL3tZvvCFL/ReQd24cWNaW1uTJJMmTcqaNWsaefrDwokn1nL00fUcc0xcUQUAAA4ZDY3Vt7zl\nLWlubu59/7G3/Q4bNixdXV2NPP1hRqgCAACHjgEH82RNTX9p4507d2bEiBGVPq+l5chGjXRI+PNa\n7ak6u6rGnqqxp+rsqhp7qs6uqrGnauypOruqxp6enYMaq8cff3zWrl2b8ePHZ9WqVZk4cWKlz9u2\nzRXYp9PTc0SamprsqaKWliPtqgJ7qsaeqrOrauypOruqxp6qsafq7Koae6puX1F/UGL1zy/6097e\nnvnz56e7uztjxoxJW1vbwTg9AAAA/UzDY/WYY47JkiVLkiSjR49OZ2dno08JAABAP9fwn7MKAAAA\nB0qsAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6xyWKnX\n66nX6309BgAAsB9ilcNCvV7PKacMzItedESamh59W7QCAEC5xCqHhVNPHZTVqwcnaU7SlNWrB+eE\nE4Zm/XrBCgAAJRKrHPLq9XpWrx6YpPaYR2vZsmVgzjhjqCusAABQILEKAABAccQqh7xarZaTT+5O\n8tgrqPWMGtWda6/dlVqttq9PBQAA+ohY5bDw7W/vzqhRe3rfHzVqTzZs2JUTTxSqAABQIrHKYaFW\nq+Xaa3elqame5ua6K6oAAFC4AX09ABwsJ55Yy9FH19PUVHNFFQAACufKKochoQoAAKUTqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByx\nCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQq\nAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasA\nAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCT1Kv11Ov1/t6DAAADmNiFehVr9dzyikD86IXHZGmpkff\nFq0AAPQFsQr0OvXUQVm9enCS5iRNWb16cE44YWjWrxesT8UV6OrsCgA4UGIVSPJoTKxePTBJ7TGP\n1rJly8CcccZQofEYrkBXZ1cHRtQDwF+IVYAD5Ap0dXZVjag/cMIe4NAnVoEkSa1Wy8kndyd57D/+\n6hk1qjvXXrsrtVptX596WHEFujq7qk7UVyfsAQ4fYhXo9e1v786oUXt63x81ak82bNiVE08UqtAo\nov7ACHuAw4dYBXrVarVce+2uNDXV09xcd0X1KbgCXZ1d8VwT9gCHF7EKPM6JJ9Zy9NH1HHNMXFHd\nB1egq7Or/RP1APDUxCqwD/6BvC+uQFdnV9WI+mqEPcDhRawCPAOuQFdnV/sn6qsT9gfGqyZXY09Q\nJrEK8Kz4B3J1dvV0RH01wr4ar5pcjT1B2cQqABRFeO2PsN8/r5pcjT1B2cQqANBPCdWn4lWTq7Gn\nZ8Yt0xxMYhUAAHhabpmmL4hVAIBDiFdNrsaeDoxbpg+MK9DPjYMeqz09Pbngggsybdq0zJo1K7/5\nzW8O9ggAAIc0r5pcjT1V45bp6lyBfm4d9FhdsWJFuru7s2TJknziE5/IggULDvYIAACHNK+aXI09\n8VxzBfo1kFXlAAAUSElEQVS5ddBj9ac//WlOPvnkJMkJJ5yQO++882CPAABwyPOqydXY0/65Zboa\nV6CfewMO9gl37NiR4cOH977f3Nycnp6eNDU9dTf/+HejM/J1fmOfzm2/r6WW2FMFdlWNPVVjT9XZ\nVTX2VJ1dVWNP1djT/t2a5PdNteztefT95qbkxQPryYf6dKzi3POYUF2a0/LJLOzDafq/gx6rw4cP\nz86dO3vff7pQTZJj9mw6CFP1by/p6wH6Ebuqxp6qsafq7Koae6rOrqqxp2rsqZot6/bmPe959O3l\ny5OXtDb37UAFmv3/9uSWW5rzl6ur9bz4xT1ZvrwpL3zhiL4crV866LE6bty43HrrrXnb296WO+64\nI8cdd9zBHgEAADhAra3Nue++vp6ibCtXPjGvann0+1d5Jmr1g3zzdL1ez4UXXphf/OIXSZLLLrss\nxx577MEcAQAAgMId9FgFAACA/TnorwYMAAAA+yNWAQAAKI5YBQAAoDhFxmpPT08uuOCCTJs2LbNm\nzcpvfvObvh6paFOnTs2sWbMya9asfOpTn+rrcYp044039u7ove99b1772tdmx44dfT1Wkfbu3Zt5\n8+Zl+vTpOf300/OrX/2qr0cqzoYNGzJr1qwkyb333pvp06dnxowZufDCC/3A7yfYvXt3PvnJT2ba\ntGmZOXNm7r777r4eqXiP/friybq7u3PuuedmxowZOe2003LLLbf09UhFW7x4caZNm5ZTTjklN954\nY1+PU6Senp7e/+7NmDEj//u//9vXIxXtgQceyBvf+Mbcc889fT1KkZ74d/iPfvSjnHPOOX04Uf92\n0H90TRUrVqxId3d3lixZkg0bNmTBggX54he/2NdjFemRRx5JknR2dvbxJGWbOnVqpk6dmiS5+OKL\nc9ppp2X48OF9PFWZbr311jQ1NeX666/P2rVr87nPfc6fv8f4yle+kptuuilHHHFEkkdf0Xzu3Llp\nbW1NR0dHVq5cmSlTpvTxlOVYunRphgwZkiVLluSee+7JOeeck2XLlvX1WMV64tcXT3bzzTdn5MiR\nWbhwYf74xz/mPe95TyZPntzXYxXp9ttvz/r167NkyZI89NBDueaaa/p6pCL9+Mc/zq5du3L99ddn\nzZo1ufLKK/P5z3++r8cqUnd3dy644IIMHTq0r0cp0hP/Dr/00ktz22235ZWvfGUfT9Z/FXll9ac/\n/WlOPvnkJMkJJ5yQO++8s48nKtfdd9+dXbt2Zfbs2TnjjDOyYcOGvh6paD/72c/yq1/9Kqeddlpf\nj1KsKVOm5OKLL06S3HfffXne857XxxOV5WUve1m+8IUv9F5B3bhxY1pbW5MkkyZNypo1a/pyvOL8\n+te/zqRJk5Ikxx57bO6//353NTyNJ3598WRtbW352Mc+luTRK2LNzX5+4b7cdtttOe644/KRj3wk\nZ511lqjfhyFDhqSrqyv1ej1dXV0ZOHBgX49UrCuuuCLTp09PS0tLX49SpCf+HT5u3Dh3XT1LRcbq\njh07HnfVq7m5OT09PX04UbmGDh2a2bNn56tf/WouuuiifOITn7Crp7F48eJ89KMf7esxitfc3Jz2\n9vZceumlecc73tHX4xTlLW95y+P+cfzY/wANGzYsXV1dfTFWsY4//vjceuutSZI77rgjDz74YB56\n6KE+nqpcT/z64smGDRuWI444Ijt27MjHP/7xzJkzp69HKtaDDz6YO++8M5///Od7/43Ak40bNy67\nd+9OW1tbLrjggsycObOvRyrSsmXLMnLkyJx00klJIsCewhP/Dn/729/eh9McGoqM1eHDh2fnzp29\n7/f09KSpqchR+9zo0aPzrne9q/fto446Ktu2bevjqcr0pz/9KZs2bcr48eP7epR+YcGCBfnhD3+Y\n+fPn5+GHH+7rcYr12L+bdu7cmREjRvThNOU55ZRTMnz48Jx++ulZsWJF799T8Gxs3rw5Z5xxRt7z\nnvfkr//6r/t6nGI9//nPz0knnZQBAwbk2GOPzeDBg/Pggw/29VjFueaaazJu3Lj88Ic/zHe+8520\nt7dn9+7dfT1WcZYtW5Y1a9Zk1qxZufvuu9Pe3p7t27f39Vgc4ooswHHjxmXVqlVJHv0/8ccdd1wf\nT1SuZcuWZcGCBUnSe3udWzOe2rp16zJhwoS+HqN4y5cvz+LFi5M8emtUrVbzP4uexvHHH5+1a9cm\nSVatWpXXv/71fTxRWf77v/87EyZMyLe+9a289a1vTUtLSwYNGtTXY9GPbd++PR/84Adz7rnn5m/+\n5m/6epyive51r8vq1auTPPpvhF27duX5z39+H09Vnl27dvV+j+GIESPS3d3tLrWn8M1vfjOdnZ3p\n7OzMK17xilx++eV5wQte0NdjcYgr8gWW3vzmN+e2227LtGnTkjz6AiY8tVNPPTXz5s3LjBkzkjy6\nK2Hx1DZt2pSXvvSlfT1G8dra2tLe3p6ZM2dmz549Oe+888TFU6jVakmS9vb2zJ8/P93d3RkzZkza\n2tr6eLKyHHvssZkzZ04WL16cQYMG5dJLL+3rkfqFP3998WRXX311urq6smjRoixatCjJo1fGBg8e\n3MeTledNb3pT1q1bl1NPPTU9PT3p6OjwtfUUZs+enXnz5uX000/Pnj17cs4552TIkCF9PRb92GP/\nnNVqNX/unoVa3Q3nAAAAFMYlOAAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgF\noDgrV65MZ2dnkmTWrFl9PM2zc/HFF+fGG2/M1q1b8+EPf/hpf+2BPtef/exnT/k5s2bNytq1aw/o\nWM/EkiVLcsMNNyRJ5s2bl82bN+/z1y5btizz5s17Rud5Lp/PXXfdlYULFz4nxwKgscQqAEXZvXt3\nvvKVr+T0009Pkqxbt66PJ3p2/vzD4F/4whfmy1/+8tP+2ufyuR6MH0I/bdq0vO9970uS3H777enp\n6WnYPM/V83nVq16VLVu25Je//OVzcjwAGmdAXw8AQP/w5S9/OT/4wQ+yd+/enHTSSTn33HPzu9/9\nLh/60IcycuTIDB48OO9617uybNmy/OEPf8jkyZMza9asnHfeedm8eXMGDBiQOXPm5OSTT85VV12V\nO+64I1u2bMnMmTMzffr03vPcdNNNGT9+fJqbm3PppZcmSd73vvflhhtuyIQJE/LqV78627dvzyc/\n+cl86Utf6r0C297enr/6q7/K1KlTs3z58nzjG99IT09PXvWqV6WjoyODBg2q9Dxvv/32fOlLX0qS\nbNmyJa997Wtz6aWXZuvWrb3PdciQIbnmmmty+eWXZ926ddm7d2+mTp2aD3zgA6nX67niiityyy23\n5AUveEEGDhyY17zmNfnd736X97///bnlllty3333Zd68efm///u/DBkyJJdeemmWLl36uOe6atWq\nXHXVVdmzZ0+OOeaYXHLJJTnqqKNy2223ZcGCBRk4cGDGjh37tM/lgQceyAc+8IHMmTMnkydPfsrf\nwyuvvDL1ej1z5sxJ8ugV0pNPPjlvf/vbe4/T3t6egQMHZuPGjdmxY0c+8pGP5N3vfneuuuqq1Gq1\nDBo0KFu3bs2ZZ56Zb37zm9m4cWMuv/zy9PT05Oijj85nPvOZ1Ov13HvvvZk1a1Y2b96ciRMn5pJL\nLnnSzAsXLsyKFSsyYMCAvO9978v73//+3o/t3bs3HR0d+fWvf53t27fn2GOPzRe+8IV0d3dn7ty5\n2b59e5Lk7LPPzuTJk/P1r389y5cvT1NTU17zmtfk4osvTpK8853vzNe+9rUsWLCg0tcEAH3DlVUA\n9mvVqlW566678u1vfzs33nhjtmzZkptuuilJsmnTpnzmM5/J17/+9dTr9WzdujXf+c53MmfOnFxy\nySWZOHFibrrppvzLv/xLPvWpT+WBBx5IknR3d+ff/u3fHheqSXLrrbfm9a9/fZLk/PPPT5LeW03/\n8Ic/5Mwzz8zy5cvT3Nz8uM+r1Wqp1Wr51a9+laVLl2bJkiVZvnx5Ro4cma9+9asH9Hw3bNiQiy++\nON///vfzyCOP5Fvf+tbjnuvXvva13HDDDanValm2bFmWLl2alStX5r/+67/y7//+77nzzjvzve99\nL1/84hfzm9/85knHv+iii9LW1pabb745Z599dq6++urHPdcHH3wwn/3sZ/O1r30tN954Y97whjfk\nM5/5THbv3p1/+qd/yuc+97ksW7Ysw4cP3+dz+OMf/5gPf/jD+ehHP5rJkyfv8/fwlFNOyXe/+90k\nyUMPPZT//M//zJvf/OYn7Xbr1q1ZunRpvvGNb+SKK67I9u3be692fvjDH+69cjxs2LCce+65ufzy\ny3PzzTfnuOOOy/Lly1Or1bJ58+YsWrQo3/ve97Jq1ar8z//8z+PO8/3vfz/r16/Pd7/73SxdujTL\nli3rDdB6vZ7169dn8ODBWbJkSX70ox/l4Ycfzn/8x3/8//buJSSqvw3g+PfoTBlFZqZWmnZFsUjG\ntCzBS2OlheXd0oVCupBSAkOIylyoiyYKklxkBQVdjBIj0CjLIsIGM1IjKiiUGkQjdTEq6lzehXRe\np5mS/6b8v+/z2Z0553c9g8Nznt/5SUtLCwEBATQ0NGAwGOjo6MBqtXLx4kUaGhpoaGjAzc2N/v5+\nACIiImhtbf1H3wkhhBB/nmRWhRBCzKitrY2uri7S0tIAGB8fJyAggE2bNuHt7c3y5cvVa0NDQ3Fz\nm3oWajQaqaqqAmDFihWEhYXR2dmJoiiEhYW5bKu3t5elS5f+si+/KgdTAY3RaKS3t5esrCxgKihe\nv379Pxrv1q1bCQwMBGDfvn3cvn2bHTt2OIy1ra2N9+/f8/LlSwDGxsb4+PEjnz59YteuXbi7u+Pp\n6Yler3eqv729nXPnzgEQGxtLbGysw/nOzk76+vrU91GtViuLFi3i48eP+Pr6snbtWgAyMjKorq52\nOYaKigp8fHzYuXOn2l9X93Dv3r34+/vT3t6OyWQiLi4OrVbrVF9mZiZubm74+fkRHh5OR0eHy3Y/\nfPiAn58fISEhAGrGtqGhgYiICBYuXAhAYGAgQ0NDDmVfvXrF7t270Wq1aLVaGhsb1XOKohAREYGn\npyfXr1/n8+fP9Pb2Mjo6ik6n4+zZs/T39xMXF0dRURHu7u7odDrS09PR6/Xk5ubi5+cHwIIFC7Db\n7QwPD7No0SKX4xBCCPH3SbAqhBBiRjabjby8PPLz84GprJ1Go2FoaIi5c+eq1ymKgoeHh3pst9sd\n6rHb7VitVgCHctMpiuKUNZ3ux3Len99hnJycVPuamJioZipHRkbUNn948uQJ58+fB0Cv11NcXOxw\nXqP578+jzWZT+zO9zzabjbKyMhISEgAYHBxk/vz5GAwGh3c3XY1Fq9U6zM2nT59Ys2aNemy1WgkP\nD1eXI4+PjzMyMkJfX59DuR8PBVwpLCzk2bNnXL9+ndzc3F/eQ4D09HTu379PX1+f01y4Gsf0OXE1\ntunMZjNmsxlFURzmFZy/HxqNxuGzr1+/snjxYvXax48fU1NTQ15eHunp6QwPDwMQFBREc3Mzz58/\np7W1lStXrtDc3ExtbS2dnZ08e/aMgoICzpw5Q2RkpNrW7+ZPCCHE3yd/pYUQQswoKiqKe/fuMTo6\nisVi4fDhwzx69Mjpup+Djy1btnDnzh0Avnz5wuvXr9HpdE7XTRcYGIjJZFKP3d3dnYJNAC8vL758\n+cLExATDw8N0dHSgKAqbN2+mpaWFwcFB7HY7FRUVXLt2zaHs9u3baWxspLGx0WVwZjQa+fbtGzab\njcbGRmJjY536HBUVRX19PRaLBbPZTE5ODl1dXWzbto2mpiYmJiYwm808ffrUqf6IiAiampoAePHi\nBSdPnnQYa1hYGG/evKGnpweA2tpaDAYDwcHBfP/+nXfv3gGoy3ddCQ0N5dSpU1y4cIH+/v7f3sPE\nxETa2tr4/v07GzdudKrLbrerbZlMJjo7O4mMjHSYE41Gg8ViYdWqVQwODqpLfOvq6rh169Yv+zld\nZGQkDx8+xGKxMDY2RkFBAQMDA+r5trY2kpKSSE1Nxdvbm/b2diwWCzdv3qSmpobExETKy8sZHBxk\naGiIpKQk1q1bR0lJCdHR0eqmSmazGbvdrmZ5hRBCzE6SWRVCCDGj+Ph43r9/T1ZWFlarlZiYGFJS\nUvj69atDhvPnbOeJEycoLy/n7t27KIpCVVUVS5Ys+e3OrvHx8RiNRmJiYoCpzGdKSopaxw/r1q0j\nNjaWPXv24O/vr77nGhISwqFDh8jLy8NmsxEaGjrjv4z5ma+vL0ePHmVgYIDo6GgyMzMxmUwO7e/f\nv5+enh5SU1OxWCxkZGSoWbu3b9+SnJyMl5cXq1evVufmR/ny8nKOHz/OjRs3mDdvnrqR1PSxVldX\nc+TIEaxWK8uWLcNgMKDRaDh37hzHjh3D3d2dDRs2/HYug4KCyMnJobKykpqaGpf3EKYyxjqdjuDg\nYJf1KIrC6OgoaWlpTE5OUllZiaenp0PbcXFxFBYWcvnyZQwGA2VlZUxOThIUFMTp06d58ODBjPOe\nkJBAd3c3qamp2O128vPzWblypdqHrKwsSktLefjwIT4+Puj1ekwmEwcPHqS0tJTk5GS0Wi3FxcV4\neXmRnZ1NRkYGHh4e+Pv7k5qaCkwtw46Pj5+xP0IIIf4uxf67x9tCCCHEHzYxMcGBAweor693Wjb6\nJxiNRurq6rh06dIfb/tvMZvN7N+/n6tXr+Lt7e103tUOwf9mJSUlFBcXz7ibshBCiL9LlgELIYSY\nVebMmUNRUZG6A++fNj0D+v+gq6sLvV5Pdna2y0D1f013dzf+/v4SqAohxL+AZFaFEEIIIYQQQsw6\nklkVQgghhBBCCDHrSLAqhBBCCCGEEGLWkWBVCCGEEEIIIcSsI8GqEEIIIYQQQohZR4JVIYQQQggh\nhBCzjgSrQgghhBBCCCFmnf8As9FFMLGuYjQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113d6a358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "key_classification_error_report(y_valid, lr_model.predict(X_valid));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By adding context we can get significant improvemnt in classification performance, eg. from 38% to 71% (!) even with a simple logistic regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import theanets\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def theano_accuracy_report(model):\n",
    "    y_train_pred = model.predict(X_train.astype(float32))\n",
    "    y_valid_pred = model.predict(X_valid.astype(float32))\n",
    "    print('training score:', accuracy_score(y_train, y_train_pred))\n",
    "    print('validation score:', accuracy_score(y_valid, y_valid_pred))\n",
    "    key_classification_error_report(y_valid, y_valid_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.feedforward:layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 theanets.layers.feedforward:41 layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"out\": (in:out)192 -> 12, softmax, 2316 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 theanets.layers.base:316 layer Feedforward \"out\": (in:out)192 -> 12, softmax, 2316 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:network has 2316 total parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 theanets.graph:114 network has 2316 total parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.dataset:144 valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.dataset:144 train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- patience = 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:378 -- patience = 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- validate_every = 10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:379 -- validate_every = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- min_improvement = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:380 -- min_improvement = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:381 -- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:382 -- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:383 -- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- momentum = 0.9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:384 -- momentum = 0.9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- nesterov = False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:385 -- nesterov = False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.adaptive:220 -- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.adaptive:221 -- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling evaluation function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:03 downhill.base:112 compiling evaluation function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling RMSProp function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:04 downhill.base:118 compiling RMSProp function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 0 loss=2.867817 err=2.867817 acc=0.081198 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:04 downhill.base:226 validation 0 loss=2.867817 err=2.867817 acc=0.081198 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 1 loss=1.057072 err=1.057072 acc=0.711126\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:05 downhill.base:226 RMSProp 1 loss=1.057072 err=1.057072 acc=0.711126\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 2 loss=0.991709 err=0.991709 acc=0.730833\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:07 downhill.base:226 RMSProp 2 loss=0.991709 err=0.991709 acc=0.730833\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 3 loss=0.986957 err=0.986957 acc=0.729553\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:08 downhill.base:226 RMSProp 3 loss=0.986957 err=0.986957 acc=0.729553\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 4 loss=0.986034 err=0.986034 acc=0.728889\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:09 downhill.base:226 RMSProp 4 loss=0.986034 err=0.986034 acc=0.728889\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 5 loss=0.985372 err=0.985372 acc=0.727331\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:10 downhill.base:226 RMSProp 5 loss=0.985372 err=0.985372 acc=0.727331\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 6 loss=0.984743 err=0.984743 acc=0.728273\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:11 downhill.base:226 RMSProp 6 loss=0.984743 err=0.984743 acc=0.728273\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 7 loss=0.984713 err=0.984713 acc=0.727485\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:12 downhill.base:226 RMSProp 7 loss=0.984713 err=0.984713 acc=0.727485\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 8 loss=0.984948 err=0.984948 acc=0.728495\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:13 downhill.base:226 RMSProp 8 loss=0.984948 err=0.984948 acc=0.728495\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 9 loss=0.984349 err=0.984349 acc=0.728129\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:14 downhill.base:226 RMSProp 9 loss=0.984349 err=0.984349 acc=0.728129\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 10 loss=0.983927 err=0.983927 acc=0.727033\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:15 downhill.base:226 RMSProp 10 loss=0.983927 err=0.983927 acc=0.727033\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 1 loss=0.989149 err=0.989149 acc=0.729196 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:15 downhill.base:226 validation 1 loss=0.989149 err=0.989149 acc=0.729196 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 11 loss=0.984644 err=0.984644 acc=0.728513\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:16 downhill.base:226 RMSProp 11 loss=0.984644 err=0.984644 acc=0.728513\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 12 loss=0.984121 err=0.984121 acc=0.726965\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:17 downhill.base:226 RMSProp 12 loss=0.984121 err=0.984121 acc=0.726965\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 13 loss=0.984243 err=0.984243 acc=0.727400\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:18 downhill.base:226 RMSProp 13 loss=0.984243 err=0.984243 acc=0.727400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 14 loss=0.984540 err=0.984540 acc=0.727946\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:19 downhill.base:226 RMSProp 14 loss=0.984540 err=0.984540 acc=0.727946\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 15 loss=0.983947 err=0.983947 acc=0.726946\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:20 downhill.base:226 RMSProp 15 loss=0.983947 err=0.983947 acc=0.726946\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 16 loss=0.984514 err=0.984514 acc=0.726187\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:21 downhill.base:226 RMSProp 16 loss=0.984514 err=0.984514 acc=0.726187\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 17 loss=0.984085 err=0.984085 acc=0.727303\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:22 downhill.base:226 RMSProp 17 loss=0.984085 err=0.984085 acc=0.727303\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 18 loss=0.984627 err=0.984627 acc=0.726330\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:23 downhill.base:226 RMSProp 18 loss=0.984627 err=0.984627 acc=0.726330\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 19 loss=0.983704 err=0.983704 acc=0.727148\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:24 downhill.base:226 RMSProp 19 loss=0.983704 err=0.983704 acc=0.727148\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 20 loss=0.984233 err=0.984233 acc=0.727881\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:25 downhill.base:226 RMSProp 20 loss=0.984233 err=0.984233 acc=0.727881\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 2 loss=0.987955 err=0.987955 acc=0.727089 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:25 downhill.base:226 validation 2 loss=0.987955 err=0.987955 acc=0.727089 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 21 loss=0.984349 err=0.984349 acc=0.727495\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:26 downhill.base:226 RMSProp 21 loss=0.984349 err=0.984349 acc=0.727495\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 22 loss=0.983837 err=0.983837 acc=0.726870\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:27 downhill.base:226 RMSProp 22 loss=0.983837 err=0.983837 acc=0.726870\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 23 loss=0.983572 err=0.983572 acc=0.727400\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:28 downhill.base:226 RMSProp 23 loss=0.983572 err=0.983572 acc=0.727400\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 24 loss=0.984239 err=0.984239 acc=0.726736\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:29 downhill.base:226 RMSProp 24 loss=0.984239 err=0.984239 acc=0.726736\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 25 loss=0.984282 err=0.984282 acc=0.727224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:30 downhill.base:226 RMSProp 25 loss=0.984282 err=0.984282 acc=0.727224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 26 loss=0.984110 err=0.984110 acc=0.727670\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:30 downhill.base:226 RMSProp 26 loss=0.984110 err=0.984110 acc=0.727670\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 27 loss=0.984089 err=0.984089 acc=0.728032\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:31 downhill.base:226 RMSProp 27 loss=0.984089 err=0.984089 acc=0.728032\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 28 loss=0.984372 err=0.984372 acc=0.727706\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:32 downhill.base:226 RMSProp 28 loss=0.984372 err=0.984372 acc=0.727706\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 29 loss=0.984408 err=0.984408 acc=0.727004\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:33 downhill.base:226 RMSProp 29 loss=0.984408 err=0.984408 acc=0.727004\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 30 loss=0.984162 err=0.984162 acc=0.727119\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:35 downhill.base:226 RMSProp 30 loss=0.984162 err=0.984162 acc=0.727119\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 3 loss=0.990214 err=0.990214 acc=0.727407\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:35 downhill.base:226 validation 3 loss=0.990214 err=0.990214 acc=0.727407\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 31 loss=0.984148 err=0.984148 acc=0.727860\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:36 downhill.base:226 RMSProp 31 loss=0.984148 err=0.984148 acc=0.727860\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 32 loss=0.984465 err=0.984465 acc=0.726908\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:37 downhill.base:226 RMSProp 32 loss=0.984465 err=0.984465 acc=0.726908\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 33 loss=0.983813 err=0.983813 acc=0.727186\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:38 downhill.base:226 RMSProp 33 loss=0.983813 err=0.983813 acc=0.727186\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 34 loss=0.984134 err=0.984134 acc=0.727350\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:39 downhill.base:226 RMSProp 34 loss=0.984134 err=0.984134 acc=0.727350\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 35 loss=0.983718 err=0.983718 acc=0.727022\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:40 downhill.base:226 RMSProp 35 loss=0.983718 err=0.983718 acc=0.727022\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 36 loss=0.984266 err=0.984266 acc=0.727023\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:41 downhill.base:226 RMSProp 36 loss=0.984266 err=0.984266 acc=0.727023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 37 loss=0.984069 err=0.984069 acc=0.727323\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:42 downhill.base:226 RMSProp 37 loss=0.984069 err=0.984069 acc=0.727323\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 38 loss=0.983773 err=0.983773 acc=0.726176\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:43 downhill.base:226 RMSProp 38 loss=0.983773 err=0.983773 acc=0.726176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 39 loss=0.984460 err=0.984460 acc=0.727581\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:44 downhill.base:226 RMSProp 39 loss=0.984460 err=0.984460 acc=0.727581\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 40 loss=0.984202 err=0.984202 acc=0.727350\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:45 downhill.base:226 RMSProp 40 loss=0.984202 err=0.984202 acc=0.727350\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 4 loss=0.989547 err=0.989547 acc=0.728763\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:45 downhill.base:226 validation 4 loss=0.989547 err=0.989547 acc=0.728763\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 41 loss=0.984240 err=0.984240 acc=0.726321\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:46 downhill.base:226 RMSProp 41 loss=0.984240 err=0.984240 acc=0.726321\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 42 loss=0.984125 err=0.984125 acc=0.727294\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:47 downhill.base:226 RMSProp 42 loss=0.984125 err=0.984125 acc=0.727294\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 43 loss=0.983844 err=0.983844 acc=0.726966\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:48 downhill.base:226 RMSProp 43 loss=0.983844 err=0.983844 acc=0.726966\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 44 loss=0.983985 err=0.983985 acc=0.726813\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:49 downhill.base:226 RMSProp 44 loss=0.983985 err=0.983985 acc=0.726813\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 45 loss=0.984097 err=0.984097 acc=0.727793\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:50 downhill.base:226 RMSProp 45 loss=0.984097 err=0.984097 acc=0.727793\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 46 loss=0.984294 err=0.984294 acc=0.727485\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:51 downhill.base:226 RMSProp 46 loss=0.984294 err=0.984294 acc=0.727485\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 47 loss=0.984080 err=0.984080 acc=0.727658\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:52 downhill.base:226 RMSProp 47 loss=0.984080 err=0.984080 acc=0.727658\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 48 loss=0.983487 err=0.983487 acc=0.726695\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:53 downhill.base:226 RMSProp 48 loss=0.983487 err=0.983487 acc=0.726695\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 49 loss=0.984433 err=0.984433 acc=0.727074\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:54 downhill.base:226 RMSProp 49 loss=0.984433 err=0.984433 acc=0.727074\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 50 loss=0.984034 err=0.984034 acc=0.726773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:55 downhill.base:226 RMSProp 50 loss=0.984034 err=0.984034 acc=0.726773\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 5 loss=0.987948 err=0.987948 acc=0.728416 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:55 downhill.base:226 validation 5 loss=0.987948 err=0.987948 acc=0.728416 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 51 loss=0.984345 err=0.984345 acc=0.727246\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:56 downhill.base:226 RMSProp 51 loss=0.984345 err=0.984345 acc=0.727246\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 52 loss=0.984430 err=0.984430 acc=0.726429\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:56 downhill.base:226 RMSProp 52 loss=0.984430 err=0.984430 acc=0.726429\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 53 loss=0.984107 err=0.984107 acc=0.727929\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:57 downhill.base:226 RMSProp 53 loss=0.984107 err=0.984107 acc=0.727929\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 54 loss=0.983981 err=0.983981 acc=0.728052\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:28:58 downhill.base:226 RMSProp 54 loss=0.983981 err=0.983981 acc=0.728052\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 55 loss=0.983950 err=0.983950 acc=0.727475\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:00 downhill.base:226 RMSProp 55 loss=0.983950 err=0.983950 acc=0.727475\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 56 loss=0.984073 err=0.984073 acc=0.727514\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:01 downhill.base:226 RMSProp 56 loss=0.984073 err=0.984073 acc=0.727514\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 57 loss=0.984372 err=0.984372 acc=0.727063\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:02 downhill.base:226 RMSProp 57 loss=0.984372 err=0.984372 acc=0.727063\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 58 loss=0.984092 err=0.984092 acc=0.727456\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:03 downhill.base:226 RMSProp 58 loss=0.984092 err=0.984092 acc=0.727456\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 59 loss=0.983836 err=0.983836 acc=0.727659\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:04 downhill.base:226 RMSProp 59 loss=0.983836 err=0.983836 acc=0.727659\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 60 loss=0.983987 err=0.983987 acc=0.728023\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:05 downhill.base:226 RMSProp 60 loss=0.983987 err=0.983987 acc=0.728023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 6 loss=0.995586 err=0.995586 acc=0.726801\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:05 downhill.base:226 validation 6 loss=0.995586 err=0.995586 acc=0.726801\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 61 loss=0.983978 err=0.983978 acc=0.726753\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:06 downhill.base:226 RMSProp 61 loss=0.983978 err=0.983978 acc=0.726753\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 62 loss=0.984062 err=0.984062 acc=0.727188\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:07 downhill.base:226 RMSProp 62 loss=0.984062 err=0.984062 acc=0.727188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 63 loss=0.984010 err=0.984010 acc=0.727160\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:08 downhill.base:226 RMSProp 63 loss=0.984010 err=0.984010 acc=0.727160\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 64 loss=0.983626 err=0.983626 acc=0.727467\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:08 downhill.base:226 RMSProp 64 loss=0.983626 err=0.983626 acc=0.727467\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 65 loss=0.984320 err=0.984320 acc=0.727977\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:09 downhill.base:226 RMSProp 65 loss=0.984320 err=0.984320 acc=0.727977\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 66 loss=0.984001 err=0.984001 acc=0.727572\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:10 downhill.base:226 RMSProp 66 loss=0.984001 err=0.984001 acc=0.727572\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 67 loss=0.983826 err=0.983826 acc=0.727525\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:11 downhill.base:226 RMSProp 67 loss=0.983826 err=0.983826 acc=0.727525\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 68 loss=0.984165 err=0.984165 acc=0.726986\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:12 downhill.base:226 RMSProp 68 loss=0.984165 err=0.984165 acc=0.726986\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 69 loss=0.984040 err=0.984040 acc=0.726945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:13 downhill.base:226 RMSProp 69 loss=0.984040 err=0.984040 acc=0.726945\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 70 loss=0.984121 err=0.984121 acc=0.727600\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:15 downhill.base:226 RMSProp 70 loss=0.984121 err=0.984121 acc=0.727600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 7 loss=0.990552 err=0.990552 acc=0.726887\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:15 downhill.base:226 validation 7 loss=0.990552 err=0.990552 acc=0.726887\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 71 loss=0.984368 err=0.984368 acc=0.726687\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:16 downhill.base:226 RMSProp 71 loss=0.984368 err=0.984368 acc=0.726687\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 72 loss=0.984244 err=0.984244 acc=0.727206\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:16 downhill.base:226 RMSProp 72 loss=0.984244 err=0.984244 acc=0.727206\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 73 loss=0.984022 err=0.984022 acc=0.728236\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:17 downhill.base:226 RMSProp 73 loss=0.984022 err=0.984022 acc=0.728236\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 74 loss=0.983962 err=0.983962 acc=0.727187\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:18 downhill.base:226 RMSProp 74 loss=0.983962 err=0.983962 acc=0.727187\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 75 loss=0.984079 err=0.984079 acc=0.727302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:19 downhill.base:226 RMSProp 75 loss=0.984079 err=0.984079 acc=0.727302\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 76 loss=0.983912 err=0.983912 acc=0.727831\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:20 downhill.base:226 RMSProp 76 loss=0.983912 err=0.983912 acc=0.727831\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 77 loss=0.984080 err=0.984080 acc=0.727708\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:21 downhill.base:226 RMSProp 77 loss=0.984080 err=0.984080 acc=0.727708\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 78 loss=0.984300 err=0.984300 acc=0.728295\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:21 downhill.base:226 RMSProp 78 loss=0.984300 err=0.984300 acc=0.728295\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 79 loss=0.983952 err=0.983952 acc=0.727669\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:22 downhill.base:226 RMSProp 79 loss=0.983952 err=0.983952 acc=0.727669\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 80 loss=0.983850 err=0.983850 acc=0.727130\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:23 downhill.base:226 RMSProp 80 loss=0.983850 err=0.983850 acc=0.727130\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 8 loss=0.993334 err=0.993334 acc=0.723453\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:23 downhill.base:226 validation 8 loss=0.993334 err=0.993334 acc=0.723453\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 81 loss=0.984103 err=0.984103 acc=0.727274\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:24 downhill.base:226 RMSProp 81 loss=0.984103 err=0.984103 acc=0.727274\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 82 loss=0.984072 err=0.984072 acc=0.726889\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:25 downhill.base:226 RMSProp 82 loss=0.984072 err=0.984072 acc=0.726889\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 83 loss=0.984256 err=0.984256 acc=0.728381\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:25 downhill.base:226 RMSProp 83 loss=0.984256 err=0.984256 acc=0.728381\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 84 loss=0.983748 err=0.983748 acc=0.726784\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:26 downhill.base:226 RMSProp 84 loss=0.983748 err=0.983748 acc=0.726784\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 85 loss=0.984495 err=0.984495 acc=0.727967\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:27 downhill.base:226 RMSProp 85 loss=0.984495 err=0.984495 acc=0.727967\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 86 loss=0.983732 err=0.983732 acc=0.726986\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:28 downhill.base:226 RMSProp 86 loss=0.983732 err=0.983732 acc=0.726986\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 87 loss=0.984224 err=0.984224 acc=0.727705\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:28 downhill.base:226 RMSProp 87 loss=0.984224 err=0.984224 acc=0.727705\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 88 loss=0.984218 err=0.984218 acc=0.727544\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:29 downhill.base:226 RMSProp 88 loss=0.984218 err=0.984218 acc=0.727544\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 89 loss=0.984236 err=0.984236 acc=0.727189\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:30 downhill.base:226 RMSProp 89 loss=0.984236 err=0.984236 acc=0.727189\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 90 loss=0.984096 err=0.984096 acc=0.727302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:31 downhill.base:226 RMSProp 90 loss=0.984096 err=0.984096 acc=0.727302\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 9 loss=0.988708 err=0.988708 acc=0.729224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:31 downhill.base:226 validation 9 loss=0.988708 err=0.988708 acc=0.729224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 91 loss=0.984124 err=0.984124 acc=0.726908\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:32 downhill.base:226 RMSProp 91 loss=0.984124 err=0.984124 acc=0.726908\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 92 loss=0.984406 err=0.984406 acc=0.728016\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:32 downhill.base:226 RMSProp 92 loss=0.984406 err=0.984406 acc=0.728016\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 93 loss=0.984264 err=0.984264 acc=0.726821\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:33 downhill.base:226 RMSProp 93 loss=0.984264 err=0.984264 acc=0.726821\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 94 loss=0.983750 err=0.983750 acc=0.727264\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:34 downhill.base:226 RMSProp 94 loss=0.983750 err=0.983750 acc=0.727264\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 95 loss=0.983938 err=0.983938 acc=0.727754\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:35 downhill.base:226 RMSProp 95 loss=0.983938 err=0.983938 acc=0.727754\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 96 loss=0.984030 err=0.984030 acc=0.727475\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 09:29:35 downhill.base:226 RMSProp 96 loss=0.984030 err=0.984030 acc=0.727475\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 97 loss=0.983864 err=0.983864 acc=0.726974\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:31 downhill.base:226 RMSProp 97 loss=0.983864 err=0.983864 acc=0.726974\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 98 loss=0.984149 err=0.984149 acc=0.727092\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:32 downhill.base:226 RMSProp 98 loss=0.984149 err=0.984149 acc=0.727092\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 99 loss=0.984112 err=0.984112 acc=0.727323\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:32 downhill.base:226 RMSProp 99 loss=0.984112 err=0.984112 acc=0.727323\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 100 loss=0.984162 err=0.984162 acc=0.728110\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:33 downhill.base:226 RMSProp 100 loss=0.984162 err=0.984162 acc=0.728110\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 10 loss=0.989118 err=0.989118 acc=0.727666\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:33 downhill.base:226 validation 10 loss=0.989118 err=0.989118 acc=0.727666\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 101 loss=0.984026 err=0.984026 acc=0.726830\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:34 downhill.base:226 RMSProp 101 loss=0.984026 err=0.984026 acc=0.726830\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 102 loss=0.984124 err=0.984124 acc=0.727313\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:35 downhill.base:226 RMSProp 102 loss=0.984124 err=0.984124 acc=0.727313\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 103 loss=0.984027 err=0.984027 acc=0.726918\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:35 downhill.base:226 RMSProp 103 loss=0.984027 err=0.984027 acc=0.726918\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 104 loss=0.984142 err=0.984142 acc=0.727177\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:36 downhill.base:226 RMSProp 104 loss=0.984142 err=0.984142 acc=0.727177\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 105 loss=0.984065 err=0.984065 acc=0.728044\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:37 downhill.base:226 RMSProp 105 loss=0.984065 err=0.984065 acc=0.728044\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 106 loss=0.984110 err=0.984110 acc=0.727349\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:38 downhill.base:226 RMSProp 106 loss=0.984110 err=0.984110 acc=0.727349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 107 loss=0.984072 err=0.984072 acc=0.727436\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:38 downhill.base:226 RMSProp 107 loss=0.984072 err=0.984072 acc=0.727436\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 108 loss=0.983700 err=0.983700 acc=0.727420\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:39 downhill.base:226 RMSProp 108 loss=0.983700 err=0.983700 acc=0.727420\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 109 loss=0.984225 err=0.984225 acc=0.727543\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:40 downhill.base:226 RMSProp 109 loss=0.984225 err=0.984225 acc=0.727543\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 110 loss=0.984035 err=0.984035 acc=0.726448\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:41 downhill.base:226 RMSProp 110 loss=0.984035 err=0.984035 acc=0.726448\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 11 loss=0.990566 err=0.990566 acc=0.727089\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:41 downhill.base:226 validation 11 loss=0.990566 err=0.990566 acc=0.727089\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:patience elapsed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 10:29:41 downhill.base:402 patience elapsed!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(OrderedDict([('loss', 0.98403544482285155), ('err', 0.98403544482285155), ('acc', 0.72644832362212242)]),\n",
       " OrderedDict([('loss', 0.98911773954842341), ('err', 0.98911773954842341), ('acc', 0.72766620498614953)]))"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# no hidden layers, no pretraining or regularization\n",
    "exp = theanets.Experiment(theanets.Classifier, layers=[X.shape[1], 12])\n",
    "exp.train(\n",
    "    (X_train.astype(float32),y_train),\n",
    "    (X_valid.astype(float32),y_valid),\n",
    "    algo='sgd', learning_rate=1e-4, momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training score: 0.731871302842\n",
      "validation score: 0.728416435826\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2U13Wd///HZ7gUkcxtyqOWGIdjduURHRZOSh1+VFPb\nFakFAtlGm25rtWC2Q4rj5YrSKTejxKyOTSYunZG07WIDOYtJK+ymlJJd7IqVgYBuNSDKwHx+f/ht\n8gp5o36Y18Dt9hczw3zez89Thbn7+sx7avV6vR4AAAAoSFNfDwAAAABPJVYBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKM7ARj1wT09Pzj333Kxbty5NTU25+OKLM2DAgLS1taWpqSmjR49Oe3t7\narVao0YAAACgn2pYrP7oRz/Ktm3bcsMNN2TlypX53Oc+lx07dmT27NlpaWlJe3t7li1blkmTJjVq\nBAAAAPqphr0MeOjQoenq6kq9Xk9XV1cGDRqUe+65Jy0tLUmSCRMmZOXKlY26PAAAAP1Yw05Wx4wZ\nk+3bt6e1tTV/+MMfcvXVV2f16tW9Hx82bFi6uroadXkAAAD6sYbF6rXXXpsxY8Zk1qxZ2bBhQz7w\ngQ9kx44dvR/funVrRowYsdvHqdfrvq8VAABgP9OwWN22bVsOPPDAJMmIESOyY8eOvPrVr86qVasy\nduzYrFixIuPHj9/t49RqtWza5AR2d5qbD7KniuyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7qq65\n+aBnfH/DYnXmzJmZM2dOTjvttOzYsSNnn312XvOa12Tu3Lnp7u7OqFGj0tra2qjLAwAA0I81LFZH\njBiRBQsWPO39HR0djbokAAAA+4iG3Q0YAAAAniuxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABSn+Fit1+t9PQIAAAB7WfGx2tRUz8knDxKtAAAA+5HiYzVpym23Dcmxxx6QO+8UrAAAAPuD\nfhCrSVLLhg2DcvrpBzhhBQAA2A/0k1gFAABgfzKwrweopp5DD92R667bllqt1tfDAAAA0GD9IlYP\nPXRH1qwRqgAAAPuL4l8GPGBA3YkqAADAfqb4WD3iiOS444QqAADA/qT4WE2EKgAAwP6mH8QqAAAA\n+xuxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHEGNvLBb7rppnR2diZJHnvssdx777355je/mUsvvTRN\nTU0ZPXp02tvbU6vVGjkGAAAA/UxDT1YnT56cjo6OdHR05LWvfW3mzp2bBQsWZPbs2bn++utTr9ez\nbNmyRo4AAABAP7RXXgb8s5/9LL/+9a9z6qmn5p577klLS0uSZMKECVm5cuXeGAEAAIB+ZK/E6sKF\nC3PWWWclSer1eu/7hw0blq6urr0xAgAAAP1IQ79nNUn+9Kc/Zd26dRk7dmySpKnpL328devWjBgx\nYreP0dx8UMPm25fYU3V2VY09VWNP1dlVNfZUnV1VY0/V2FN1dlWNPT0/DY/V1atXZ9y4cb1vH3PM\nMVm1alXGjh2bFStWZPz48bt9jE2bnL7uTnPzQfZUkV1VY0/V2FN1dlWNPVVnV9XYUzX2VJ1dVWNP\n1e0q6hseq+vWrcsrXvGK3rfb2toyd+7cdHd3Z9SoUWltbW30CAAAAPQzDY/VmTNnPuntkSNHpqOj\no9GXBQAAoB/bKzdYAgAAgD0hVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiDGzkgy9cuDDLly9Pd3d3pk+fnjFjxqStrS1N\nTU0ZPXp02tvbU6vVGjkCAAAA/VDDTlbvuOOO3HnnnVm0aFE6Ojry29/+NvPmzcvs2bNz/fXXp16v\nZ9myZY26PAAAAP1Yw2L19ttvz9FHH52PfvSjOfPMMzNx4sTcc889aWlpSZJMmDAhK1eubNTlAQAA\n6Mca9jLghx9+OOvXr8/ChQvz29/+NmeeeWbq9Xrvx4cNG5aurq5GXR4AAIB+rGGx+uIXvzijRo3K\nwIEDc9RRR2XIkCHZuHFj78e3bt2aESNGVHqs5uaDGjXmPsWeqrOrauypGnuqzq6qsafq7Koae6rG\nnqqzq2rs6flpWKwef/zx+frXv56//du/zYMPPphHH30048aNy6pVqzJ27NisWLEi48ePr/RYmzY5\ngd2d5uaD7Kkiu6rGnqqxp+rsqhp7qs6uqrGnauypOruqxp6q21XUNyxW3/SmN2X16tU55ZRT0tPT\nk/b29hx++OGZO3duuru7M2rUqLS2tjbq8gAAAPRjDf3RNeecc87T3tfR0dHISwIAALAPaNjdgAEA\nAOC5EqsAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByx\nCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQq\nAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasA\nAAAUR6wCAABQnIGNvsDkyZMzfPjwJMnLX/7ynHHGGWlra0tTU1NGjx6d9vb21Gq1Ro8BAABAP9LQ\nWH3ssceSJB0dHb3vO/PMMzN79uy0tLSkvb09y5Yty6RJkxo5BgAAAP1MQ18GfO+992bbtm2ZOXNm\nTj/99Nx1111Zu3ZtWlpakiQTJkzIypUrGzkCAAAA/VBDT1YPOOCAzJw5M6eeemrWrVuXD3/4w0/6\n+LBhw9LV1dXIEQAAAOiHGhqrI0eOzJFHHtn764MPPjg///nPez++devWjBgxYreP09x8UMNm3JfY\nU3V2VY09VWNP1dlVNfZUnV1VY0/V2FN1dlWNPT0/DY3Vzs7O/OIXv0h7e3sefPDBbN26NW94wxuy\natWqjB07NitWrMj48eN3+zibNjl93Z3m5oPsqSK7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnqrb\nVdQ3NFZPOeWUzJkzJ9OmTUuSXHbZZTn44IMzd+7cdHd3Z9SoUWltbW3kCAAAAPRDDY3VgQMHZv78\n+U97/xPvDgwAAABP1dC7AQMAAMBzIVYBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4uxRrD766KPZsmXLHl3goYceyhvf+Mbc\nd999uf/++zN16tRMmzYtF1xwQer1+h49FgAAAPuHyrG6ePHivO9978uUKVNy5ZVXVvqc7u7unH/+\n+TnggANSr9dz2WWXZfbs2bn++utTr9ezbNmy5zw4AAAA+65dxuovf/nLJ729bNmy3HzzzfnOd76T\npUuXVnrwK664IlOnTk1zc3OSZO3atWlpaUmSTJgwIStXrnyucwMAALAPG7irD9x4443p7u7OP/zD\nP+RlL3tZXvWqV2XmzJkZOHBgRo8evdsH7uzszCGHHJITTzwxCxcuTL1ef9LLfocNG5aurq4X5lkA\nAACwT6nVn+UbR++7774sWLAghx12WP7u7/4umzZtyvbt2/OqV71qtw88ffr01Gq1JMm9996bkSNH\n5uc//3nuvvvuJMnSpUvz4x//OHPnzn3Wxxk5Mlm3rvoTAgAAoP971lj9s5/85Ce55pprMm7cuEyb\nNi2DBg3ao4vMmDEjF154Ya644op86EMfytixY3P++edn/Pjxedvb3vasnztyZLJ6tRPY3WluPiib\nNtlTFXZVjT1VY0/V2VU19lSdXVVjT9XYU3V2VY09VdfcfNAzvn+X37N6/fXXZ9KkSXnrW9+ajRs3\n5uqrr85hhx2WM888MzfffPMeD1Cr1dLW1parrroqU6ZMyc6dO9Pa2rrHjwMAAMC+b5ffs3rDDTfk\n+9//frZv355p06altbU1b3nLWzJx4sT867/+6x5dpKOj4xl/DQAAAM9kl7Ha3Nycf/7nf86jjz6a\nV77ylX/5hIEDc9ppp+2V4QAAANg/7TJWr7766tx2220ZPHhw3vCGN+zNmQAAANjP7TJWhwwZkkmT\nJu3NWQAAACDJs9xgCQAAAPqKWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKI\nVQAAAIojVgEAACjOwEY++M6dO3Peeedl3bp1qdVqufDCCzN48OC0tbWlqakpo0ePTnt7e2q1WiPH\nAAAAoJ9paKwuX748TU1NueGGG7Jq1ap89rOfTZLMnj07LS0taW9vz7JlyzJp0qRGjgEAAEA/09CX\nAU+aNCkXXXRRkuSBBx7Ii170otxzzz1paWlJkkyYMCErV65s5AgAAAD0Qw3/ntUBAwakra0tl156\nad75znemXq/3fmzYsGHp6upq9AgAAAD0M7X6E+uxgTZv3pxTTz01jzzySO64444kydKlS/PjH/84\nc+fO3eXnjRyZrFu3NyYEAACgFA39ntUlS5bkwQcfzBlnnJGhQ4emqakpr33ta7Nq1aqMHTs2K1as\nyPjx43f7OJs2OX3dnebmg+ypIruqxp6qsafq7Koae6rOrqqxp2rsqTq7qsaeqmtuPugZ39/QWG1t\nbU1bW1umT5+eHTt25Nxzz80rX/nKzJ07N93d3Rk1alRaW1sbOQIAAAD9UENjdejQobnyyiuf9v6O\njo5GXhYAAIB+ruE3WAIAAIA9JVYBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOAMb9cDd3d359Kc/nd///vfZvn17/v7v/z6jRo1KW1tb\nmpqaMnr06LS3t6dWqzVqBAAAAPqphsXqLbfckkMOOSTz58/PH//4x7z73e/OMccck9mzZ6elpSXt\n7e1ZtmxZJk2a1KgRAAAA6Kca9jLg1tbWfPzjH0+S9PT0ZODAgVm7dm1aWlqSJBMmTMjKlSsbdXkA\nAAD6sYbF6rBhw3LggQdmy5Yt+cQnPpF//Md/TE9Pz5M+3tXV1ajLAwAA0I817GXASbJ+/fqcddZZ\nmTZtWt7xjndk/vz5vR/bunVrRowYUelxmpsPatSI+xR7qs6uqrGnauypOruqxp6qs6tq7Kkae6rO\nrqqxp+enYbG6efPmfOhDH0p7e3vGjRuXJDnmmGOyatWqjB07NitWrMj48eMrPdamTU5gd6e5+SB7\nqsiuqrGnauypOruqxp6qs6tq7Kkae6rOrqqxp+p2FfUNi9Wrr746XV1dWbBgQRYsWJAkOffcc3Pp\npZemu7s7o0aNSmtra6MuDwAAQD/WsFg977zzct555z3t/R0dHY26JAAAAPuIht1gCQAAAJ4rsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEes\nAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUp+GxumbNmsyYMSNJcv/992fq1KmZNm1aLrjggtTr9UZfHgAAgH6oobH65S9/Oeed\nd166u7uTJJdddllmz56d66+/PvV6PcuWLWvk5QEAAOinGhqrRx55ZL7whS/0nqCuXbs2LS0tSZIJ\nEyZk5cqVjbw8AAAA/VRDY/Utb3lLBgwY0Pv2E1/2O2zYsHR1dTXy8gAAAPRTA/fmxZqa/tLGW7du\nzYgRIyp9XnPzQY0aaZ9iT9XZVTX2VI09VWdX1dhTdXZVjT1VY0/V2VU19vT87NVYPeaYY7Jq1aqM\nHTs2K1asyPjx4yt93qZNTmB3p7n5IHuqyK6qsadq7Kk6u6rGnqqzq2rsqRp7qs6uqrGn6nYV9Xsl\nVmu1WpKkra0tc+fOTXd3d0aNGpXW1ta9cXkAAAD6mYbH6hFHHJFFixYlSUaOHJmOjo5GXxIAAIB+\nruE/Z5XGq9frfmYtAACwTxGr/Vi9Xs/JJw/Ky152YJqaHv+1aAUAAPYFYrUfO+WUwbnttiFJBiRp\nym23Dcmxxx6QO+8UrAAAQP8mVvuper2e224blKT2hPfWsmHDoJx++gFOWAEAgH5NrAIAAFAcsdpP\n1Wq1nHRSd5InnqDWc+ih3bnuum29Py6IJ3MzKgAA6B/Eaj/2rW9tz6GH7uh9+9BDd2TNmm057jih\n+lRuRgUAAP2LWO3HarVarrtuW5qa6hkwoO5E9Vm4GRUAAPQvYrWfO+64Wg4/vJ4jjogT1V1wMyoA\nAOh/xOo+Q6gCAAD7DrHKPs/NqAAAoP8Rq+wX3IwKAAD6F7HKfsHNqAAAoH8Z2NcDwN7y55tRNTXV\nnKgCAEDhnKyyHxKqAABQOrEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQo8Tb1eT71e7+sxAADYj4lVoFe9Xs/JJw/Ky152YJqaHv+1aAUAoC+IVaDXKacMzm23\nDUkyIElTbrttSI499oDceadgBQBg7xKrQJLHT1Vvu21QktoT3lvLhg2DcvrpBzhhfQZeLg0A0Dhi\nFWAPebn0nhP21dgTAPyFWAWSJLVaLSed1J3kiV8o13Pood257rptqdVqu/rU/Y6XS1cn7KuxJwB4\nOrEK9PrWt7bn0EN39L596KE7smbNthx3nFD9My+X3jPCvhp72nNOoQH2fWIV6FWr1XLdddvS1FTP\ngAF1J6o8L8K+GnvaM06hAfYfYhV4kuOOq+Xww+s54og4UX0GXi4NfcspNMD+Q6wCuyC6dsXLpasR\n9tXYU3VOoQH2L2IVYA95uXR1wr4aewKApxOrAM+Bl0tXI+yrsadqnELvOTeiAvozsQrwvPjieHeE\nfTX2VI1T6GrciArYF4hVAPYSMVGNPT0bp9DVuBHVnnECDWUSqwBAv+IU+tm5EVV1TqD3nLBnbxKr\nAEA/JVR5fpxAVyfs6QtiFQBgH+JGVNU4gd4zwn7POIF+Yez1WO3p6cn555+fKVOmZMaMGfnNb36z\nt0cAANinuREVLyRhX50T6BfWXo/VpUuXpru7O4sWLconP/nJzJs3b2+PAACwT3Mjqt1zAk0jOIF+\nYe31WP3JT36Sk046KUly7LHH5u67797bIwAA7PPciGr3nEBXI+yrcQL9whu4ty+4ZcuWDB8+vPft\nAQMGpKenJ01Nz9zNP/rdyBxyvH+wz+b239dSS+ypAruqxp6qsafq7Koae6rOrqqxp93733ry4P+L\ni5fV6xnSE3N6AAAT8UlEQVR8Qh8PVKjlSX7fVMvOnsffHtCUHDaonny4T8cqzn1PCNXFOTWfyvw+\nnKb/2+uxOnz48GzdurX37WcL1SQ5Yse6vTBV//byvh6gH7GrauypGnuqzq6qsafq7Koae9q9A5Ic\n+f9OvJwQPrsNq3fmPe95/NdLliQvbxnQtwMVaOb/tyO33jogfzldreeww3qyZElTXvrSEX05Wr+0\n12N1zJgxWb58ed72trflrrvuytFHH723RwAAgF4itZqWlgF54IG+nqJsy5Y9Na9qefz7V3kuavW9\n/OLper2eCy64IL/4xS+SJJdddlmOOuqovTkCAAAAhdvrsQoAAAC7s9fvBgwAAAC7I1YBAAAojlgF\nAACgOEXGak9PT84///xMmTIlM2bMyG9+85u+HqlokydPzowZMzJjxox8+tOf7utxinTTTTf17uh9\n73tfXv/612fLli19PVaRdu7cmTlz5mTq1Kk57bTT8qtf/aqvRyramjVrMmPGjL4eo0hP3M3999+f\nqVOnZtq0abngggv8YPRnsH379nzqU5/KlClTMn369Nx77719PVKRuru7c84552TatGk59dRTc+ut\nt/b1SEVbuHBhpkyZkpNPPjk33XRTX49TpJ6ent6/96ZNm5b//d//7euRivbQQw/ljW98Y+67776+\nHqVIT/264Ic//GHOPvvsPpyof9vrP7qmiqVLl6a7uzuLFi3KmjVrMm/evHzxi1/s67GK9NhjjyVJ\nOjo6+niSsk2ePDmTJ09Oklx00UU59dRTM3z48D6eqkzLly9PU1NTbrjhhqxatSqf+9zn/Pe3C1/+\n8pdz880358ADD+zrUYrz1N1cdtllmT17dlpaWtLe3p5ly5Zl0qRJfTxlWRYvXpyhQ4dm0aJFue++\n+3L22Wens7Ozr8cqzi233JJDDjkk8+fPzx//+Me85z3vycSJE/t6rCLdcccdufPOO7No0aI88sgj\nufbaa/t6pCL96Ec/yrZt23LDDTdk5cqVufLKK/P5z3++r8cqUnd3d84///wccMABfT1KkZ76d98l\nl1yS22+/Pa9+9av7eLL+q8iT1Z/85Cc56aSTkiTHHnts7r777j6eqFz33ntvtm3blpkzZ+b000/P\nmjVr+nqkov3sZz/Lr371q5x66ql9PUqxJk2alIsuuihJ8sADD+RFL3pRH09UriOPPDJf+MIXnBI+\ng6fuZu3atWlpaUmSTJgwIStXruzL8Yr061//OhMmTEiSHHXUUXnwwQe9AuQZtLa25uMf/3iSx0/E\nBgzw8wt35fbbb8/RRx+dj370oznzzDNF/S4MHTo0XV1dqdfr6erqyqBBg/p6pGJdccUVmTp1apqb\nm/t6lCI99e++MWPGeDXR81RkrG7ZsuVJp14DBgxIT09PH05UrgMOOCAzZ87MV77ylVx44YX55Cc/\naVfPYuHChfnYxz7W12MUb8CAAWlra8sll1ySd7zjHX09TrHe8pa3+EJ5F566myf+RT1s2LB0dXX1\nxVhFO+aYY7J8+fIkyV133ZWHH344jzzySB9PVZ5hw4blwAMPzJYtW/KJT3wis2bN6uuRivXwww/n\n7rvvzuc///nerxF4ujFjxmT79u1pbW3N+eefn+nTp/f1SEXq7OzMIYcckhNPPDFJBNgzeOrffW9/\n+9v7cJp9Q5GxOnz48GzdurX37Z6enjQ1FTlqnxs5cmTe9a539f764IMPzqZNm/p4qjL96U9/yrp1\n6zJ27Ni+HqVfmDdvXn7wgx9k7ty5efTRR/t6HPq5J/4ZvnXr1owYMaIPpynTySefnOHDh+e0007L\n0qVLe/9M5+nWr1+f008/Pe95z3vyN3/zN309TrFe/OIX58QTT8zAgQNz1FFHZciQIXn44Yf7eqzi\nXHvttRkzZkx+8IMf5Nvf/nba2tqyffv2vh6rOJ2dnVm5cmVmzJiRe++9N21tbdm8eXNfj8U+rsgC\nHDNmTFasWJHk8f+7fPTRR/fxROXq7OzMvHnzkqT3JWNemvHMVq9enXHjxvX1GMVbsmRJFi5cmOTx\nl0bVajX/s4jn7ZhjjsmqVauSJCtWrMgJJ5zQxxOV56c//WnGjRuXb37zm3nrW9+a5ubmDB48uK/H\nKs7mzZvzoQ99KOecc07e+9739vU4RTv++ONz2223JXn8a4Rt27blxS9+cR9PVZ5t27b1fo/hiBEj\n0t3d7VVqz+Ab3/hGOjo60tHRkVe96lW5/PLL85KXvKSvx2IfV+QNlt785jfn9ttvz5QpU5I8fmMO\nntkpp5ySOXPmZNq0aUke35WweGbr1q3LK17xir4eo3itra1pa2vL9OnTs2PHjpx77rm+YN6NWq3W\n1yMU68+7aWtry9y5c9Pd3Z1Ro0altbW1jycrz1FHHZVZs2Zl4cKFGTx4cC655JK+HqlIV199dbq6\nurJgwYIsWLAgyeMnY0OGDOnjycrzpje9KatXr84pp5ySnp6etLe3+/PqGcycOTNz5szJaaedlh07\nduTss8/O0KFD+3os+rEn/ndWq9X8d/c81OpecA4AAEBhHMEBAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgDFWbZsWTo6OpIkM2bM6ONpnp+LLrooN910UzZu3JiPfOQjz/p79/S5\n/uxnP3vGz5kxY0ZWrVq1R4/1XCxatCg33nhjkmTOnDlZv379Ln9vZ2dn5syZ85yu80I+n3vuuSfz\n589/QR4LgMYSqwAUZfv27fnyl7+c0047LUmyevXqPp7o+fnzD4N/6UtfmmuuueZZf+8L+Vz3xg+h\nnzJlSt7//vcnSe6444709PQ0bJ4X6vm85jWvyYYNG/LLX/7yBXk8ABpnYF8PAED/cM011+T73/9+\ndu7cmRNPPDHnnHNOfve73+XDH/5wDjnkkAwZMiTvete70tnZmT/84Q+ZOHFiZsyYkXPPPTfr16/P\nwIEDM2vWrJx00km56qqrctddd2XDhg2ZPn16pk6d2nudm2++OWPHjs2AAQNyySWXJEne//7358Yb\nb8y4cePy2te+Nps3b86nPvWpfOlLX+o9gW1ra8tf//VfZ/LkyVmyZEm+/vWvp6enJ695zWvS3t6e\nwYMHV3qed9xxR770pS8lSTZs2JDXv/71ueSSS7Jx48be5zp06NBce+21ufzyy7N69ers3LkzkydP\nzgc/+MHU6/VcccUVufXWW/OSl7wkgwYNyute97r87ne/ywc+8IHceuuteeCBBzJnzpz83//9X4YO\nHZpLLrkkixcvftJzXbFiRa666qrs2LEjRxxxRC6++OIcfPDBuf322zNv3rwMGjQoo0ePftbn8tBD\nD+WDH/xgZs2alYkTJz7jP8Mrr7wy9Xo9s2bNSvL4CelJJ52Ut7/97b2P09bWlkGDBmXt2rXZsmVL\nPvrRj+bd7353rrrqqtRqtQwePDgbN27MGWeckW984xtZu3ZtLr/88vT09OTwww/PZz7zmdTr9dx/\n//2ZMWNG1q9fn/Hjx+fiiy9+2szz58/P0qVLM3DgwLz//e/PBz7wgd6P7dy5M+3t7fn1r3+dzZs3\n56ijjsoXvvCFdHd3Z/bs2dm8eXOS5KyzzsrEiRPzta99LUuWLElTU1Ne97rX5aKLLkqSvPOd78xX\nv/rVzJs3r9K/EwD0DSerAOzWihUrcs899+Rb3/pWbrrppmzYsCE333xzkmTdunX5zGc+k6997Wup\n1+vZuHFjvv3tb2fWrFm5+OKLM378+Nx88835l3/5l3z605/OQw89lCTp7u7Ov/3bvz0pVJNk+fLl\nOeGEE5Ik5513XpL0vtT0D3/4Q84444wsWbIkAwYMeNLn1Wq11Gq1/OpXv8rixYuzaNGiLFmyJIcc\ncki+8pWv7NHzXbNmTS666KJ873vfy2OPPZZvfvObT3quX/3qV3PjjTemVquls7MzixcvzrJly/Jf\n//Vf+fd///fcfffd+e53v5svfvGL+c1vfvO0x7/wwgvT2tqaW265JWeddVauvvrqJz3Xhx9+OJ/9\n7Gfz1a9+NTfddFPe8IY35DOf+Uy2b9+ef/qnf8rnPve5dHZ2Zvjw4bt8Dn/84x/zkY98JB/72Mcy\nceLEXf4zPPnkk/Od73wnSfLII4/kP//zP/PmN7/5abvduHFjFi9enK9//eu54oorsnnz5t7Tzo98\n5CO9J8fDhg3LOeeck8svvzy33HJLjj766CxZsiS1Wi3r16/PggUL8t3vfjcrVqzI//zP/zzpOt/7\n3vdy55135jvf+U4WL16czs7O3gCt1+u58847M2TIkCxatCg//OEP8+ijj+Y//uM/snTp0hxxxBHp\n7OzM/Pnz89///d/ZuXNnrrnmmnR2dqazszNNTU158MEHkyQnnHBCli9fvkf/TgCw9zlZBWC3fvzj\nH+enP/1p3vve9yZJHnvssRxxxBE5/vjj81d/9Vc57LDDen/vq1/96jQ1Pf7/Qu+4445ceumlSZKX\nv/z/b+/eQqL69gCOf3fOlFGkZmohR7uJYpGMaFmCl8ZKC8vJa/qgkD5IKYEhRGU+mA8ZCUm+WEGB\nlVFhBBp2lQgTM1IjTDAUHUTD0YdRSedyHqR9nGZKzkv5P+f3eduuva5blN/81l7zL8LCwujq6kJR\nFMLCwlz2NTg4yPr16385ll/Vg/mApr29ncHBQTIyMoD5oHjbtm3/1Xx3795NQEAAAEeOHOH+/fvs\n27fPYa5tbW309vby7t07AGZmZujr66O/v58DBw7g5uaGh4cHer3eqf2Ojg6qq6sBiI2NJTY21qG8\nq6uLkZER9X1Uq9WKp6cnfX19+Pr6snXrVgDS0tKorKx0OYfy8nJ8fHzYv3+/Ol5Xz/Dw4cP4+/vT\n0dGB0WgkLi4OrVbr1F56ejrLli3Dz8+P8PBwOjs7Xfb75csX/Pz8CAkJAVAzto8ePSIiIoI1a9YA\nEBAQwMTEhEPd9+/fc/DgQbRaLVqtlsbGRrVMURQiIiLw8PCgvr6er1+/Mjg4yPT0NDqdjitXrjA6\nOkpcXByFhYW4ubmh0+lITU1Fr9eTk5ODn58fAKtXr8ZutzM5OYmnp6fLeQghhPj7JFgVQgixKJvN\nRm5uLnl5ecB81k6j0TAxMcGKFSvU+xRFwd3dXb222+0O7djtdqxWK4BDvYUURXHKmi70Yzvvz+8w\nzs3NqWNNTExUM5VTU1Nqnz+8fPmSq1evAqDX6ykqKnIo12j+8+/RZrOp41k4ZpvNRmlpKQkJCQCY\nTCZWrVpFVVWVw7ubruai1Wod1qa/v58tW7ao11arlfDwcHU78vfv35mammJkZMSh3o8PBVwpKCig\ntbWV+vp6cnJyfvkMAVJTU3ny5AkjIyNOa+FqHgvXxNXcFjKbzZjNZhRFcVhXcP790Gg0Dj8bHh5m\n7dq16r0vXrygpqaG3NxcUlNTmZycBCAwMJDm5mbevHnDq1evuHnzJs3NzdTW1tLV1UVrayv5+flc\nvnyZyMhIta/frZ8QQoi/T/5KCyGEWFRUVBSPHz9menoai8XCyZMnefbsmdN9Pwcfu3bt4sGDBwAM\nDQ3x4cMHdDqd030LBQQEYDQa1Ws3NzenYBPAy8uLoaEhZmdnmZycpLOzE0VR2LlzJ8+fP8dkMmG3\n2ykvL+f27dsOdffu3UtjYyONjY0ug7P29na+ffuGzWajsbGR2NhYpzFHRUXR0NCAxWLBbDaTnZ1N\nd3c3e/bsoampidnZWcxmM69fv3ZqPyIigqamJgDevn3L+fPnHeYaFhbGx48fGRgYAKC2tpaqqiqC\ng4MZHx/n8+fPAOr2XVdCQ0O5cOEC165dY3R09LfPMDExkba2NsbHx9mxY4dTW3a7Xe3LaDTS1dVF\nZGSkw5poNBosFgubNm3CZDKpW3zr6uq4d+/eL8e5UGRkJC0tLVgsFmZmZsjPz2dsbEwtb2trIykp\nCYPBgLe3Nx0dHVgsFu7evUtNTQ2JiYmUlZVhMpmYmJggKSmJoKAgiouLiY6OVg9VMpvN2O12Ncsr\nhBBiaZLMqhBCiEXFx8fT29tLRkYGVquVmJgYUlJSGB4edshw/pztPHfuHGVlZTx8+BBFUbh48SLr\n1q377cmu8fHxtLe3ExMTA8xnPlNSUtQ2fggKCiI2NpZDhw7h7++vvucaEhLCiRMnyM3NxWazERoa\nuuhXxvzM19eX06dPMzY2RnR0NOnp6RiNRof+s7KyGBgYwGAwYLFYSEtLU7N2nz59Ijk5GS8vLzZv\n3qyuzY/6ZWVlnD17ljt37rBy5Ur1IKmFc62srOTUqVNYrVY2bNhAVVUVGo2G6upqzpw5g5ubG9u3\nb//tWgYGBpKdnU1FRQU1NTUunyHMZ4x1Oh3BwcEu21EUhenpaY4ePcrc3BwVFRV4eHg49B0XF0dB\nQQE3btygqqqK0tJS5ubmCAwM5NKlSzx9+nTRdU9ISKCnpweDwYDdbicvL4+NGzeqY8jIyKCkpISW\nlhZ8fHzQ6/UYjUaOHz9OSUkJycnJaLVaioqK8PLyIjMzk7S0NNzd3fH398dgMADz27Dj4+MXHY8Q\nQoi/S7H/7uNtIYQQ4g+bnZ3l2LFjNDQ0OG0b/RPa29upq6vj+vXrf7zvv8VsNpOVlcWtW7fw9vZ2\nKnd1QvA/WXFxMUVFRYuepiyEEOLvkm3AQgghlpTly5dTWFionsD7py3MgP4/6O7uRq/Xk5mZ6TJQ\n/V/T09ODv7+/BKpCCPEPIJlVIYQQQgghhBBLjmRWhRBCCCGEEEIsORKsCiGEEEIIIYRYciRYFUII\nIYQQQgix5EiwKoQQQgghhBBiyZFgVQghhBBCCCHEkiPBqhBCCCGEEEKIJeffrzM+1wB2RDcAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113f53f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "theano_accuracy_report(exp.network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:data/beatles/models/chords_keys_synth_theano_01.model: saved model\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:09:42 theanets.graph:470 data/beatles/models/chords_keys_synth_theano_01.model: saved model\n"
     ]
    }
   ],
   "source": [
    "exp.save('data/beatles/models/chords_keys_synth_theano_01.model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.feedforward:layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 theanets.layers.feedforward:41 layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 theanets.layers.base:316 layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"out\": (hid1:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 theanets.layers.base:316 layer Feedforward \"out\": (hid1:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:network has 20512 total parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 theanets.graph:114 network has 20512 total parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.dataset:144 valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.dataset:144 train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- patience = 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:378 -- patience = 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- validate_every = 10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:379 -- validate_every = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- min_improvement = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:380 -- min_improvement = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:381 -- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:382 -- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:383 -- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- momentum = 0.9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:384 -- momentum = 0.9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- nesterov = False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:385 -- nesterov = False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.adaptive:220 -- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.adaptive:221 -- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling evaluation function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:08 downhill.base:112 compiling evaluation function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling RMSProp function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:10 downhill.base:118 compiling RMSProp function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 0 loss=3.061227 err=3.061227 acc=0.084632 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:13 downhill.base:226 validation 0 loss=3.061227 err=3.061227 acc=0.084632 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 1 loss=0.990295 err=0.990295 acc=0.726495\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:16 downhill.base:226 RMSProp 1 loss=0.990295 err=0.990295 acc=0.726495\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 2 loss=0.834510 err=0.834510 acc=0.751580\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:19 downhill.base:226 RMSProp 2 loss=0.834510 err=0.834510 acc=0.751580\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 3 loss=0.784133 err=0.784133 acc=0.761101\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:22 downhill.base:226 RMSProp 3 loss=0.784133 err=0.784133 acc=0.761101\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 4 loss=0.752851 err=0.752851 acc=0.767729\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:24 downhill.base:226 RMSProp 4 loss=0.752851 err=0.752851 acc=0.767729\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 5 loss=0.729180 err=0.729180 acc=0.774395\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:27 downhill.base:226 RMSProp 5 loss=0.729180 err=0.729180 acc=0.774395\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 6 loss=0.710382 err=0.710382 acc=0.779407\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:29 downhill.base:226 RMSProp 6 loss=0.710382 err=0.710382 acc=0.779407\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 7 loss=0.693436 err=0.693436 acc=0.783666\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:32 downhill.base:226 RMSProp 7 loss=0.693436 err=0.693436 acc=0.783666\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 8 loss=0.677871 err=0.677871 acc=0.788716\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:35 downhill.base:226 RMSProp 8 loss=0.677871 err=0.677871 acc=0.788716\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 9 loss=0.663869 err=0.663869 acc=0.792025\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:37 downhill.base:226 RMSProp 9 loss=0.663869 err=0.663869 acc=0.792025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 10 loss=0.650934 err=0.650934 acc=0.796806\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:40 downhill.base:226 RMSProp 10 loss=0.650934 err=0.650934 acc=0.796806\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 1 loss=0.700569 err=0.700569 acc=0.784078 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:40 downhill.base:226 validation 1 loss=0.700569 err=0.700569 acc=0.784078 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 11 loss=0.638698 err=0.638698 acc=0.801153\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:43 downhill.base:226 RMSProp 11 loss=0.638698 err=0.638698 acc=0.801153\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 12 loss=0.627088 err=0.627088 acc=0.804596\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:46 downhill.base:226 RMSProp 12 loss=0.627088 err=0.627088 acc=0.804596\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 13 loss=0.616982 err=0.616982 acc=0.807375\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:49 downhill.base:226 RMSProp 13 loss=0.616982 err=0.616982 acc=0.807375\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 14 loss=0.607168 err=0.607168 acc=0.810185\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:52 downhill.base:226 RMSProp 14 loss=0.607168 err=0.607168 acc=0.810185\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 15 loss=0.597838 err=0.597838 acc=0.813955\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:55 downhill.base:226 RMSProp 15 loss=0.597838 err=0.597838 acc=0.813955\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 16 loss=0.589798 err=0.589798 acc=0.815715\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:10:58 downhill.base:226 RMSProp 16 loss=0.589798 err=0.589798 acc=0.815715\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 17 loss=0.582369 err=0.582369 acc=0.818303\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:00 downhill.base:226 RMSProp 17 loss=0.582369 err=0.582369 acc=0.818303\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 18 loss=0.575377 err=0.575377 acc=0.821081\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:03 downhill.base:226 RMSProp 18 loss=0.575377 err=0.575377 acc=0.821081\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 19 loss=0.568472 err=0.568472 acc=0.823392\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:06 downhill.base:226 RMSProp 19 loss=0.568472 err=0.568472 acc=0.823392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 20 loss=0.562110 err=0.562110 acc=0.825382\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:08 downhill.base:226 RMSProp 20 loss=0.562110 err=0.562110 acc=0.825382\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 2 loss=0.691086 err=0.691086 acc=0.790945 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:09 downhill.base:226 validation 2 loss=0.691086 err=0.691086 acc=0.790945 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 21 loss=0.556801 err=0.556801 acc=0.826978\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:12 downhill.base:226 RMSProp 21 loss=0.556801 err=0.556801 acc=0.826978\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 22 loss=0.551639 err=0.551639 acc=0.828700\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:15 downhill.base:226 RMSProp 22 loss=0.551639 err=0.551639 acc=0.828700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 23 loss=0.546399 err=0.546399 acc=0.830690\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:18 downhill.base:226 RMSProp 23 loss=0.546399 err=0.546399 acc=0.830690\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 24 loss=0.541094 err=0.541094 acc=0.832681\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:21 downhill.base:226 RMSProp 24 loss=0.541094 err=0.541094 acc=0.832681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 25 loss=0.537590 err=0.537590 acc=0.832624\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:24 downhill.base:226 RMSProp 25 loss=0.537590 err=0.537590 acc=0.832624\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 26 loss=0.533278 err=0.533278 acc=0.834768\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:27 downhill.base:226 RMSProp 26 loss=0.533278 err=0.533278 acc=0.834768\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 27 loss=0.528871 err=0.528871 acc=0.836077\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:30 downhill.base:226 RMSProp 27 loss=0.528871 err=0.528871 acc=0.836077\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 28 loss=0.525249 err=0.525249 acc=0.837021\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:32 downhill.base:226 RMSProp 28 loss=0.525249 err=0.525249 acc=0.837021\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 29 loss=0.521410 err=0.521410 acc=0.838068\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:35 downhill.base:226 RMSProp 29 loss=0.521410 err=0.521410 acc=0.838068\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 30 loss=0.518323 err=0.518323 acc=0.840090\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:37 downhill.base:226 RMSProp 30 loss=0.518323 err=0.518323 acc=0.840090\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 3 loss=0.706092 err=0.706092 acc=0.791811\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:37 downhill.base:226 validation 3 loss=0.706092 err=0.706092 acc=0.791811\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 31 loss=0.515352 err=0.515352 acc=0.840935\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:40 downhill.base:226 RMSProp 31 loss=0.515352 err=0.515352 acc=0.840935\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 32 loss=0.511889 err=0.511889 acc=0.840945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:43 downhill.base:226 RMSProp 32 loss=0.511889 err=0.511889 acc=0.840945\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 33 loss=0.509321 err=0.509321 acc=0.842638\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:45 downhill.base:226 RMSProp 33 loss=0.509321 err=0.509321 acc=0.842638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 34 loss=0.506315 err=0.506315 acc=0.843126\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:48 downhill.base:226 RMSProp 34 loss=0.506315 err=0.506315 acc=0.843126\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 35 loss=0.502970 err=0.502970 acc=0.844781\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:51 downhill.base:226 RMSProp 35 loss=0.502970 err=0.502970 acc=0.844781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 36 loss=0.500946 err=0.500946 acc=0.845766\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:54 downhill.base:226 RMSProp 36 loss=0.500946 err=0.500946 acc=0.845766\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 37 loss=0.499128 err=0.499128 acc=0.845821\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:11:57 downhill.base:226 RMSProp 37 loss=0.499128 err=0.499128 acc=0.845821\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 38 loss=0.496116 err=0.496116 acc=0.846554\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:00 downhill.base:226 RMSProp 38 loss=0.496116 err=0.496116 acc=0.846554\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 39 loss=0.493522 err=0.493522 acc=0.847139\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:03 downhill.base:226 RMSProp 39 loss=0.493522 err=0.493522 acc=0.847139\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 40 loss=0.491133 err=0.491133 acc=0.848717\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:06 downhill.base:226 RMSProp 40 loss=0.491133 err=0.491133 acc=0.848717\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 4 loss=0.723751 err=0.723751 acc=0.789733\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:06 downhill.base:226 validation 4 loss=0.723751 err=0.723751 acc=0.789733\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 41 loss=0.488863 err=0.488863 acc=0.849534\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:09 downhill.base:226 RMSProp 41 loss=0.488863 err=0.488863 acc=0.849534\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 42 loss=0.486897 err=0.486897 acc=0.849977\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:13 downhill.base:226 RMSProp 42 loss=0.486897 err=0.486897 acc=0.849977\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 43 loss=0.484993 err=0.484993 acc=0.849370\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:16 downhill.base:226 RMSProp 43 loss=0.484993 err=0.484993 acc=0.849370\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 44 loss=0.482839 err=0.482839 acc=0.851113\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:19 downhill.base:226 RMSProp 44 loss=0.482839 err=0.482839 acc=0.851113\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 45 loss=0.481664 err=0.481664 acc=0.851292\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:21 downhill.base:226 RMSProp 45 loss=0.481664 err=0.481664 acc=0.851292\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 46 loss=0.479087 err=0.479087 acc=0.852179\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:24 downhill.base:226 RMSProp 46 loss=0.479087 err=0.479087 acc=0.852179\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 47 loss=0.477889 err=0.477889 acc=0.852854\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:26 downhill.base:226 RMSProp 47 loss=0.477889 err=0.477889 acc=0.852854\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 48 loss=0.475876 err=0.475876 acc=0.853265\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:29 downhill.base:226 RMSProp 48 loss=0.475876 err=0.475876 acc=0.853265\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 49 loss=0.474521 err=0.474521 acc=0.853728\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:31 downhill.base:226 RMSProp 49 loss=0.474521 err=0.474521 acc=0.853728\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 50 loss=0.472296 err=0.472296 acc=0.853969\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:34 downhill.base:226 RMSProp 50 loss=0.472296 err=0.472296 acc=0.853969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 5 loss=0.750334 err=0.750334 acc=0.785607\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:34 downhill.base:226 validation 5 loss=0.750334 err=0.750334 acc=0.785607\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 51 loss=0.470513 err=0.470513 acc=0.854681\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:37 downhill.base:226 RMSProp 51 loss=0.470513 err=0.470513 acc=0.854681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 52 loss=0.468879 err=0.468879 acc=0.855499\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:40 downhill.base:226 RMSProp 52 loss=0.468879 err=0.468879 acc=0.855499\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 53 loss=0.467853 err=0.467853 acc=0.855583\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:42 downhill.base:226 RMSProp 53 loss=0.467853 err=0.467853 acc=0.855583\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 54 loss=0.466314 err=0.466314 acc=0.855660\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:45 downhill.base:226 RMSProp 54 loss=0.466314 err=0.466314 acc=0.855660\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 55 loss=0.464907 err=0.464907 acc=0.857152\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:48 downhill.base:226 RMSProp 55 loss=0.464907 err=0.464907 acc=0.857152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 56 loss=0.462803 err=0.462803 acc=0.857460\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:51 downhill.base:226 RMSProp 56 loss=0.462803 err=0.462803 acc=0.857460\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 57 loss=0.462073 err=0.462073 acc=0.858171\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:55 downhill.base:226 RMSProp 57 loss=0.462073 err=0.462073 acc=0.858171\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 58 loss=0.461382 err=0.461382 acc=0.857565\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:12:58 downhill.base:226 RMSProp 58 loss=0.461382 err=0.461382 acc=0.857565\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 59 loss=0.458700 err=0.458700 acc=0.858576\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:01 downhill.base:226 RMSProp 59 loss=0.458700 err=0.458700 acc=0.858576\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 60 loss=0.457813 err=0.457813 acc=0.858354\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:04 downhill.base:226 RMSProp 60 loss=0.457813 err=0.457813 acc=0.858354\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 6 loss=0.775764 err=0.775764 acc=0.783039\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:04 downhill.base:226 validation 6 loss=0.775764 err=0.775764 acc=0.783039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 61 loss=0.456298 err=0.456298 acc=0.859989\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:08 downhill.base:226 RMSProp 61 loss=0.456298 err=0.456298 acc=0.859989\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 62 loss=0.455526 err=0.455526 acc=0.859691\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:10 downhill.base:226 RMSProp 62 loss=0.455526 err=0.455526 acc=0.859691\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 63 loss=0.454037 err=0.454037 acc=0.860779\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:13 downhill.base:226 RMSProp 63 loss=0.454037 err=0.454037 acc=0.860779\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 64 loss=0.453662 err=0.453662 acc=0.861634\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:15 downhill.base:226 RMSProp 64 loss=0.453662 err=0.453662 acc=0.861634\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 65 loss=0.452126 err=0.452126 acc=0.861153\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:18 downhill.base:226 RMSProp 65 loss=0.452126 err=0.452126 acc=0.861153\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 66 loss=0.450701 err=0.450701 acc=0.861710\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:21 downhill.base:226 RMSProp 66 loss=0.450701 err=0.450701 acc=0.861710\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 67 loss=0.449561 err=0.449561 acc=0.862163\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:24 downhill.base:226 RMSProp 67 loss=0.449561 err=0.449561 acc=0.862163\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 68 loss=0.448446 err=0.448446 acc=0.863114\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:27 downhill.base:226 RMSProp 68 loss=0.448446 err=0.448446 acc=0.863114\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 69 loss=0.447633 err=0.447633 acc=0.862182\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:29 downhill.base:226 RMSProp 69 loss=0.447633 err=0.447633 acc=0.862182\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 70 loss=0.447253 err=0.447253 acc=0.863251\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:32 downhill.base:226 RMSProp 70 loss=0.447253 err=0.447253 acc=0.863251\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 7 loss=0.797328 err=0.797328 acc=0.783674\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:32 downhill.base:226 validation 7 loss=0.797328 err=0.797328 acc=0.783674\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 71 loss=0.445966 err=0.445966 acc=0.862625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:34 downhill.base:226 RMSProp 71 loss=0.445966 err=0.445966 acc=0.862625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 72 loss=0.444749 err=0.444749 acc=0.863625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:37 downhill.base:226 RMSProp 72 loss=0.444749 err=0.444749 acc=0.863625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 73 loss=0.443551 err=0.443551 acc=0.864000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:40 downhill.base:226 RMSProp 73 loss=0.443551 err=0.443551 acc=0.864000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 74 loss=0.442668 err=0.442668 acc=0.863684\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:42 downhill.base:226 RMSProp 74 loss=0.442668 err=0.442668 acc=0.863684\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 75 loss=0.441872 err=0.441872 acc=0.864625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:45 downhill.base:226 RMSProp 75 loss=0.441872 err=0.441872 acc=0.864625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 76 loss=0.440991 err=0.440991 acc=0.865021\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:48 downhill.base:226 RMSProp 76 loss=0.440991 err=0.440991 acc=0.865021\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 77 loss=0.439715 err=0.439715 acc=0.865116\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:51 downhill.base:226 RMSProp 77 loss=0.439715 err=0.439715 acc=0.865116\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 78 loss=0.439554 err=0.439554 acc=0.866031\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:54 downhill.base:226 RMSProp 78 loss=0.439554 err=0.439554 acc=0.866031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 79 loss=0.438084 err=0.438084 acc=0.865886\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:13:56 downhill.base:226 RMSProp 79 loss=0.438084 err=0.438084 acc=0.865886\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 80 loss=0.437310 err=0.437310 acc=0.865107\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:14:00 downhill.base:226 RMSProp 80 loss=0.437310 err=0.437310 acc=0.865107\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 8 loss=0.817045 err=0.817045 acc=0.780557\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:14:00 downhill.base:226 validation 8 loss=0.817045 err=0.817045 acc=0.780557\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:patience elapsed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:14:00 downhill.base:402 patience elapsed!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(OrderedDict([('loss', 0.4373099297977624), ('err', 0.4373099297977624), ('acc', 0.86510736725358472)]),\n",
       " OrderedDict([('loss', 0.7973277453240265), ('err', 0.7973277453240265), ('acc', 0.78367382271468145)]))"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# two hidden layers of size 100, autoencoer pretraining, no regularization\n",
    "exp02 = theanets.Experiment(theanets.Classifier, layers=[X.shape[1], 100, 12])\n",
    "exp02.train(\n",
    "    (X_train.astype(float32),y_train),\n",
    "    (X_valid.astype(float32),y_valid),\n",
    "    algo='pretraining', learning_rate=1e-4, momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training score: 0.832934160535\n",
      "validation score: 0.790945290859\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2U1nWd//HXNdyKSOY25VEqjMMxu/OIDgsnpQ4/qqnt\njtQCgWyjTbe1WjDbIcXxdkXplJtRYlbHJhOXzkjadrOBnMWkFXZTSsludsXKQEC3GhBlYK7fH54m\nTZEv6sV8Bh6Pv5hrmO/3fb0B8cn3e11Tq9fr9QAAAEBBmvp6AAAAAPhLYhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4gxs1IF7enpy7rnnZv369WlqasrFF1+cAQMGpK2tLU1NTRkzZkza29tT\nq9UaNQIAAAD9VMNi9Yc//GG2b9+eG264IatWrcpnP/vZ7Ny5M3PmzElLS0va29uzfPnyTJ48uVEj\nAAAA0E817DbgoUOHpqurK/V6PV1dXRk0aFDuueeetLS0JEkmTpyYVatWNer0AAAA9GMNu7I6duzY\n7NixI62trfn973+fq6++OmvWrOn9/LBhw9LV1dWo0wMAANCPNSxWr7322owdOzazZ8/Oxo0b8/73\nvz87d+7s/fy2bdsyYsSIPR6nXq97XSsAAMABpmGxun379hx88MFJkhEjRmTnzp151ateldWrV2fc\nuHFZuXJlJkyYsMfj1Gq1bN7sCuyeNDcfYk8V2VU19lSNPVVnV9XYU3V2VY09VWNP1dlVNfZUXXPz\nIU/7eMNiddasWZk7d25OO+207Ny5M2effXZe/epXZ968eenu7s7o0aPT2traqNMDAADQjzUsVkeM\nGJGFCxc+5fGOjo5GnRIAAID9RMPeDfj5Uq/X+3oEAAAA9rHiY7WpqZ6TTx4kWgEAAA4gxcdq0pTb\nbhuSY489KHfeKVgBAAAOBP0gVpOklo0bB+X00w9yhRUAAOAA0E9iFQAAgANJw94N+PlVz+GH78x1\n121PrVbr62EAAABosH4Rq4cfvjNr1wpVAACAA0XxtwEPGFB3RRUAAOAAU3ysjhyZHHecUAUAADiQ\nFB+riVAFAAA40PSDWAUAAOBAI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKM7ARh78pptuSmdnZ5Lksccey7333ptvfOMbufTSS9PU\n1JQxY8akvb09tVqtkWMAAADQzzT0yuqUKVPS0dGRjo6OvOY1r8m8efOycOHCzJkzJ9dff33q9XqW\nL1/eyBEAAADoh/bJbcA//elP86tf/Sqnnnpq7rnnnrS0tCRJJk6cmFWrVu2LEQAAAOhH9kmsLlq0\nKGeddVaSpF6v9z4+bNiwdHV17YsRAAAA6Eca+prVJPnjH/+Y9evXZ9y4cUmSpqY/9/G2bdsyYsSI\nPR6jufmQhs23P7Gn6uyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7em4aHqtr1qzJ+PHjez8+5phj\nsnr16owbNy4rV67MhAkT9niMzZtdfd2T5uZD7Kkiu6rGnqqxp+rsqhp7qs6uqrGnauypOruqxp6q\n213UNzxW169fn5e97GW9H7e1tWXevHnp7u7O6NGj09ra2ugRAAAA6GcaHquzZs160sejRo1KR0dH\no08LAABAP7ZP3mAJAAAA9oZYBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIozsJEHX7RoUVasWJHu7u7MmDEjY8eOTVtbW5qa\nmjJmzJi0t7enVqs1cgQAAAD6oYZdWb3jjjty5513ZvHixeno6MhvfvObzJ8/P3PmzMn111+fer2e\n5cuXN+r0AAAA9GMNi9Xbb789Rx99dD7ykY/kzDPPzKRJk3LPPfekpaUlSTJx4sSsWrWqUacHAACg\nH2vYbcAPP/xwNmzYkEWLFuU3v/lNzjzzzNTr9d7PDxs2LF1dXY06PQAAAP1Yw2L1hS98YUaPHp2B\nAwfmqKOOypAhQ7Jp06bez2/bti0jRoyodKzm5kMaNeZ+xZ6qs6tq7Kkae6rOrqqxp+rsqhp7qsae\nqrOrauzpuWlYrB5//PH52te+lr/927/Ngw8+mEcffTTjx4/P6tWrM27cuKxcuTITJkyodKzNm12B\n3ZPm5kPsqSK7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnqrbXdQ3LFbf+MY3Zs2aNTnllFPS09OT\n9vb2HHnkkZk3b166u7szevTotLa2Nur0AAAA9GMN/dY155xzzlMe6+joaOQpAQAA2A807N2AAQAA\n4NkSqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEes\nAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFCcgY0+wZQpUzJ8+PAkyUtf+tKcccYZaWtrS1NTU8aMGZP29vbUarVGjwEAAEA/0tBY\nfeyxx5IkHR0dvY+deeaZmTNnTlpaWtLe3p7ly5dn8uTJjRwDAACAfqahtwHfe++92b59e2bNmpXT\nTz89d911V9atW5eWlpYkycSJE7Nq1apGjgAAAEA/1NArqwcddFBmzZqVU089NevXr8+HPvShJ31+\n2LBh6erqauQIAAAA9EMNjdVRo0bl5S9/ee+PDz300PzsZz/r/fy2bdsyYsSIPR6nufmQhs24P7Gn\n6uyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7em4aGqudnZ35+c9/nvb29jz44IPZtm1bXv/612f1\n6tUZN25cVq5cmQkTJuzxOJs3u/q6J83Nh9hTRXZVjT1VY0/V2VU19lSdXVVjT9XYU3V2VY09Vbe7\nqG9orJ5yyimZO3dupk+fniS57LLLcuihh2bevHnp7u7O6NGj09ra2sgRAAAA6IcaGqsDBw7MggUL\nnvL4E98dGAAAAP5SQ98NGAAAAJ4NsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUZ69i9dFHH83WrVv36gQPPfRQ3vCGN+S+\n++7L/fffn2nTpmX69Om54IILUq/X9+pYAAAAHBgqx+qSJUvy3ve+N1OnTs2VV15Z6Wu6u7tz/vnn\n56CDDkq9Xs9ll12WOXPm5Prrr0+9Xs/y5cuf9eAAAADsv3Ybq7/4xS+e9PHy5ctz880359vf/naW\nLVtW6eBXXHFFpk2blubm5iTJunXr0tLSkiSZOHFiVq1a9WznBgAAYD82cHefuPHGG9Pd3Z1/+Id/\nyEte8pK88pWvzKxZszJw4MCMGTNmjwfu7OzMYYcdlhNPPDGLFi1KvV5/0m2/w4YNS1dX1/PzLAAA\nANiv1OrP8MLR++67LwsXLswRRxyRv/u7v8vmzZuzY8eOvPKVr9zjgWfMmJFarZYkuffeezNq1Kj8\n7Gc/y913350kWbZsWX70ox9l3rx5z3icUaOS9eurPyEAAAD6v2eM1T/58Y9/nGuuuSbjx4/P9OnT\nM2jQoL06ycyZM3PhhRfmiiuuyAc/+MGMGzcu559/fiZMmJC3vvWtz/i1o0Yla9a4Arsnzc2HZPNm\ne6rCrqqxp2rsqTq7qsaeqrOrauypGnuqzq6qsafqmpsPedrHd/ua1euvvz6TJ0/OW97ylmzatClX\nX311jjjiiJx55pm5+eab93qAWq2Wtra2XHXVVZk6dWp27dqV1tbWvT4OAAAA+7/dvmb1hhtuyPe+\n973s2LEj06dPT2tra9785jdn0qRJ+dd//de9OklHR8fT/hgAAACezm5jtbm5Of/8z/+cRx99NK94\nxSv+/AUDB+a0007bJ8MBAABwYNptrF599dW57bbbMnjw4Lz+9a/flzMBAABwgNttrA4ZMiSTJ0/e\nl7MAAABAkmd4gyUAAADoK2IVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOAMbefBdu3blvPPOy/r161Or1XLhhRdm8ODBaWtrS1NTU8aMGZP29vbUarVGjgEA\nAEA/09BYXbFiRZqamnLDDTdk9erV+cxnPpMkmTNnTlpaWtLe3p7ly5dn8uTJjRwDAACAfqahtwFP\nnjw5F110UZLkgQceyAte8ILcc889aWlpSZJMnDgxq1atauQIAAAA9EMNf83qgAED0tbWlksvvTTv\neMc7Uq/Xez83bNiwdHV1NXoEAAAA+pla/Yn12EBbtmzJqaeemkceeSR33HFHkmTZsmX50Y9+lHnz\n5u3260aNStav3xcTAgAAUIqGvmZ16dKlefDBB3PGGWdk6NChaWpqymte85qsXr0648aNy8qVKzNh\nwoQ9HmfzZldf96S5+RB7qsiuqrGnauypOruqxp6qs6tq7Kkae6rOrqqxp+qamw952scbGqutra1p\na2vLjBkzsnPnzpx77rl5xStekXnz5qW7uzujR49Oa2trI0cAAACgH2porA4dOjRXXnnlUx7v6Oho\n5GkBAADo5xr+BksAAACwt8QqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFGdgow7c3d2dT33qU/nd736XHTt25O///u8zevTotLW1pamp\nKWPGjEl7e3tqtVqjRgAAAKCfalis3nLLLTnssMOyYMGC/OEPf8i73vWuHHPMMZkzZ05aWlrS3t6e\n5cuXZ/LkyY0aAQAAgH6qYbcBt7a25mMf+1iSpKenJwMHDsy6devS0tKSJJk4cWJWrVrVqNMDAADQ\njzUsVocNG5aDDz44W7duzcc//vH84z/+Y3p6ep70+a6urkadHgAAgH6sYbcBJ8mGDRty1llnZfr0\n6Xn729+eBQsW9H5u27ZtGTFiRKXjNDcf0qgR9yv2VJ1dVWNP1dhTdXZVjT1VZ1fV2FM19lSdXVVj\nT89Nw2J1y5Yt+eAHP5j29vaMHz8+SXLMMcdk9erVGTduXFauXJkJEyZUOtbmza7A7klz8yH2VJFd\nVWNP1dhTdXZVjT1VZ1fV2FM19lSdXVVjT9XtLuobFqtXX311urq6snDhwixcuDBJcu655+bSSy9N\nd3d3Ro8endbW1kadHgAAgH6sYbF63nnn5bzzznvK4x0dHY06JQAAAPuJhr3BEgAAADxbYhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIoj\nVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5Y\nBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAoTsNjde3atZk5c2aS5P7778+0adMyffr0XHDBBanX640+PQAAAP1QQ2P1S1/6Us4777x0\nd3cnSS677LLMmTMn119/fer1epYvX97I0wMAANBPNTRWX/7yl+fzn/987xXUdevWpaWlJUkyceLE\nrFq1qpGnBwAAoJ9qaKy++c1vzoABA3o/fuJtv8OGDUtXV1cjTw8AAEA/NXBfnqyp6c9tvG3btowY\nMaLS1zU3H9KokfYr9lSdXVVjT9XYU3V2VY09VWdX1dhTNfZUnV1VY0/PzT6N1WOOOSarV6/OuHHj\nsnLlykyYMKHS123e7ArsnjQ3H2JPFdlVNfZUjT1VZ1fV2FN1dlWNPVVjT9XZVTX2VN3uon6fxGqt\nVkuStLW1Zd68eenu7s7o0aPT2tq6L04PAABAP9PwWB05cmQWL16cJBk1alQ6OjoafUoAAAD6uYZ/\nn1UAAADYW2IVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKI\nVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ojV/UC9Xk+9\nXu/rMQAAAJ43YrUfq9frOfnkQXnJSw5OU9PjPxatAADA/kCs9mOnnDI4t902JMmAJE257bYhOfbY\ng3LnnYIVAADo38RqP1Wv13PbbYOS1J7waC0bNw7K6acf5ArrbrhlGgAA+gexygHBLdMAANC/iNV+\nqlar5aSTupM8MbjqOfzw7lx33fbUarXdfekByS3TAADQv4jVfuyb39yRww/f2fvx4YfvzNq123Pc\ncUL1idwyDQAA/Y9Y7cdqtVquu257mprqGTCg7ooqAACw3xCr/dxxx9Vy5JH1jBwZV1R3wy3TAADQ\n/4jV/YbgeiZumQYAgP5FrHJAcMs0AAD0LwP7egDYV/50y3RTU80VVQAAKJwrqxyAhCoAAJROrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrwFPU6/XU6/W+HgMAgAOYWAV61ev1nHzyoLzkJQenqenxH4tWAAD6glgFep1yyuDcdtuQJAOS\nNOW224bk2GMPyp13ClYAAPYtsQokefyq6m23DUpSe8KjtWzcOCinn36QK6xPw+3SAACNI1YB9pLb\npQEAGk+sAkmSWq2Wk07qTvLE6Krn8MO7c91121Or1Xb3pQcct0sDADSeWAV6ffObO3L44Tt7Pz78\n8J1Zu3Z7jjtOqP6J26WfHbdMV2NPAPBnYhXoVavVct1129PUVM+AAXVXVHnO3DJdjT0BwFOJVeBJ\njjuuliOPrGfkyLii+jTcLr133DJdjT3tPVehAfZ/YhXYDdG1O26XrsYt09XY095xFRrgwCFWAfaS\n26Wh77gKDXDgEKsAz4LbpffMLdPV2FN1rkIDHFjEKsBzIiSeiVumq7EnGsVre4H+TKwC0DBuma7G\nnqpxFbo6r+0F9gdiFYCGcst0NfZUjavQ1XhtL7A/EKsA7CNiohp7eiauQu+Z1/YC+4uBfT0AAMDe\n+NNV6KammiuqPC+8trc6u2JfcmUVAOinhOrT8dre6ry2tzq72jui/vmxz2O1p6cn559/fqZOnZqZ\nM2fm17/+9b4eAQBgv+a1vdV4bW91dlWNqH9+7fNYXbZsWbq7u7N48eJ84hOfyPz58/f1CAAA+zWv\n7d0zr+2tzq6qE/XPr30eqz/+8Y9z0kknJUmOPfbY3H333ft6BACA/Z53mIZ9S9Q///b5Gyxt3bo1\nw4cP7/14wIAB6enpSVPT03fzD387Kocd7xf2mdz+u1pqiT1VYFfV2FM19lSdXVVjT9XZVTX29Mx+\nNyR57LGigHgUAAAT5UlEQVQnh/yApuRF9XoGn9BHQxXKrqq57wmhuiSn5pNZ0IfT9H/7PFaHDx+e\nbdu29X78TKGaJCN3rt8HU/VvL+3rAfoRu6rGnqqxp+rsqhp7qs6uqrGnZ3Z4vZ6RI3vyu98NSJIc\nccSu/Pa3TW6Zfhp2Vc2s/7czt946IH++ulrPEUf0ZOnSprz4xSP6crR+aZ/H6tixY7NixYq89a1v\nzV133ZWjjz56X48AAACp1Wp54IEBT3hkwG5/7oHOrqpZvvwv86oWu3r2avV9fPN0vV7PBRdckJ//\n/OdJkssuuyxHHXXUvhwBAACAwu3zWAUAAIA92efvBgwAAAB7IlYBAAAojlgFAACgOEXGak9PT84/\n//xMnTo1M2fOzK9//eu+HqlYN910U2bOnJmZM2fmve99b173utdl69atfT1WkaZMmdK7q0996lN9\nPU6xdu3alblz52batGk57bTT8stf/rKvRyra2rVrM3PmzL4eo0hP3M3999+fadOmZfr06bngggt8\nY/Sn0d3dnXPOOSfTp0/PqaeemltvvbWvRyrSjh078slPfjJTp07NjBkzcu+99/b1SEVbtGhRpk6d\nmpNPPjk33XRTX49TtIceeihveMMbct999/X1KEXq6enp/f+D6dOn53//93/7eqQi/eX/F/zgBz/I\n2Wef3YcT9W/7/FvXVLFs2bJ0d3dn8eLFWbt2bebPn58vfOELfT1WkaZMmZIpU6YkSS666KKceuqp\nGT58eB9PVZ7HHnssSdLR0dHHk5RvxYoVaWpqyg033JDVq1fns5/9rD9/u/GlL30pN998cw4++OC+\nHqU4f7mbyy67LHPmzElLS0va29uzfPnyTJ48uY+nLMstt9ySww47LAsWLMgf/vCHvPvd786kSZP6\neqziLFmyJEOHDs3ixYtz33335eyzz05nZ2dfj1WkO+64I3feeWcWL16cRx55JNdee21fj1Ss7u7u\nnH/++TnooIP6epRi/fCHP8z27dtzww03ZNWqVbnyyivzuc99rq/HKspf/t13ySWX5Pbbb8+rXvWq\nPp6s/yryyuqPf/zjnHTSSUmSY489NnfffXcfT1S+n/70p/nlL3+ZU089ta9HKdK9996b7du3Z9as\nWTn99NOzdu3avh6pWJMnT85FF12UJHnggQfyghe8oI8nKtfLX/7yfP7zn3eV8Gn85W7WrVuXlpaW\nJMnEiROzatWqvhyvSK2trfnYxz6W5PErGAMG+L58T+dXv/pVJk6cmCQ56qij8uCDD7qjaDduv/32\nHH300fnIRz6SM8880z9+PIMrrrgi06ZNS3Nzc1+PUqyhQ4emq6sr9Xo9XV1dGTRoUF+PVJy//Ltv\n7Nix7iZ6joqM1a1btz7p6uCAAQPS09PThxOVb9GiRfnoRz/a12MU66CDDsqsWbPy5S9/ORdeeGE+\n8YlP+D31DAYMGJC2trZccsklefvb397X4xTrzW9+s6DYjb/czRP/oh42bFi6urr6YqyiDRs2LAcf\nfHC2bt2aj3/845k9e3Zfj1SkY445JitWrEiS3HXXXXn44YfzyCOP9PFUZXr44Ydz991353Of+1zv\n3308VWdnZw477LCceOKJSSIsdmPs2LHZsWNHWltbc/7552fGjBl9PVJx/vLvvre97W19OM3+ochY\nHT58eLZt29b7cU9PT5qaihy1CH/84x+zfv36jBs3rq9HKdaoUaPyzne+s/fHhx56aDZv3tzHU5Vt\n/vz5+f73v5958+bl0Ucf7etx6Oee+N/wbdu2ZcSIEX04Tbk2bNiQ008/Pe9+97vzN3/zN309TpFO\nPvnkDB8+PKeddlqWLVvW+990nuqFL3xhTjzxxAwcODBHHXVUhgwZkocffrivxypOZ2dnVq1alZkz\nZ+bee+9NW1tbtmzZ0tdjFefaa6/N2LFj8/3vfz/f+ta30tbWlh07dvT1WOzniizAsWPHZuXKlUke\n/1fTo48+uo8nKtuaNWsyfvz4vh6jaJ2dnZk/f36S9N4y5lafp7d06dIsWrQoyeO3/NRqNf9YxHN2\nzDHHZPXq1UmSlStX5oQTTujjicqzZcuWfPCDH8w555yT97znPX09TrF+8pOfZPz48fnGN76Rt7zl\nLWlubs7gwYP7eqwiHX/88bntttuSPP533/bt2/PCF76wj6cqz9e//vV0dHSko6Mjr3zlK3P55Zfn\nRS96UV+PVZzt27f3vhZzxIgR6e7udpcaDVfkGyy96U1vyu23356pU6cmefyNOdi99evX52Uve1lf\nj1G0U045JXPnzs306dOTPP57SoA9vdbW1rS1tWXGjBnZuXNnzj33XP8juAe1Wq2vRyjWn3bT1taW\nefPmpbu7O6NHj05ra2sfT1aeq6++Ol1dXVm4cGEWLlyY5PErGUOGDOnjycpy1FFHZfbs2Vm0aFEG\nDx6cSy65pK9HKtYb3/jGrFmzJqecckp6enrS3t7uv1c8a7NmzcrcuXNz2mmnZefOnTn77LMzdOjQ\nvh6rSE/8c1ar1fy5ew5qdTfmAwAAUBiXlgAAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgF\nAACgOGIVgOIsX748HR0dSZKZM2f28TTPzUUXXZSbbropmzZtyoc//OFn/Ll7+1x/+tOfPu3XzJw5\nM6tXr96rYz0bixcvzo033pgkmTt3bjZs2LDbn9vZ2Zm5c+c+q/M8n8/nnnvuyYIFC56XYwHQWGIV\ngKLs2LEjX/rSl3LaaaclSdasWdPHEz03f/pm8C9+8YtzzTXXPOPPfT6f6774JvRTp07N+973viTJ\nHXfckZ6enobN83w9n1e/+tXZuHFjfvGLXzwvxwOgcQb29QAA9A/XXHNNvve972XXrl058cQTc845\n5+S3v/1tPvShD+Wwww7LkCFD8s53vjOdnZ35/e9/n0mTJmXmzJk599xzs2HDhgwcODCzZ8/OSSed\nlKuuuip33XVXNm7cmBkzZmTatGm957n55pszbty4DBgwIJdcckmS5H3ve19uvPHGjB8/Pq95zWuy\nZcuWfPKTn8wXv/jF3iuwbW1t+eu//utMmTIlS5cuzde+9rX09PTk1a9+ddrb2zN48OBKz/OOO+7I\nF7/4xSTJxo0b87rXvS6XXHJJNm3a1Ptchw4dmmuvvTaXX3551qxZk127dmXKlCn5wAc+kHq9niuu\nuCK33nprXvSiF2XQoEF57Wtfm9/+9rd5//vfn1tvvTUPPPBA5s6dm//7v//L0KFDc8kll2TJkiVP\neq4rV67MVVddlZ07d2bkyJG5+OKLc+ihh+b222/P/PnzM2jQoIwZM+YZn8tDDz2UD3zgA5k9e3Ym\nTZr0tL+GV155Zer1embPnp3k8SukJ510Ut72trf1HqetrS2DBg3KunXrsnXr1nzkIx/Ju971rlx1\n1VWp1WoZPHhwNm3alDPOOCNf//rXs27dulx++eXp6enJkUcemU9/+tOp1+u5//77M3PmzGzYsCET\nJkzIxRdf/JSZFyxYkGXLlmXgwIF53/vel/e///29n9u1a1fa29vzq1/9Klu2bMlRRx2Vz3/+8+nu\n7s6cOXOyZcuWJMlZZ52VSZMm5atf/WqWLl2apqamvPa1r81FF12UJHnHO96Rr3zlK5k/f36l3xMA\n9A1XVgHYo5UrV+aee+7JN7/5zdx0003ZuHFjbr755iTJ+vXr8+lPfzpf/epXU6/Xs2nTpnzrW9/K\n7Nmzc/HFF2fChAm5+eab8y//8i/51Kc+lYceeihJ0t3dnX/7t397UqgmyYoVK3LCCSckSc4777wk\n6b3V9Pe//33OOOOMLF26NAMGDHjS19VqtdRqtfzyl7/MkiVLsnjx4ixdujSHHXZYvvzlL+/V8127\ndm0uuuiifPe7381jjz2Wb3zjG096rl/5yldy4403plarpbOzM0uWLMny5cvzX//1X/n3f//33H33\n3fnOd76TL3zhC/n1r3/9lONfeOGFaW1tzS233JKzzjorV1999ZOe68MPP5zPfOYz+cpXvpKbbrop\nr3/96/PpT386O3bsyD/90z/ls5/9bDo7OzN8+PDdPoc//OEP+fCHP5yPfvSjmTRp0m5/DU8++eR8\n+9vfTpI88sgj+c///M+86U1vespuN23alCVLluRrX/tarrjiimzZsqX3aueHP/zh3ivHw4YNyznn\nnJPLL788t9xyS44++ugsXbo0tVotGzZsyMKFC/Od73wnK1euzP/8z/886Tzf/e53c+edd+bb3/52\nlixZks7Ozt4ArdfrufPOOzNkyJAsXrw4P/jBD/Loo4/mP/7jP7Js2bKMHDkynZ2dWbBgQf77v/87\nu3btyjXXXJPOzs50dnamqakpDz74YJLkhBNOyIoVK/bq9wQA+54rqwDs0Y9+9KP85Cc/yXve854k\nyWOPPZaRI0fm+OOPz1/91V/liCOO6P25r3rVq9LU9Pi/hd5xxx259NJLkyQvfelL/3979xIS1dsH\ncPx7cqaMIjVTiwHtJopFomhZgpfGSgvLyWu6UEgXUkpgCFGZC3ORkZDkxgoKrIwKI9Cwq0TYYEZq\nhAmGooNoeFmMSjqX/0I6r9NMybsp/+/7++yOz3muR5Tf/J7zDKGhoXR2dqIoCqGhoS77GhgYYP36\n9b8cy6/qwXxAYzQaGRgYICMjA5gPirdt2/ZfzXf37t34+/sDcOTIEe7fv8++ffsc5trW1kZPTw/v\n3r0DYGZmht7eXvr6+jhw4ABubm54eHig1+ud2m9vb6e6uhqA2NhYYmNjHco7OzsZHh5W30e1Wq14\nenrS29uLr68vW7duBSAtLY3KykqXcygvL8fHx4f9+/er43X1DA8fPoxOp6O9vR2TyURcXBxardap\nvfT0dJYtW4afnx/h4eF0dHS47PfLly/4+fkRHBwMoGZsHz16REREBGvWrAHA39+fiYkJh7rv37/n\n4MGDaLVatFotjY2NapmiKERERODh4UF9fT1fv35lYGCA6elpwsLCuHLlCiMjI8TFxVFYWIibmxth\nYWGkpqai1+vJycnBz88PgNWrV2O325mcnMTT09PlPIQQQvx9EqwKIYRYlM1mIzc3l7y8PGA+a6fR\naJiYmGDFihXqfYqi4O7url7b7XaHdux2O1arFcCh3kKKojhlTRf6sZ3353cY5+bm1LEmJiaqmcqp\nqSm1zx9evnzJ1atXAdDr9RQVFTmUazT/+fdos9nU8Swcs81mo7S0lISEBADGx8dZtWoVVVVVDu9u\nupqLVqt1WJu+vj62bNmiXlutVsLDw9XtyN+/f2dqaorh4WGHej8+FHCloKCA1tZW6uvrycnJ+eUz\nBEhNTeXJkycMDw87rYWreSxcE1dzW8hsNmM2m1EUxWFdwfn3Q6PROPxsaGiItWvXqve+ePGCmpoa\ncnNzSU1NZXJyEoCAgACam5t58+YNr1694ubNmzQ3N1NbW0tnZyetra3k5+dz+fJlIiMj1b5+t35C\nCCH+PvkrLYQQYlFRUVE8fvyY6elpLBYLJ0+e5NmzZ073/Rx87Nq1iwcPHgAwODjIhw8fCAsLc7pv\nIX9/f0wmk3rt5ubmFGwCeHl5MTg4yOzsLJOTk3R0dKAoCjt37uT58+eMj49jt9spLy/n9u3bDnX3\n7t1LY2MjjY2NLoMzo9HIt2/fsNlsNDY2Ehsb6zTmqKgoGhoasFgsmM1msrOz6erqYs+ePTQ1NTE7\nO4vZbOb169dO7UdERNDU1ATA27dvOX/+vMNcQ0ND+fjxI/39/QDU1tZSVVVFUFAQY2NjfP78GUDd\nvutKSEgIFy5c4Nq1a4yMjPz2GSYmJtLW1sbY2Bg7duxwastut6t9mUwmOjs7iYyMdFgTjUaDxWJh\n06ZNjI+Pq1t86+rquHfv3i/HuVBkZCQtLS1YLBZmZmbIz89ndHRULW9rayMpKQmDwYC3tzft7e1Y\nLBbu3r1LTU0NiYmJlJWVMT4+zsTEBElJSQQGBlJcXEx0dLR6qJLZbMZut6tZXiGEEEuTZFaFEEIs\nKj4+np6eHjIyMrBarcTExJCSksLQ0JBDhvPnbOe5c+coKyvj4cOHKIrCxYsXWbdu3W9Pdo2Pj8do\nNBITEwPMZz5TUlLUNn4IDAwkNjaWQ4cOodPp1Pdcg4ODOXHiBLm5udhsNkJCQhb9ypif+fr6cvr0\naUZHR4mOjiY9PR2TyeTQf1ZWFv39/RgMBiwWC2lpaWrW7tOnTyQnJ+Pl5cXmzZvVtflRv6ysjLNn\nz3Lnzh1WrlypHiS1cK6VlZWcOnUKq9XKhg0bqKqqQqPRUF1dzZkzZ3Bzc2P79u2/XcuAgACys7Op\nqKigpqbG5TOE+YxxWFgYQUFBLttRFIXp6WmOHj3K3NwcFRUVeHh4OPQdFxdHQUEBN27coKqqitLS\nUubm5ggICODSpUs8ffp00XVPSEigu7sbg8GA3W4nLy+PjRs3qmPIyMigpKSElpYWfHx80Ov1mEwm\njh8/TklJCcnJyWi1WoqKivDy8iIzM5O0tDTc3d3R6XQYDAZgfht2fHz8ouMRQgjxdyn23328LYQQ\nQvxhs7OzHDt2jIaGBqdto3+C0Wikrq6O69ev//G+/xaz2UxWVha3bt3C29vbqdzVCcH/ZsXFxRQV\nFS16mrIQQoi/S7YBCyGEWFKWL19OYWGhegLvn7YwA/r/oKurC71eT2ZmpstA9X9Nd3c3Op1OAlUh\nhPgXkMyqEEIIIYQQQoglRzKrQgghhBBCCCGWHAlWhRBCCCGEEEIsORKsCiGEEEIIIYRYciRYFUII\nIYQQQgix5EiwKoQQQgghhBBiyZFgVQghhBBCCCHEkvMP3DMz2PixHlAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115e00b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "theano_accuracy_report(exp02.network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:data/beatles/models/chords_keys_synth_theano_02.model: saved model\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:09:57 theanets.graph:470 data/beatles/models/chords_keys_synth_theano_02.model: saved model\n"
     ]
    }
   ],
   "source": [
    "exp02.save('data/beatles/models/chords_keys_synth_theano_02.model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.feedforward:layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 theanets.layers.feedforward:41 layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 theanets.layers.base:316 layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"out\": (hid1:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 theanets.layers.base:316 layer Feedforward \"out\": (hid1:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:network has 20512 total parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 theanets.graph:114 network has 20512 total parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.dataset:144 valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.dataset:144 train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- patience = 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:378 -- patience = 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- validate_every = 10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:379 -- validate_every = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- min_improvement = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:380 -- min_improvement = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:381 -- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:382 -- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:383 -- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- momentum = 0.9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:384 -- momentum = 0.9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- nesterov = False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:385 -- nesterov = False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.adaptive:220 -- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.adaptive:221 -- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling evaluation function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:112 compiling evaluation function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling RMSProp function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:13 downhill.base:118 compiling RMSProp function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 0 loss=3.129999 err=3.129999 acc=0.083708 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:15 downhill.base:226 validation 0 loss=3.129999 err=3.129999 acc=0.083708 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 1 loss=0.982786 err=0.982786 acc=0.726408\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:18 downhill.base:226 RMSProp 1 loss=0.982786 err=0.982786 acc=0.726408\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 2 loss=0.828139 err=0.828139 acc=0.751166\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:22 downhill.base:226 RMSProp 2 loss=0.828139 err=0.828139 acc=0.751166\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 3 loss=0.781537 err=0.781537 acc=0.760612\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:25 downhill.base:226 RMSProp 3 loss=0.781537 err=0.781537 acc=0.760612\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 4 loss=0.751381 err=0.751381 acc=0.767546\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:28 downhill.base:226 RMSProp 4 loss=0.751381 err=0.751381 acc=0.767546\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 5 loss=0.728414 err=0.728414 acc=0.773684\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:31 downhill.base:226 RMSProp 5 loss=0.728414 err=0.728414 acc=0.773684\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 6 loss=0.709894 err=0.709894 acc=0.778801\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:34 downhill.base:226 RMSProp 6 loss=0.709894 err=0.709894 acc=0.778801\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 7 loss=0.693021 err=0.693021 acc=0.783773\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:37 downhill.base:226 RMSProp 7 loss=0.693021 err=0.693021 acc=0.783773\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 8 loss=0.678755 err=0.678755 acc=0.787331\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:39 downhill.base:226 RMSProp 8 loss=0.678755 err=0.678755 acc=0.787331\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 9 loss=0.664763 err=0.664763 acc=0.792555\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:42 downhill.base:226 RMSProp 9 loss=0.664763 err=0.664763 acc=0.792555\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 10 loss=0.652719 err=0.652719 acc=0.796211\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:45 downhill.base:226 RMSProp 10 loss=0.652719 err=0.652719 acc=0.796211\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 1 loss=0.702708 err=0.702708 acc=0.782029 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:45 downhill.base:226 validation 1 loss=0.702708 err=0.702708 acc=0.782029 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 11 loss=0.640963 err=0.640963 acc=0.800115\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:48 downhill.base:226 RMSProp 11 loss=0.640963 err=0.640963 acc=0.800115\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 12 loss=0.629784 err=0.629784 acc=0.803472\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:51 downhill.base:226 RMSProp 12 loss=0.629784 err=0.629784 acc=0.803472\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 13 loss=0.619569 err=0.619569 acc=0.806435\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:54 downhill.base:226 RMSProp 13 loss=0.619569 err=0.619569 acc=0.806435\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 14 loss=0.609920 err=0.609920 acc=0.810917\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:33:56 downhill.base:226 RMSProp 14 loss=0.609920 err=0.609920 acc=0.810917\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 15 loss=0.600936 err=0.600936 acc=0.813503\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:00 downhill.base:226 RMSProp 15 loss=0.600936 err=0.600936 acc=0.813503\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 16 loss=0.592219 err=0.592219 acc=0.815764\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:02 downhill.base:226 RMSProp 16 loss=0.592219 err=0.592219 acc=0.815764\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 17 loss=0.584867 err=0.584867 acc=0.818457\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:05 downhill.base:226 RMSProp 17 loss=0.584867 err=0.584867 acc=0.818457\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 18 loss=0.577438 err=0.577438 acc=0.821102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:09 downhill.base:226 RMSProp 18 loss=0.577438 err=0.577438 acc=0.821102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 19 loss=0.571053 err=0.571053 acc=0.821832\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:13 downhill.base:226 RMSProp 19 loss=0.571053 err=0.571053 acc=0.821832\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 20 loss=0.564836 err=0.564836 acc=0.824854\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:16 downhill.base:226 RMSProp 20 loss=0.564836 err=0.564836 acc=0.824854\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 2 loss=0.694497 err=0.694497 acc=0.787540 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:16 downhill.base:226 validation 2 loss=0.694497 err=0.694497 acc=0.787540 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 21 loss=0.559111 err=0.559111 acc=0.826941\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:19 downhill.base:226 RMSProp 21 loss=0.559111 err=0.559111 acc=0.826941\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 22 loss=0.553686 err=0.553686 acc=0.828298\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:22 downhill.base:226 RMSProp 22 loss=0.553686 err=0.553686 acc=0.828298\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 23 loss=0.548367 err=0.548367 acc=0.829663\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:26 downhill.base:226 RMSProp 23 loss=0.548367 err=0.548367 acc=0.829663\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 24 loss=0.544032 err=0.544032 acc=0.831269\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:28 downhill.base:226 RMSProp 24 loss=0.544032 err=0.544032 acc=0.831269\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 25 loss=0.539354 err=0.539354 acc=0.832067\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:31 downhill.base:226 RMSProp 25 loss=0.539354 err=0.539354 acc=0.832067\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 26 loss=0.535004 err=0.535004 acc=0.833895\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:33 downhill.base:226 RMSProp 26 loss=0.535004 err=0.535004 acc=0.833895\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 27 loss=0.530864 err=0.530864 acc=0.835395\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:37 downhill.base:226 RMSProp 27 loss=0.530864 err=0.530864 acc=0.835395\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 28 loss=0.526694 err=0.526694 acc=0.836319\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:39 downhill.base:226 RMSProp 28 loss=0.526694 err=0.526694 acc=0.836319\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 29 loss=0.523344 err=0.523344 acc=0.837127\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:42 downhill.base:226 RMSProp 29 loss=0.523344 err=0.523344 acc=0.837127\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 30 loss=0.520100 err=0.520100 acc=0.838714\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:45 downhill.base:226 RMSProp 30 loss=0.520100 err=0.520100 acc=0.838714\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 3 loss=0.700443 err=0.700443 acc=0.790166\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:45 downhill.base:226 validation 3 loss=0.700443 err=0.700443 acc=0.790166\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 31 loss=0.516322 err=0.516322 acc=0.840224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:47 downhill.base:226 RMSProp 31 loss=0.516322 err=0.516322 acc=0.840224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 32 loss=0.514068 err=0.514068 acc=0.840762\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:50 downhill.base:226 RMSProp 32 loss=0.514068 err=0.514068 acc=0.840762\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 33 loss=0.510770 err=0.510770 acc=0.841830\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:53 downhill.base:226 RMSProp 33 loss=0.510770 err=0.510770 acc=0.841830\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 34 loss=0.507829 err=0.507829 acc=0.842830\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:56 downhill.base:226 RMSProp 34 loss=0.507829 err=0.507829 acc=0.842830\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 35 loss=0.505716 err=0.505716 acc=0.843234\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:34:59 downhill.base:226 RMSProp 35 loss=0.505716 err=0.505716 acc=0.843234\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 36 loss=0.503011 err=0.503011 acc=0.844696\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:01 downhill.base:226 RMSProp 36 loss=0.503011 err=0.503011 acc=0.844696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 37 loss=0.499832 err=0.499832 acc=0.845033\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:05 downhill.base:226 RMSProp 37 loss=0.499832 err=0.499832 acc=0.845033\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 38 loss=0.497688 err=0.497688 acc=0.845331\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:08 downhill.base:226 RMSProp 38 loss=0.497688 err=0.497688 acc=0.845331\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 39 loss=0.495581 err=0.495581 acc=0.846485\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:11 downhill.base:226 RMSProp 39 loss=0.495581 err=0.495581 acc=0.846485\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 40 loss=0.493524 err=0.493524 acc=0.848024\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:14 downhill.base:226 RMSProp 40 loss=0.493524 err=0.493524 acc=0.848024\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 4 loss=0.732142 err=0.732142 acc=0.786328\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:14 downhill.base:226 validation 4 loss=0.732142 err=0.732142 acc=0.786328\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 41 loss=0.491602 err=0.491602 acc=0.847447\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:17 downhill.base:226 RMSProp 41 loss=0.491602 err=0.491602 acc=0.847447\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 42 loss=0.488982 err=0.488982 acc=0.849438\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:19 downhill.base:226 RMSProp 42 loss=0.488982 err=0.488982 acc=0.849438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 43 loss=0.487208 err=0.487208 acc=0.849544\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:22 downhill.base:226 RMSProp 43 loss=0.487208 err=0.487208 acc=0.849544\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 44 loss=0.484786 err=0.484786 acc=0.850968\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:25 downhill.base:226 RMSProp 44 loss=0.484786 err=0.484786 acc=0.850968\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 45 loss=0.483815 err=0.483815 acc=0.851440\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:29 downhill.base:226 RMSProp 45 loss=0.483815 err=0.483815 acc=0.851440\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 46 loss=0.481623 err=0.481623 acc=0.851795\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:32 downhill.base:226 RMSProp 46 loss=0.481623 err=0.481623 acc=0.851795\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 47 loss=0.480152 err=0.480152 acc=0.851852\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:36 downhill.base:226 RMSProp 47 loss=0.480152 err=0.480152 acc=0.851852\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 48 loss=0.478543 err=0.478543 acc=0.852718\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:39 downhill.base:226 RMSProp 48 loss=0.478543 err=0.478543 acc=0.852718\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 49 loss=0.477042 err=0.477042 acc=0.853372\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:42 downhill.base:226 RMSProp 49 loss=0.477042 err=0.477042 acc=0.853372\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 50 loss=0.474917 err=0.474917 acc=0.853853\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:45 downhill.base:226 RMSProp 50 loss=0.474917 err=0.474917 acc=0.853853\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 5 loss=0.752278 err=0.752278 acc=0.784164\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:45 downhill.base:226 validation 5 loss=0.752278 err=0.752278 acc=0.784164\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 51 loss=0.474338 err=0.474338 acc=0.853709\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:47 downhill.base:226 RMSProp 51 loss=0.474338 err=0.474338 acc=0.853709\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 52 loss=0.472411 err=0.472411 acc=0.854680\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:50 downhill.base:226 RMSProp 52 loss=0.472411 err=0.472411 acc=0.854680\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 53 loss=0.471243 err=0.471243 acc=0.854622\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:52 downhill.base:226 RMSProp 53 loss=0.471243 err=0.471243 acc=0.854622\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 54 loss=0.469291 err=0.469291 acc=0.855691\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:55 downhill.base:226 RMSProp 54 loss=0.469291 err=0.469291 acc=0.855691\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 55 loss=0.468112 err=0.468112 acc=0.856952\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:35:57 downhill.base:226 RMSProp 55 loss=0.468112 err=0.468112 acc=0.856952\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 56 loss=0.466635 err=0.466635 acc=0.856613\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:00 downhill.base:226 RMSProp 56 loss=0.466635 err=0.466635 acc=0.856613\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 57 loss=0.465100 err=0.465100 acc=0.856220\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:03 downhill.base:226 RMSProp 57 loss=0.465100 err=0.465100 acc=0.856220\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 58 loss=0.464344 err=0.464344 acc=0.857211\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:05 downhill.base:226 RMSProp 58 loss=0.464344 err=0.464344 acc=0.857211\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 59 loss=0.462572 err=0.462572 acc=0.858674\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:08 downhill.base:226 RMSProp 59 loss=0.462572 err=0.462572 acc=0.858674\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 60 loss=0.461327 err=0.461327 acc=0.858347\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:10 downhill.base:226 RMSProp 60 loss=0.461327 err=0.461327 acc=0.858347\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 6 loss=0.773662 err=0.773662 acc=0.783530\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:11 downhill.base:226 validation 6 loss=0.773662 err=0.773662 acc=0.783530\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 61 loss=0.460689 err=0.460689 acc=0.858606\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:13 downhill.base:226 RMSProp 61 loss=0.460689 err=0.460689 acc=0.858606\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 62 loss=0.458956 err=0.458956 acc=0.859423\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:16 downhill.base:226 RMSProp 62 loss=0.458956 err=0.458956 acc=0.859423\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 63 loss=0.458056 err=0.458056 acc=0.859771\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:19 downhill.base:226 RMSProp 63 loss=0.458056 err=0.458056 acc=0.859771\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 64 loss=0.456913 err=0.456913 acc=0.859461\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:23 downhill.base:226 RMSProp 64 loss=0.456913 err=0.456913 acc=0.859461\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 65 loss=0.455929 err=0.455929 acc=0.860299\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:26 downhill.base:226 RMSProp 65 loss=0.455929 err=0.455929 acc=0.860299\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 66 loss=0.454681 err=0.454681 acc=0.860492\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:30 downhill.base:226 RMSProp 66 loss=0.454681 err=0.454681 acc=0.860492\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 67 loss=0.453741 err=0.453741 acc=0.861214\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:33 downhill.base:226 RMSProp 67 loss=0.453741 err=0.453741 acc=0.861214\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 68 loss=0.452925 err=0.452925 acc=0.862030\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:35 downhill.base:226 RMSProp 68 loss=0.452925 err=0.452925 acc=0.862030\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 69 loss=0.452019 err=0.452019 acc=0.862231\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:38 downhill.base:226 RMSProp 69 loss=0.452019 err=0.452019 acc=0.862231\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 70 loss=0.450340 err=0.450340 acc=0.862059\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:40 downhill.base:226 RMSProp 70 loss=0.450340 err=0.450340 acc=0.862059\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 7 loss=0.795075 err=0.795075 acc=0.780904\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:41 downhill.base:226 validation 7 loss=0.795075 err=0.795075 acc=0.780904\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 71 loss=0.450045 err=0.450045 acc=0.862358\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:43 downhill.base:226 RMSProp 71 loss=0.450045 err=0.450045 acc=0.862358\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 72 loss=0.449250 err=0.449250 acc=0.862243\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:46 downhill.base:226 RMSProp 72 loss=0.449250 err=0.449250 acc=0.862243\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 73 loss=0.448008 err=0.448008 acc=0.862733\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:48 downhill.base:226 RMSProp 73 loss=0.448008 err=0.448008 acc=0.862733\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 74 loss=0.446889 err=0.446889 acc=0.863579\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:52 downhill.base:226 RMSProp 74 loss=0.446889 err=0.446889 acc=0.863579\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 75 loss=0.446306 err=0.446306 acc=0.863792\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:55 downhill.base:226 RMSProp 75 loss=0.446306 err=0.446306 acc=0.863792\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 76 loss=0.445537 err=0.445537 acc=0.863627\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:36:57 downhill.base:226 RMSProp 76 loss=0.445537 err=0.445537 acc=0.863627\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 77 loss=0.444171 err=0.444171 acc=0.864752\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:01 downhill.base:226 RMSProp 77 loss=0.444171 err=0.444171 acc=0.864752\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 78 loss=0.443748 err=0.443748 acc=0.864945\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:04 downhill.base:226 RMSProp 78 loss=0.443748 err=0.443748 acc=0.864945\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 79 loss=0.442534 err=0.442534 acc=0.864637\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:07 downhill.base:226 RMSProp 79 loss=0.442534 err=0.442534 acc=0.864637\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 80 loss=0.441827 err=0.441827 acc=0.864809\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:10 downhill.base:226 RMSProp 80 loss=0.441827 err=0.441827 acc=0.864809\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 8 loss=0.816560 err=0.816560 acc=0.780211\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:10 downhill.base:226 validation 8 loss=0.816560 err=0.816560 acc=0.780211\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:patience elapsed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:37:10 downhill.base:402 patience elapsed!\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2U1nWd//HXNdyKSOY25VEqjMMxu/OIDgsnpQ4/qqnt\njtQCgWyjTbe1WjDbIcXxdkXplJtRYlbHJhOXzkjadrOBnMWkFXZTSsludsXKQEC3GhBlYK7fH54m\nTZEv6sV8Bh6Pv5hrmO/3fb0B8cn3e11Tq9fr9QAAAEBBmvp6AAAAAPhLYhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4gxs1IF7enpy7rnnZv369WlqasrFF1+cAQMGpK2tLU1NTRkzZkza29tT\nq9UaNQIAAAD9VMNi9Yc//GG2b9+eG264IatWrcpnP/vZ7Ny5M3PmzElLS0va29uzfPnyTJ48uVEj\nAAAA0E817DbgoUOHpqurK/V6PV1dXRk0aFDuueeetLS0JEkmTpyYVatWNer0AAAA9GMNu7I6duzY\n7NixI62trfn973+fq6++OmvWrOn9/LBhw9LV1dWo0wMAANCPNSxWr7322owdOzazZ8/Oxo0b8/73\nvz87d+7s/fy2bdsyYsSIPR6nXq97XSsAAMABpmGxun379hx88MFJkhEjRmTnzp151ateldWrV2fc\nuHFZuXJlJkyYsMfj1Gq1bN7sCuyeNDcfYk8V2VU19lSNPVVnV9XYU3V2VY09VWNP1dlVNfZUXXPz\nIU/7eMNiddasWZk7d25OO+207Ny5M2effXZe/epXZ968eenu7s7o0aPT2traqNMDAADQjzUsVkeM\nGJGFCxc+5fGOjo5GnRIAAID9RMPeDfj5Uq/X+3oEAAAA9rHiY7WpqZ6TTx4kWgEAAA4gxcdq0pTb\nbhuSY489KHfeKVgBAAAOBP0gVpOklo0bB+X00w9yhRUAAOAA0E9iFQAAgANJw94N+PlVz+GH78x1\n121PrVbr62EAAABosH4Rq4cfvjNr1wpVAACAA0XxtwEPGFB3RRUAAOAAU3ysjhyZHHecUAUAADiQ\nFB+riVAFAAA40PSDWAUAAOBAI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKM7ARh78pptuSmdnZ5Lksccey7333ptvfOMbufTSS9PU\n1JQxY8akvb09tVqtkWMAAADQzzT0yuqUKVPS0dGRjo6OvOY1r8m8efOycOHCzJkzJ9dff33q9XqW\nL1/eyBEAAADoh/bJbcA//elP86tf/Sqnnnpq7rnnnrS0tCRJJk6cmFWrVu2LEQAAAOhH9kmsLlq0\nKGeddVaSpF6v9z4+bNiwdHV17YsRAAAA6Eca+prVJPnjH/+Y9evXZ9y4cUmSpqY/9/G2bdsyYsSI\nPR6jufmQhs23P7Gn6uyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7em4aHqtr1qzJ+PHjez8+5phj\nsnr16owbNy4rV67MhAkT9niMzZtdfd2T5uZD7Kkiu6rGnqqxp+rsqhp7qs6uqrGnauypOruqxp6q\n213UNzxW169fn5e97GW9H7e1tWXevHnp7u7O6NGj09ra2ugRAAAA6GcaHquzZs160sejRo1KR0dH\no08LAABAP7ZP3mAJAAAA9oZYBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIozsJEHX7RoUVasWJHu7u7MmDEjY8eOTVtbW5qa\nmjJmzJi0t7enVqs1cgQAAAD6oYZdWb3jjjty5513ZvHixeno6MhvfvObzJ8/P3PmzMn111+fer2e\n5cuXN+r0AAAA9GMNi9Xbb789Rx99dD7ykY/kzDPPzKRJk3LPPfekpaUlSTJx4sSsWrWqUacHAACg\nH2vYbcAPP/xwNmzYkEWLFuU3v/lNzjzzzNTr9d7PDxs2LF1dXY06PQAAAP1Yw2L1hS98YUaPHp2B\nAwfmqKOOypAhQ7Jp06bez2/bti0jRoyodKzm5kMaNeZ+xZ6qs6tq7Kkae6rOrqqxp+rsqhp7qsae\nqrOrauzpuWlYrB5//PH52te+lr/927/Ngw8+mEcffTTjx4/P6tWrM27cuKxcuTITJkyodKzNm12B\n3ZPm5kPsqSK7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnqrbXdQ3LFbf+MY3Zs2aNTnllFPS09OT\n9vb2HHnkkZk3b166u7szevTotLa2Nur0AAAA9GMN/dY155xzzlMe6+joaOQpAQAA2A807N2AAQAA\n4NkSqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEes\nAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFCcgY0+wZQpUzJ8+PAkyUtf+tKcccYZaWtrS1NTU8aMGZP29vbUarVGjwEAAEA/0tBY\nfeyxx5IkHR0dvY+deeaZmTNnTlpaWtLe3p7ly5dn8uTJjRwDAACAfqahtwHfe++92b59e2bNmpXT\nTz89d911V9atW5eWlpYkycSJE7Nq1apGjgAAAEA/1NArqwcddFBmzZqVU089NevXr8+HPvShJ31+\n2LBh6erqauQIAAAA9EMNjdVRo0bl5S9/ee+PDz300PzsZz/r/fy2bdsyYsSIPR6nufmQhs24P7Gn\n6uyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7em4aGqudnZ35+c9/nvb29jz44IPZtm1bXv/612f1\n6tUZN25cVq5cmQkTJuzxOJs3u/q6J83Nh9hTRXZVjT1VY0/V2VU19lSdXVVjT9XYU3V2VY09Vbe7\nqG9orJ5yyimZO3dupk+fniS57LLLcuihh2bevHnp7u7O6NGj09ra2sgRAAAA6IcaGqsDBw7MggUL\nnvL4E98dGAAAAP5SQ98NGAAAAJ4NsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoA\nAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUZ69i9dFHH83WrVv36gQPPfRQ3vCGN+S+\n++7L/fffn2nTpmX69Om54IILUq/X9+pYAAAAHBgqx+qSJUvy3ve+N1OnTs2VV15Z6Wu6u7tz/vnn\n56CDDkq9Xs9ll12WOXPm5Prrr0+9Xs/y5cuf9eAAAADsv3Ybq7/4xS+e9PHy5ctz880359vf/naW\nLVtW6eBXXHFFpk2blubm5iTJunXr0tLSkiSZOHFiVq1a9WznBgAAYD82cHefuPHGG9Pd3Z1/+Id/\nyEte8pK88pWvzKxZszJw4MCMGTNmjwfu7OzMYYcdlhNPPDGLFi1KvV5/0m2/w4YNS1dX1/PzLAAA\nANiv1OrP8MLR++67LwsXLswRRxyRv/u7v8vmzZuzY8eOvPKVr9zjgWfMmJFarZYkuffeezNq1Kj8\n7Gc/y913350kWbZsWX70ox9l3rx5z3icUaOS9eurPyEAAAD6v2eM1T/58Y9/nGuuuSbjx4/P9OnT\nM2jQoL06ycyZM3PhhRfmiiuuyAc/+MGMGzcu559/fiZMmJC3vvWtz/i1o0Yla9a4Arsnzc2HZPNm\ne6rCrqqxp2rsqTq7qsaeqrOrauypGnuqzq6qsafqmpsPedrHd/ua1euvvz6TJ0/OW97ylmzatClX\nX311jjjiiJx55pm5+eab93qAWq2Wtra2XHXVVZk6dWp27dqV1tbWvT4OAAAA+7/dvmb1hhtuyPe+\n973s2LEj06dPT2tra9785jdn0qRJ+dd//de9OklHR8fT/hgAAACezm5jtbm5Of/8z/+cRx99NK94\nxSv+/AUDB+a0007bJ8MBAABwYNptrF599dW57bbbMnjw4Lz+9a/flzMBAABwgNttrA4ZMiSTJ0/e\nl7MAAABAkmd4gyUAAADoK2IVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi\nFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOAMbefBdu3blvPPOy/r161Or1XLhhRdm8ODBaWtrS1NTU8aMGZP29vbUarVGjgEA\nAEA/09BYXbFiRZqamnLDDTdk9erV+cxnPpMkmTNnTlpaWtLe3p7ly5dn8uTJjRwDAACAfqahtwFP\nnjw5F110UZLkgQceyAte8ILcc889aWlpSZJMnDgxq1atauQIAAAA9EMNf83qgAED0tbWlksvvTTv\neMc7Uq/Xez83bNiwdHV1NXoEAAAA+pla/Yn12EBbtmzJqaeemkceeSR33HFHkmTZsmX50Y9+lHnz\n5u3260aNStav3xcTAgAAUIqGvmZ16dKlefDBB3PGGWdk6NChaWpqymte85qsXr0648aNy8qVKzNh\nwoQ9HmfzZldf96S5+RB7qsiuqrGnauypOruqxp6qs6tq7Kkae6rOrqqxp+qamw952scbGqutra1p\na2vLjBkzsnPnzpx77rl5xStekXnz5qW7uzujR49Oa2trI0cAAACgH2porA4dOjRXXnnlUx7v6Oho\n5GkBAADo5xr+BksAAACwt8QqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFGdgow7c3d2dT33qU/nd736XHTt25O///u8zevTotLW1pamp\nKWPGjEl7e3tqtVqjRgAAAKCfalis3nLLLTnssMOyYMGC/OEPf8i73vWuHHPMMZkzZ05aWlrS3t6e\n5cuXZ/LkyY0aAQAAgH6qYbcBt7a25mMf+1iSpKenJwMHDsy6devS0tKSJJk4cWJWrVrVqNMDAADQ\njzUsVocNG5aDDz44W7duzcc//vH84z/+Y3p6ep70+a6urkadHgAAgH6sYbcBJ8mGDRty1llnZfr0\n6Xn729+eBQsW9H5u27ZtGTFiRKXjNDcf0qgR9yv2VJ1dVWNP1dhTdXZVjT1VZ1fV2FM19lSdXVVj\nT89Nw2J1y5Yt+eAHP5j29vaMHz8+SXLMMcdk9erVGTduXFauXJkJEyZUOtbmza7A7klz8yH2VJFd\nVWNP1dhTdXZVjT1VZ1fV2FM19lSdXVVjT9XtLuobFqtXX311urq6snDhwixcuDBJcu655+bSSy9N\nd3d3Ro8endbW1kadHgAAgH6sYbF63nnn5bzzznvK4x0dHY06JQAAAPuJhr3BEgAAADxbYhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIoj\nVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5Y\nBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACK\nI1YBAAAoTsNjde3atZk5c2aS5P7778+0adMyffr0XHDBBanX640+PQAAAP1QQ2P1S1/6Us4777x0\nd3cnSS677LLMmTMn119/fer1epYvX97I0wMAANBPNTRWX/7yl+fzn/987xXUdevWpaWlJUkyceLE\nrFq1qpGnBwAAoJ9qaKy++c1vzoABA3o/fuJtv8OGDUtXV1cjTw8AAEA/NXBfnqyp6c9tvG3btowY\nMaLS1zU3H9KokfYr9lSdXVVjT9XYU3V2VY09VWdX1dhTNfZUnV1VY0/PzT6N1WOOOSarV6/OuHHj\nsnLlykyYMKHS123e7ArsnjQ3H2JPFdlVNfZUjT1VZ1fV2FN1dlWNPVVjT9XZVTX2VN3uon6fxGqt\nVkuStLW1Zd68eenu7s7o0aPT2tq6L04PAABAP9PwWB05cmQWL16cJBk1alQ6OjoafUoAAAD6uYZ/\nn1UAAADYW2IVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKI\nVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ojV/UC9Xk+9\nXu/rMQAAAJ43YrUfq9frOfnkQXnJSw5OU9PjPxatAADA/kCs9mOnnDI4t902JMmAJE257bYhOfbY\ng3LnnYIVAADo38RqP1Wv13PbbYOS1J7waC0bNw7K6acf5ArrbrhlGgAA+gexygHBLdMAANC/iNV+\nqlar5aSTupM8MbjqOfzw7lx33fbUarXdfekByS3TAADQv4jVfuyb39yRww/f2fvx4YfvzNq123Pc\ncUL1idwyDQAA/Y9Y7cdqtVquu257mprqGTCg7ooqAACw3xCr/dxxx9Vy5JH1jBwZV1R3wy3TAADQ\n/4jV/YbgeiZumQYAgP5FrHJAcMs0AAD0LwP7egDYV/50y3RTU80VVQAAKJwrqxyAhCoAAJROrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrwFPU6/XU6/W+HgMAgAOYWAV61ev1nHzyoLzkJQenqenxH4tWAAD6glgFep1yyuDcdtuQJAOS\nNOW224bk2GMPyp13ClYAAPYtsQokefyq6m23DUpSe8KjtWzcOCinn36QK6xPw+3SAACNI1YB9pLb\npQEAGk+sAkmSWq2Wk07qTvLE6Krn8MO7c91121Or1Xb3pQcct0sDADSeWAV6ffObO3L44Tt7Pz78\n8J1Zu3Z7jjtOqP6J26WfHbdMV2NPAPBnYhXoVavVct1129PUVM+AAXVXVHnO3DJdjT0BwFOJVeBJ\njjuuliOPrGfkyLii+jTcLr133DJdjT3tPVehAfZ/YhXYDdG1O26XrsYt09XY095xFRrgwCFWAfaS\n26Wh77gKDXDgEKsAz4LbpffMLdPV2FN1rkIDHFjEKsBzIiSeiVumq7EnGsVre4H+TKwC0DBuma7G\nnqpxFbo6r+0F9gdiFYCGcst0NfZUjavQ1XhtL7A/EKsA7CNiohp7eiauQu+Z1/YC+4uBfT0AAMDe\n+NNV6KammiuqPC+8trc6u2JfcmUVAOinhOrT8dre6ry2tzq72jui/vmxz2O1p6cn559/fqZOnZqZ\nM2fm17/+9b4eAQBgv+a1vdV4bW91dlWNqH9+7fNYXbZsWbq7u7N48eJ84hOfyPz58/f1CAAA+zWv\n7d0zr+2tzq6qE/XPr30eqz/+8Y9z0kknJUmOPfbY3H333ft6BACA/Z53mIZ9S9Q///b5Gyxt3bo1\nw4cP7/14wIAB6enpSVPT03fzD387Kocd7xf2mdz+u1pqiT1VYFfV2FM19lSdXVVjT9XZVTX29Mx+\nNyR57LGigHgUAAAT5UlEQVQnh/yApuRF9XoGn9BHQxXKrqq57wmhuiSn5pNZ0IfT9H/7PFaHDx+e\nbdu29X78TKGaJCN3rt8HU/VvL+3rAfoRu6rGnqqxp+rsqhp7qs6uqrGnZ3Z4vZ6RI3vyu98NSJIc\nccSu/Pa3TW6Zfhp2Vc2s/7czt946IH++ulrPEUf0ZOnSprz4xSP6crR+aZ/H6tixY7NixYq89a1v\nzV133ZWjjz56X48AAACp1Wp54IEBT3hkwG5/7oHOrqpZvvwv86oWu3r2avV9fPN0vV7PBRdckJ//\n/OdJkssuuyxHHXXUvhwBAACAwu3zWAUAAIA92efvBgwAAAB7IlYBAAAojlgFAACgOEXGak9PT84/\n//xMnTo1M2fOzK9//eu+HqlYN910U2bOnJmZM2fmve99b173utdl69atfT1WkaZMmdK7q0996lN9\nPU6xdu3alblz52batGk57bTT8stf/rKvRyra2rVrM3PmzL4eo0hP3M3999+fadOmZfr06bngggt8\nY/Sn0d3dnXPOOSfTp0/PqaeemltvvbWvRyrSjh078slPfjJTp07NjBkzcu+99/b1SEVbtGhRpk6d\nmpNPPjk33XRTX49TtIceeihveMMbct999/X1KEXq6enp/f+D6dOn53//93/7eqQi/eX/F/zgBz/I\n2Wef3YcT9W/7/FvXVLFs2bJ0d3dn8eLFWbt2bebPn58vfOELfT1WkaZMmZIpU6YkSS666KKceuqp\nGT58eB9PVZ7HHnssSdLR0dHHk5RvxYoVaWpqyg033JDVq1fns5/9rD9/u/GlL30pN998cw4++OC+\nHqU4f7mbyy67LHPmzElLS0va29uzfPnyTJ48uY+nLMstt9ySww47LAsWLMgf/vCHvPvd786kSZP6\neqziLFmyJEOHDs3ixYtz33335eyzz05nZ2dfj1WkO+64I3feeWcWL16cRx55JNdee21fj1Ss7u7u\nnH/++TnooIP6epRi/fCHP8z27dtzww03ZNWqVbnyyivzuc99rq/HKspf/t13ySWX5Pbbb8+rXvWq\nPp6s/yryyuqPf/zjnHTSSUmSY489NnfffXcfT1S+n/70p/nlL3+ZU089ta9HKdK9996b7du3Z9as\nWTn99NOzdu3avh6pWJMnT85FF12UJHnggQfyghe8oI8nKtfLX/7yfP7zn3eV8Gn85W7WrVuXlpaW\nJMnEiROzatWqvhyvSK2trfnYxz6W5PErGAMG+L58T+dXv/pVJk6cmCQ56qij8uCDD7qjaDduv/32\nHH300fnIRz6SM8880z9+PIMrrrgi06ZNS3Nzc1+PUqyhQ4emq6sr9Xo9XV1dGTRoUF+PVJy//Ltv\n7Nix7iZ6joqM1a1btz7p6uCAAQPS09PThxOVb9GiRfnoRz/a12MU66CDDsqsWbPy5S9/ORdeeGE+\n8YlP+D31DAYMGJC2trZccsklefvb397X4xTrzW9+s6DYjb/czRP/oh42bFi6urr6YqyiDRs2LAcf\nfHC2bt2aj3/845k9e3Zfj1SkY445JitWrEiS3HXXXXn44YfzyCOP9PFUZXr44Ydz991353Of+1zv\n3308VWdnZw477LCceOKJSSIsdmPs2LHZsWNHWltbc/7552fGjBl9PVJx/vLvvre97W19OM3+ochY\nHT58eLZt29b7cU9PT5qaihy1CH/84x+zfv36jBs3rq9HKdaoUaPyzne+s/fHhx56aDZv3tzHU5Vt\n/vz5+f73v5958+bl0Ucf7etx6Oee+N/wbdu2ZcSIEX04Tbk2bNiQ008/Pe9+97vzN3/zN309TpFO\nPvnkDB8+PKeddlqWLVvW+990nuqFL3xhTjzxxAwcODBHHXVUhgwZkocffrivxypOZ2dnVq1alZkz\nZ+bee+9NW1tbtmzZ0tdjFefaa6/N2LFj8/3vfz/f+ta30tbWlh07dvT1WOzniizAsWPHZuXKlUke\n/1fTo48+uo8nKtuaNWsyfvz4vh6jaJ2dnZk/f36S9N4y5lafp7d06dIsWrQoyeO3/NRqNf9YxHN2\nzDHHZPXq1UmSlStX5oQTTujjicqzZcuWfPCDH8w555yT97znPX09TrF+8pOfZPz48fnGN76Rt7zl\nLWlubs7gwYP7eqwiHX/88bntttuSPP533/bt2/PCF76wj6cqz9e//vV0dHSko6Mjr3zlK3P55Zfn\nRS96UV+PVZzt27f3vhZzxIgR6e7udpcaDVfkGyy96U1vyu23356pU6cmefyNOdi99evX52Uve1lf\nj1G0U045JXPnzs306dOTPP57SoA9vdbW1rS1tWXGjBnZuXNnzj33XP8juAe1Wq2vRyjWn3bT1taW\nefPmpbu7O6NHj05ra2sfT1aeq6++Ol1dXVm4cGEWLlyY5PErGUOGDOnjycpy1FFHZfbs2Vm0aFEG\nDx6cSy65pK9HKtYb3/jGrFmzJqecckp6enrS3t7uv1c8a7NmzcrcuXNz2mmnZefOnTn77LMzdOjQ\nvh6rSE/8c1ar1fy5ew5qdTfmAwAAUBiXlgAAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgF\nAACgOGIVgOIsX748HR0dSZKZM2f28TTPzUUXXZSbbropmzZtyoc//OFn/Ll7+1x/+tOfPu3XzJw5\nM6tXr96rYz0bixcvzo033pgkmTt3bjZs2LDbn9vZ2Zm5c+c+q/M8n8/nnnvuyYIFC56XYwHQWGIV\ngKLs2LEjX/rSl3LaaaclSdasWdPHEz03f/pm8C9+8YtzzTXXPOPPfT6f6774JvRTp07N+973viTJ\nHXfckZ6enobN83w9n1e/+tXZuHFjfvGLXzwvxwOgcQb29QAA9A/XXHNNvve972XXrl058cQTc845\n5+S3v/1tPvShD+Wwww7LkCFD8s53vjOdnZ35/e9/n0mTJmXmzJk599xzs2HDhgwcODCzZ8/OSSed\nlKuuuip33XVXNm7cmBkzZmTatGm957n55pszbty4DBgwIJdcckmS5H3ve19uvPHGjB8/Pq95zWuy\nZcuWfPKTn8wXv/jF3iuwbW1t+eu//utMmTIlS5cuzde+9rX09PTk1a9+ddrb2zN48OBKz/OOO+7I\nF7/4xSTJxo0b87rXvS6XXHJJNm3a1Ptchw4dmmuvvTaXX3551qxZk127dmXKlCn5wAc+kHq9niuu\nuCK33nprXvSiF2XQoEF57Wtfm9/+9rd5//vfn1tvvTUPPPBA5s6dm//7v//L0KFDc8kll2TJkiVP\neq4rV67MVVddlZ07d2bkyJG5+OKLc+ihh+b222/P/PnzM2jQoIwZM+YZn8tDDz2UD3zgA5k9e3Ym\nTZr0tL+GV155Zer1embPnp3k8SukJ510Ut72trf1HqetrS2DBg3KunXrsnXr1nzkIx/Ju971rlx1\n1VWp1WoZPHhwNm3alDPOOCNf//rXs27dulx++eXp6enJkUcemU9/+tOp1+u5//77M3PmzGzYsCET\nJkzIxRdf/JSZFyxYkGXLlmXgwIF53/vel/e///29n9u1a1fa29vzq1/9Klu2bMlRRx2Vz3/+8+nu\n7s6cOXOyZcuWJMlZZ52VSZMm5atf/WqWLl2apqamvPa1r81FF12UJHnHO96Rr3zlK5k/f36l3xMA\n9A1XVgHYo5UrV+aee+7JN7/5zdx0003ZuHFjbr755iTJ+vXr8+lPfzpf/epXU6/Xs2nTpnzrW9/K\n7Nmzc/HFF2fChAm5+eab8y//8i/51Kc+lYceeihJ0t3dnX/7t397UqgmyYoVK3LCCSckSc4777wk\n6b3V9Pe//33OOOOMLF26NAMGDHjS19VqtdRqtfzyl7/MkiVLsnjx4ixdujSHHXZYvvzlL+/V8127\ndm0uuuiifPe7381jjz2Wb3zjG096rl/5yldy4403plarpbOzM0uWLMny5cvzX//1X/n3f//33H33\n3fnOd76TL3zhC/n1r3/9lONfeOGFaW1tzS233JKzzjorV1999ZOe68MPP5zPfOYz+cpXvpKbbrop\nr3/96/PpT386O3bsyD/90z/ls5/9bDo7OzN8+PDdPoc//OEP+fCHP5yPfvSjmTRp0m5/DU8++eR8\n+9vfTpI88sgj+c///M+86U1vespuN23alCVLluRrX/tarrjiimzZsqX3aueHP/zh3ivHw4YNyznn\nnJPLL788t9xyS44++ugsXbo0tVotGzZsyMKFC/Od73wnK1euzP/8z/886Tzf/e53c+edd+bb3/52\nlixZks7Ozt4ArdfrufPOOzNkyJAsXrw4P/jBD/Loo4/mP/7jP7Js2bKMHDkynZ2dWbBgQf77v/87\nu3btyjXXXJPOzs50dnamqakpDz74YJLkhBNOyIoVK/bq9wQA+54rqwDs0Y9+9KP85Cc/yXve854k\nyWOPPZaRI0fm+OOPz1/91V/liCOO6P25r3rVq9LU9Pi/hd5xxx259NJLkyQvfelL/3979xIS1dsH\ncPx7cqaMIjVTiwHtJopFomhZgpfGSgvLyWu6UEgXUkpgCFGZC3ORkZDkxgoKrIwKI9Cwq0TYYEZq\nhAmGooNoeFmMSjqX/0I6r9NMybsp/+/7++yOz3muR5Tf/J7zDKGhoXR2dqIoCqGhoS77GhgYYP36\n9b8cy6/qwXxAYzQaGRgYICMjA5gPirdt2/ZfzXf37t34+/sDcOTIEe7fv8++ffsc5trW1kZPTw/v\n3r0DYGZmht7eXvr6+jhw4ABubm54eHig1+ud2m9vb6e6uhqA2NhYYmNjHco7OzsZHh5W30e1Wq14\nenrS29uLr68vW7duBSAtLY3KykqXcygvL8fHx4f9+/er43X1DA8fPoxOp6O9vR2TyURcXBxardap\nvfT0dJYtW4afnx/h4eF0dHS47PfLly/4+fkRHBwMoGZsHz16REREBGvWrAHA39+fiYkJh7rv37/n\n4MGDaLVatFotjY2NapmiKERERODh4UF9fT1fv35lYGCA6elpwsLCuHLlCiMjI8TFxVFYWIibmxth\nYWGkpqai1+vJycnBz88PgNWrV2O325mcnMTT09PlPIQQQvx9EqwKIYRYlM1mIzc3l7y8PGA+a6fR\naJiYmGDFihXqfYqi4O7url7b7XaHdux2O1arFcCh3kKKojhlTRf6sZ3353cY5+bm1LEmJiaqmcqp\nqSm1zx9evnzJ1atXAdDr9RQVFTmUazT/+fdos9nU8Swcs81mo7S0lISEBADGx8dZtWoVVVVVDu9u\nupqLVqt1WJu+vj62bNmiXlutVsLDw9XtyN+/f2dqaorh4WGHej8+FHCloKCA1tZW6uvrycnJ+eUz\nBEhNTeXJkycMDw87rYWreSxcE1dzW8hsNmM2m1EUxWFdwfn3Q6PROPxsaGiItWvXqve+ePGCmpoa\ncnNzSU1NZXJyEoCAgACam5t58+YNr1694ubNmzQ3N1NbW0tnZyetra3k5+dz+fJlIiMj1b5+t35C\nCCH+PvkrLYQQYlFRUVE8fvyY6elpLBYLJ0+e5NmzZ073/Rx87Nq1iwcPHgAwODjIhw8fCAsLc7pv\nIX9/f0wmk3rt5ubmFGwCeHl5MTg4yOzsLJOTk3R0dKAoCjt37uT58+eMj49jt9spLy/n9u3bDnX3\n7t1LY2MjjY2NLoMzo9HIt2/fsNlsNDY2Ehsb6zTmqKgoGhoasFgsmM1msrOz6erqYs+ePTQ1NTE7\nO4vZbOb169dO7UdERNDU1ATA27dvOX/+vMNcQ0ND+fjxI/39/QDU1tZSVVVFUFAQY2NjfP78GUDd\nvutKSEgIFy5c4Nq1a4yMjPz2GSYmJtLW1sbY2Bg7duxwastut6t9mUwmOjs7iYyMdFgTjUaDxWJh\n06ZNjI+Pq1t86+rquHfv3i/HuVBkZCQtLS1YLBZmZmbIz89ndHRULW9rayMpKQmDwYC3tzft7e1Y\nLBbu3r1LTU0NiYmJlJWVMT4+zsTEBElJSQQGBlJcXEx0dLR6qJLZbMZut6tZXiGEEEuTZFaFEEIs\nKj4+np6eHjIyMrBarcTExJCSksLQ0JBDhvPnbOe5c+coKyvj4cOHKIrCxYsXWbdu3W9Pdo2Pj8do\nNBITEwPMZz5TUlLUNn4IDAwkNjaWQ4cOodPp1Pdcg4ODOXHiBLm5udhsNkJCQhb9ypif+fr6cvr0\naUZHR4mOjiY9PR2TyeTQf1ZWFv39/RgMBiwWC2lpaWrW7tOnTyQnJ+Pl5cXmzZvVtflRv6ysjLNn\nz3Lnzh1WrlypHiS1cK6VlZWcOnUKq9XKhg0bqKqqQqPRUF1dzZkzZ3Bzc2P79u2/XcuAgACys7Op\nqKigpqbG5TOE+YxxWFgYQUFBLttRFIXp6WmOHj3K3NwcFRUVeHh4OPQdFxdHQUEBN27coKqqitLS\nUubm5ggICODSpUs8ffp00XVPSEigu7sbg8GA3W4nLy+PjRs3qmPIyMigpKSElpYWfHx80Ov1mEwm\njh8/TklJCcnJyWi1WoqKivDy8iIzM5O0tDTc3d3R6XQYDAZgfht2fHz8ouMRQgjxdyn23328LYQQ\nQvxhs7OzHDt2jIaGBqdto3+C0Wikrq6O69ev//G+/xaz2UxWVha3bt3C29vbqdzVCcH/ZsXFxRQV\nFS16mrIQQoi/S7YBCyGEWFKWL19OYWGhegLvn7YwA/r/oKurC71eT2ZmpstA9X9Nd3c3Op1OAlUh\nhPgXkMyqEEIIIYQQQoglRzKrQgghhBBCCCGWHAlWhRBCCCGEEEIsORKsCiGEEEIIIYRYciRYFUII\nIYQQQgix5EiwKoQQQgghhBBiyZFgVQghhBBCCCHEkvMP3DMz2PixHlAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114061518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# on hidden layer of size 100, dropout regularization\n",
    "exp03 = theanets.Experiment(theanets.Classifier, layers=[X.shape[1], 100, 12])\n",
    "exp03.train(\n",
    "    (X_train.astype(float32),y_train),\n",
    "    (X_valid.astype(float32),y_valid),\n",
    "    algo='pretraining', learning_rate=1e-4, momentum=0.9, input_dropout=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training score: 0.828759678738\n",
      "validation score: 0.787540397045\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2U1nWd//HXNdyKSOY25TG3MA7H7M4jNiyclDr8qKa2\nO1ILBLKNNt3WasFshxTH2xWlU25GiVkdm0xcOiNp280GchaSVthNKSW72RUrAwHdakCUgbl+f3h2\n8gbki3oxn4HH4y9mhvl+39cbEZ58rmumVq/X6wEAAICCNPX1AAAAAPBUYhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4gxs1IV7enpy3nnnZf369Wlqasoll1ySAQMGpK2tLU1NTRk9enTa29tT\nq9UaNQIAAAD9VMNi9Uc/+lG2b9+eG2+8MatWrcrnPve57Ny5M7Nnz05LS0va29uzbNmyTJo0qVEj\nAAAA0E817GnAQ4cOTVdXV+r1erq6ujJo0KDcc889aWlpSZJMmDAhq1atatTtAQAA6McadrI6ZsyY\n7NixI62trfnDH/6Qa665JmvWrOn9+LBhw9LV1dWo2wMAANCPNSxWr7vuuowZMyazZs3Kxo0b84EP\nfCA7d+7s/fi2bdsyYsSIvV6nXq97XSsAAMBBpmGxun379hx66KFJkhEjRmTnzp151ateldWrV2fs\n2LFZsWJFxo8fv9fr1Gq1bN7sBHZvmpsPs6eK7Koae6rGnqqzq2rsqTq7qsaeqrGn6uyqGnuqrrn5\nsN2+v2GxOnPmzMyZMyenn356du7cmXPOOSevfvWrM3fu3HR3d2fUqFFpbW1t1O0BAADoxxoWqyNG\njMiCBQue9v6Ojo5G3RIAAIADRMO+GjAAAAA8W2IVAACA4ohVAAAAilN8rNbr9b4eAQAAgP2s+Fht\naqrnlFMGiVYAAICDSPGxmjRl5cohOf74Q3LnnYIVAADgYNAPYjVJatm4cVDOOOMQJ6wAAAAHgX4S\nqwAAABxMBvb1ANXUc+SRO3P99dtTq9X6ehgAAAAarF/E6pFH7szatUIVAADgYFH804AHDKg7UQUA\nADjIFB+rRx+dnHCCUAUAADiYFB+riVAFAAA42PSDWAUAAOBgI1YBAAAojlgFAACgOGIVAACA4ohV\nAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YB\nAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOAMbefGbb745nZ2dSZLHHnss9957b775zW/msssuS1NT\nU0aPHp329vbUarVGjgEAAEA/09CT1cmTJ6ejoyMdHR15zWtek7lz52bBggWZPXt2brjhhtTr9Sxb\ntqyRIwAAANAP7ZenAf/sZz/Lr3/965x22mm555570tLSkiSZMGFCVq1atT9GAAAAoB/ZL7G6cOHC\nnH322UmSer3e+/5hw4alq6trf4wAAABAP9LQ16wmyZ/+9KesX78+Y8eOTZI0Nf25j7dt25YRI0bs\n9RrNzYc1bL4DiT1VZ1fV2FM19lSdXVVjT9XZVTX2VI09VWdX1djTc9PwWF2zZk3GjRvX+/Zxxx2X\n1atXZ+zYsVmxYkXGjx+/12ts3uz0dW+amw+zp4rsqhp7qsaeqrOrauypOruqxp6qsafq7Koae6pu\nT1Hf8Fhdv359Xvayl/W+3dbWlrlz56a7uzujRo1Ka2tro0cAAACgn2l4rM6cOfNJb48cOTIdHR2N\nvi0AAAD92H75AksAAACwL8QqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUJyBjbz4woULs3z58nR3d2f69OkZM2ZM2tra0tTU\nlNGjR6e9vT21Wq2RIwAAANAPNexk9Y477sidd96ZRYsWpaOjI7/97W8zb968zJ49OzfccEPq9XqW\nLVvWqNsDAADQjzUsVm+//fYce+yx+ehHP5qzzjorEydOzD333JOWlpYkyYQJE7Jq1apG3R4AAIB+\nrGFPA3744YezYcOGLFy4ML/97W9z1llnpV6v93582LBh6erqatTtAQAA6McaFqsvfOELM2rUqAwc\nODDHHHNMhgwZkk2bNvV+fNu2bRkxYkSlazU3H9aoMQ8o9lSdXVVjT9XYU3V2VY09VWdX1dhTNfZU\nnV1VY0/PTcNi9cQTT8zXv/71/M3f/E0efPDBPProoxk3blxWr16dsWPHZsWKFRk/fnyla23e7AR2\nb5qbD7OniuyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7qm5PUd+wWH3Tm96UNWvW5NRTT01PT0/a\n29vz0pe+NHPnzk13d3dGjRqV1tbWRt0eAACAfqyh37rm3HPPfdr7Ojo6GnlLAAAADgAN+2rAAAAA\n8GyJVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKI\nVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNW\nAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgF\nAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUA\nAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAA\nAIojVgEAACjOwEbfYPLkyRk+fHiS5C//8i9z5plnpq2tLU1NTRk9enTa29tTq9UaPQYAAAD9SENj\n9bHHHkuSdHR09L7vrLPOyuzZs9PS0pL29vYsW7YskyZNauQYAAAA9DMNfRrwvffem+3bt2fmzJk5\n44wzctddd2XdunVpaWlJkkyYMCGrVq1q5AgAAAD0Qw09WT3kkEMyc+bMnHbaaVm/fn0+/OEPP+nj\nw4YNS1dXVyNHAAAAoB9qaKyOHDkyL3/5y3t/fPjhh+fnP/9578e3bduWESNG7PU6zc2HNWzGA4k9\nVWdX1dhTNfZUnV1VY0/V2VU19lSNPVVnV9XY03PT0Fjt7OzML37xi7S3t+fBBx/Mtm3b8oY3vCGr\nV6/O2LFjs2LFiowfP36v19m82enr3jQ3H2ZPFdlVNfZUjT1VZ1fV2FN1dlWNPVVjT9XZVTX2VN2e\nor6hsXrqqadmzpw5mTZtWpLk8ssvz+GHH565c+emu7s7o0aNSmtrayNHAAAAoB9qaKwOHDgw8+fP\nf9r7n/jVgQEAAOCpGvrVgAEAAODZEKsAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByx\nCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQq\nAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasA\nAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAcfYpVh999NFs3bp1n27w0EMP5Y1vfGPu\nu+++3H///Zk6dWqmTZuWCy+8MPV6fZ+uBQAAwMGhcqwuXrw473vf+zJlypRcddVVlT6nu7s7F1xw\nQQ455JDU6/VcfvnlmT17dm644YbU6/UsW7bsWQ8OAADAgWuPsfrLX/7ySW8vW7Yst9xyS77zne9k\n6dKllS5+5ZVXZurUqWlubk6SrFu3Li0tLUmSCRMmZNWqVc92bgAAAA5gA/f0gZtuuind3d35+7//\n+7zkJS/JK1/5ysycOTMDBw7M6NGj93rhzs7OHHHEETnppJOycOHC1Ov1Jz3td9iwYenq6np+HgUA\nAAAHlFr9GV44et9992XBggU56qij8rd/+7fZvHlzduzYkVe+8pV7vfD06dNTq9WSJPfee29GjhyZ\nn//857n77ruTJEuXLs2Pf/zjzJ079xmvM3Jksn599QcEAABA//eMsfp/fvKTn+Taa6/NuHHjMm3a\ntAwaNGifbjJjxoxcdNFFufLKK/OhD30oY8eOzQUXXJDx48fnbW972zN+7siRyZo1TmD3prn5sGze\nbE9V2FU19lSNPVVnV9XYU3V2VY09VWNP1dlVNfZUXXPzYbt9/x5fs3rDDTdk0qRJeetb35pNmzbl\nmmuuyVFHHZWzzjort9xyyz4PUKvV0tbWlquvvjpTpkzJrl270traus/XAQAA4MC3x9es3njjjfn+\n97+fHTt2ZNq0aWltbc1b3vKWTJw4Mf/yL/+yTzfp6OjY7Y8BAABgd/YYq83Nzfmnf/qnPProo3nF\nK17x508YODCnn376fhkOAACAg9MeY/Waa67JypUrM3jw4LzhDW/YnzMBAABwkNtjrA4ZMiSTJk3a\nn7MAAABAkmf4AksAAADQV8QqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAA\nQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAA\nxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAU\nR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAc\nsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHE\nKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGr\nAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wC\nAABQHLEKAABAcQY28uK7du3K+eefn/Xr16dWq+Wiiy7K4MGD09bWlqampowePTrt7e2p1WqNHAMA\nAIB+pqGxunz58jQ1NeXGG2/M6tWr89nPfjZJMnv27LS0tKS9vT3Lli3LpEmTGjkGAAAA/UxDnwY8\nadKkXHzxxUmSBx54IC94wQtyzz33pKWlJUkyYcKErFq1qpEjAAAA0A81/DWrAwYMSFtbWy677LK8\n853vTL1e7/3YsGHD0tXV1egRAAAA6Gdq9SfWYwNt2bIlp512Wh555JHccccdSZKlS5fmxz/+cebO\nnbvHzxs5Mlm/fn9MCAAAQCka+prVJUuW5MEHH8yZZ56ZoUOHpqmpKa95zWuyevXqjB07NitWrMj4\n8eP3ep3Nm52+7k1z82H2VJFdVWNP1dhTdXZVjT1VZ1fV2FM19lSdXVVjT9U1Nx+22/c3NFZbW1vT\n1taW6dOnZ+fOnTnvvPPyile8InPnzk13d3dGjRqV1tbWRo4AAABAP9TQWB06dGiuuuqqp72/o6Oj\nkbcFAACgn2v4F1gCAACAfSVWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUA\nAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAA\nAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAA\nKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACg\nOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIoj\nVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5Y\nBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDgDG3Xh7u7ufPrTn87vf//77NixI3/3d3+XUaNGpa2tLU1N\nTRk9enTa29tTq9UaNQIAAAD9VMNi9dZbb80RRxyR+fPn549//GPe/e5357jjjsvs2bPT0tKS9vb2\nLFu2LJMmTWrUCAAAAPRTDXsacGtraz7+8Y8nSXp6ejJw4MCsW7cuLS0tSZIJEyZk1apVjbo9AAAA\n/VjDYnXYsGE59NBDs3Xr1nziE5/IP/zDP6Snp+dJH+/q6mrU7QEAAOjHGvY04CTZsGFDzj777Eyb\nNi3veMc7Mn/+/N6Pbdu2LSNGjKh0nebmwxo14gHFnqqzq2rsqRp7qs6uqrGn6uyqGnuqxp6qs6tq\n7Om5aVisbtmyJR/60IfS3t6ecePGJUmOO+64rF69OmPHjs2KFSsyfvz4StfavNkJ7N40Nx9mTxXZ\nVTX2VI09VWdX1dhTdXZVjT1VY0/V2VU19lTdnqK+YbF6zTXXpKurKwsWLMiCBQuSJOedd14uu+yy\ndHd3Z9SoUWltbW3U7QEAAOjHGhar559/fs4///ynvb+jo6NRtwQAAOAA0bAvsAQAAADPllgFAACg\nOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIoj\nVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5Y\nBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIV\nAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUA\nAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAitPwWF27dm1mzJiRJLn//vszderUTJs2LRdeeGHq9Xqjbw8AAEA/1NBY/fKXv5zzzz8/\n3d3dSZLLL788s2fPzg033JB6vZ5ly5Y18vYAAAD0Uw2N1Ze//OX5whe+0HuCum7durS0tCRJJkyY\nkFWrVjXy9gAAAPRTDY3Vt7zlLRkwYEDv20982u+wYcPS1dXVyNsDAADQTw3cnzdravpzG2/bti0j\nRoyo9HnNzYc1aqQDij1VZ1fV2FM19lSdXVVjT9XZVTX2VI09VWdX1djTc7NfY/W4447L6tWrM3bs\n2KxYsSLjx4+v9HmbNzuB3Zvm5sPsqSK7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnqrbU9Tvl1it\n1WpJkra2tsydOzfd3d0ZNWpUWltb98ftAQAA6GcaHqtHH310Fi1alCQZOXJkOjo6Gn1LAAAA+rmG\nf59VAAAA2FdiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACg\nOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhi9QBQr9dTr9f7egwA\nAIDnjVjtx+r1ek45ZVBe8pJD09T0+I9FKwAAcCAQq/3YqacOzsqVQ5IMSNKUlSuH5PjjD8mddwrW\nPXEKDQAA/YNY7afq9XpWrhyUpPaE99ayceOgnHHGIYLsKZxCAwBA/yJWOSg4hQYAgP5FrPZTtVot\nJ5/cneSJsVXPkUd25/rrt6dWq+3pUw86TqEBAKD/Eav92Le+tSNHHrmz9+0jj9yZtWu354QThCoA\nANC/idV+rFar5frrt6epqZ4BA+pOVPfAKTQAAPQ/A/t6AJ6bE06o5aUvraepqeZE9Rl861s7cvzx\nA7Jx46Akfz6FFqoAAFAmJ6sHDNH1TJxCAwBA/+JklYOGU2gAAOg/nKxyEBKqAABQOrEKAABAccQq\nAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasA\nAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIA\nAFAcsQoAAEBxxCoAAADFEavA09Tr9dTr9b4eAwCAg5hYBXrV6/WccsqgvOQlh6ap6fEfi1YAAPqC\nWAV6nXrq4KxcOSTJgCRNWblySI4//pDceadg3R0n0AAAjSNWgSSPh9fKlYOS1J7w3lo2bhyUM844\nRJQ9gRNoAIDGE6sA+8gJ9L5zCg0A7CuxCiRJarVaTj65O8kTg6KeI4/szvXXb0+tVtvTpx5UnEDv\nG6fQ+0bUA8CfiVWg17e+tSNHHrmz9+0jj9yZtWu354QThCrPjlPoakQ9ADydWAV61Wq1XH/99jQ1\n1TNgQN2J6m44ga7OKXR1on7fOYUGOPCJVeBJTjihlpe+tJ6jj44T1T1wAs3zSdTvG6fQAAcPsQrs\ngfDaEyfQ1TiFphGcQgMcPMQqwLPgBLoap9B7J+qrcwoNcHARqwDPiZB4Jk6hqxH1NIrX9gL9mVgF\noKGcQu+dqK/GKXR1XtsLHAjEKgD7iZB4JqK+GqfQ1XhtL3AgEKsAUBTR9UycQu+d1/buO0+Xrs6u\n2J/EKgDQrziF5vni6dLV2RV9QawCAP2UUN0dr+2tztOlq7OrfeME+vmx32O1p6cnF1xwQaZMmZIZ\nM2bkN7/5zf4eAQDggOa1vXvn6dLV2VV1TqCfX/s9VpcuXZru7u4sWrQon/zkJzNv3rz9PQIAwAHN\na3uhbziBfn7t91j9yU9+kpNPPjlJcvzxx+fuu+/e3yMAABzwvLb3mXm6dHV2VY0T6OffwP19w61b\nt2b48OG9bw8YMCA9PT1patp9N//odyNzxIl+YZ/J7b+vpZbYUwV2VY09VWNP1dlVNfZUnV1VY0/P\nbHmS3zeeXnnrAAAT40lEQVTVsqvn8bcHNCVHDaonH+7TsYpkV9Xc94RQXZzT8qnM78Np+r/9HqvD\nhw/Ptm3bet9+plBNkqN3rt8PU/Vvf9nXA/QjdlWNPVVjT9XZVTX2VJ1dVWNPe7dxza685z2P/3jJ\nkuQvWwb07UAFs6u9m/n/dua22wbkz6er9Rx1VE+WLGnKi188oi9H65f2e6yOGTMmy5cvz9ve9rbc\nddddOfbYY/f3CAAAkCRpaRmQBx7o6yn6B7vau2XLnppXtTz++lWejVp9Pz95ul6v58ILL8wvfvGL\nJMnll1+eY445Zn+OAAAAQOH2e6wCAADA3uz3rwYMAAAAeyNWAQAAKI5YBQAAoDhFxmpPT08uuOCC\nTJkyJTNmzMhvfvObvh6paJMnT86MGTMyY8aMfPrTn+7rcYp088039+7ofe97X173utdl69atfT1W\nkXbt2pU5c+Zk6tSpOf300/OrX/2qr0cq2tq1azNjxoy+HqNIT9zN/fffn6lTp2batGm58MILfWP0\n3eju7s65556badOm5bTTTsttt93W1yMVaceOHfnUpz6VKVOmZPr06bn33nv7eqSiLVy4MFOmTMkp\np5ySm2++ua/HKVJPT0/vn3vTpk3L//zP//T1SEV76KGH8sY3vjH33XdfX49SpKf+veCHP/xhzjnn\nnD6cqH/b79+6poqlS5emu7s7ixYtytq1azNv3rx88Ytf7OuxivTYY48lSTo6Ovp4krJNnjw5kydP\nTpJcfPHFOe200zJ8+PA+nqpMy5cvT1NTU2688casXr06n/vc5/z+24Mvf/nLueWWW3LooYf29SjF\neepuLr/88syePTstLS1pb2/PsmXLMmnSpD6esiy33nprjjjiiMyfPz9//OMf8573vCcTJ07s67GK\ns3jx4gwdOjSLFi3Kfffdl3POOSednZ19PVaR7rjjjtx5551ZtGhRHnnkkVx33XV9PVKRfvSjH2X7\n9u258cYbs2rVqlx11VX5/Oc/39djFam7uzsXXHBBDjnkkL4epUhP/bPv0ksvze23355XvepVfTxZ\n/1XkyepPfvKTnHzyyUmS448/PnfffXcfT1Sue++9N9u3b8/MmTNzxhlnZO3atX09UtF+9rOf5Ve/\n+lVOO+20vh6lWJMmTcrFF1+cJHnggQfyghe8oI8nKtfLX/7yfOELX3BKuBtP3c26devS0tKSJJkw\nYUJWrVrVl+MVqbW1NR//+MeTPH7SM2CA78u3O7/+9a8zYcKEJMkxxxyTBx980DNl9uD222/Pscce\nm49+9KM566yz/OPHHgwdOjRdXV2p1+vp6urKoEGD+nqkYl155ZWZOnVqmpub+3qUIj31z74xY8Z4\nNtFzVGSsbt269UmnXgMGDEhPT08fTlSuQw45JDNnzsxXvvKVXHTRRfnkJz9pV89g4cKF+djHPtbX\nYxRvwIABaWtry6WXXpp3vOMdfT1Osd7ylrcIij146m6e+Af1sGHD0tXV1RdjFW3YsGE59NBDs3Xr\n1nziE5/IrFmz+nqkIh133HFZvnx5kuSuu+7Kww8/nEceeaSPpyrTww8/nLvvvjuf//zne/+OwNON\nGTMmO3bsSGtray644IJMnz69r0cqUmdnZ4444oicdNJJSSLAduOpf/a9/e1v78NpDgxFxurw4cOz\nbdu23rd7enrS1FTkqH1u5MiRede73tX748MPPzybN2/u46nK9Kc//Snr16/P2LFj+3qUfmHevHn5\nwQ9+kLlz5+bRRx/t63Ho5574//Bt27ZlxIgRfThNuTZs2JAzzjgj73nPe/LXf/3XfT1OkU455ZQM\nHz48p59+epYuXdr7Zx9P98IXvjAnnXRSBg4cmGOOOSZDhgzJww8/3NdjFee6667LmDFj8oMf/CDf\n/va309bWlh07dvT1WMXp7OzMqlWrMmPGjNx7771pa2vLli1b+nosDnBFFuCYMWOyYsWKJI//q+mx\nxx7bxxOVq7OzM/PmzUuS3qdCeWrG7q1Zsybjxo3r6zGKt2TJkixcuDDJ40+NqtVq/rGI5+y4447L\n6tWrkyQrVqzI61//+j6eqDxbtmzJhz70oZx77rl573vf29fjFOunP/1pxo0bl29+85t561vfmubm\n5gwePLivxyrSiSeemJUrVyZ5/O8I27dvzwtf+MI+nqo827dv732N4YgRI9Ld3e1ZarvxjW98Ix0d\nHeno6MgrX/nKXHHFFXnRi17U12NxgCvyCyy9+c1vzu23354pU6YkefwLc7B7p556aubMmZNp06Yl\neXxXwmL31q9fn5e97GV9PUbxWltb09bWlunTp2fnzp0577zz/EVwL2q1Wl+PUKz/201bW1vmzp2b\n7u7ujBo1Kq2trX08WXmuueaadHV1ZcGCBVmwYEGSx098hgwZ0seTleWYY47JrFmzsnDhwgwePDiX\nXnppX49UrDe96U1Zs2ZNTj311PT09KS9vd3/r3Zj5syZmTNnTk4//fTs3Lkz55xzToYOHdrXY9GP\nPfH3Wa1W8/vuOajVPeEcAACAwjiCAwAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAirNs2bJ0dHQkSWbMmNHH0zw3F198cW6++eZs2rQpH/nIR57x5+7rY/3Zz36228+ZMWNGVq9e\nvU/XejYWLVqUm266KUkyZ86cbNiwYY8/t7OzM3PmzHlW93k+H88999yT+fPnPy/XAqCxxCoARdmx\nY0e+/OUv5/TTT0+SrFmzpo8nem7+75vBv/jFL8611177jD/3+Xys++Ob0E+ZMiXvf//7kyR33HFH\nenp6GjbP8/V4Xv3qV2fjxo355S9/+bxcD4DGGdjXAwDQP1x77bX5/ve/n127duWkk07Kueeem9/9\n7nf58Ic/nCOOOCJDhgzJu971rnR2duYPf/hDJk6cmBkzZuS8887Lhg0bMnDgwMyaNSsnn3xyrr76\n6tx1113ZuHFjpk+fnqlTp/be55ZbbsnYsWMzYMCAXHrppUmS97///bnpppsybty4vOY1r8mWLVvy\nqU99Kl/60pd6T2Db2tryV3/1V5k8eXKWLFmSr3/96+np6cmrX/3qtLe3Z/DgwZUe5x133JEvfelL\nSZKNGzfmda97XS699NJs2rSp97EOHTo01113Xa644oqsWbMmu3btyuTJk/PBD34w9Xo9V155ZW67\n7ba86EUvyqBBg/La1742v/vd7/KBD3wgt912Wx544IHMmTMn//u//5uhQ4fm0ksvzeLFi5/0WFes\nWJGrr746O3fuzNFHH51LLrkkhx9+eG6//fbMmzcvgwYNyujRo5/xsTz00EP54Ac/mFmzZmXixIm7\n/TW86qqrUq/XM2vWrCSPn5CefPLJefvb3957nba2tgwaNCjr1q3L1q1b89GPfjTvfve7c/XVV6dW\nq2Xw4MHZtGlTzjzzzHzjG9/IunXrcsUVV6SnpycvfelL85nPfCb1ej33339/ZsyYkQ0bNmT8+PG5\n5JJLnjbz/Pnzs3Tp0gwcODDvf//784EPfKD3Y7t27Up7e3t+/etfZ8uWLTnmmGPyhS98Id3d3Zk9\ne3a2bNmSJDn77LMzceLEfO1rX8uSJUvS1NSU1772tbn44ouTJO985zvz1a9+NfPmzav03wQAfcPJ\nKgB7tWLFitxzzz351re+lZtvvjkbN27MLbfckiRZv359PvOZz+RrX/ta6vV6Nm3alG9/+9uZNWtW\nLrnkkowfPz633HJL/vmf/zmf/vSn89BDDyVJuru786//+q9PCtUkWb58eV7/+tcnSc4///wk6X2q\n6R/+8IeceeaZWbJkSQYMGPCkz6vVaqnVavnVr36VxYsXZ9GiRVmyZEmOOOKIfOUrX9mnx7t27dpc\nfPHF+d73vpfHHnss3/zmN5/0WL/61a/mpptuSq1WS2dnZxYvXpxly5blP//zP/Nv//Zvufvuu/Pd\n7343X/ziF/Ob3/zmade/6KKL0tramltvvTVnn312rrnmmic91ocffjif/exn89WvfjU333xz3vCG\nN+Qzn/lMduzYkX/8x3/M5z73uXR2dmb48OF7fAx//OMf85GPfCQf+9jHMnHixD3+Gp5yyin5zne+\nkyR55JFH8h//8R9585vf/LTdbtq0KYsXL87Xv/71XHnlldmyZUvvaedHPvKR3pPjYcOG5dxzz80V\nV1yRW2+9Nccee2yWLFmSWq2WDRs2ZMGCBfnud7+bFStW5L//+7+fdJ/vfe97ufPOO/Od73wnixcv\nTmdnZ2+A1uv13HnnnRkyZEgWLVqUH/7wh3n00Ufz7//+71m6dGmOPvrodHZ2Zv78+fmv//qv7Nq1\nK9dee206OzvT2dmZpqamPPjgg0mS17/+9Vm+fPk+/TcBwP7nZBWAvfrxj3+cn/70p3nve9+bJHns\nscdy9NFH58QTT8xf/MVf5Kijjur9ua961avS1PT4v4Xecccdueyyy/L/27uXkKjePoDj35MzZRSp\nmVpI2k0Ui2REyxIcbayssDSv6UIhXUgpgSFEZS7MRUZCkhsrKLAyKoxAw64SYWJGoxEmGIoOouFl\nMSrpXP4L6bxOMyXvpvy/7++zOz7nuR5RfvN7zjMA69evJywsDKPRiKIohIWFueyrv7+ftWvX/nIs\nv6oHcwFNW1sb/f39pKenA3NB8datW/+r+e7atYuAgAAAjhw5wv3799m7d6/DXFtbW+nu7ubdu3cA\nTE9P09PTQ29vL/v378fNzQ0PDw8MBoNT++3t7VRVVQGg1+vR6/UO5UajkaGhIfV9VKvViqenJz09\nPfj6+rJlyxYAUlNTqaiocDmHsrIyfHx82LdvnzpeV8/w8OHD+Pv7097ejslkIjY2Fq1W69ReWloa\nS5Yswc/Pj/DwcDo6Olz2++XLF/z8/AgJCQFQM7aPHj0iIiKCVatWARAQEMD4+LhD3ffv33Pw4EG0\nWi1arZaGhga1TFEUIiIi8PDwoK6ujq9fv9Lf38/U1BQ6nY4rV64wPDxMbGwsBQUFuLm5odPpSElJ\nwWAwkJ2djZ+fHwArV67EbrczMTGBp6eny3kIIYT4+yRYFUIIsSCbzUZOTg65ubnAXNZOo9EwPj7O\nsmXL1PsURcHd3V29ttvtDu3Y7XasViuAQ735FEVxyprO92M778/vMM7OzqpjTUhIUDOVk5OTap8/\nvHz5kqtXrwJgMBgoLCx0KNdo/vPv0WazqeOZP2abzUZJSQnx8fEAjI2NsWLFCiorKx3e3XQ1F61W\n67A2vb29bN68Wb22Wq2Eh4er25G/f//O5OQkQ0NDDvV+fCjgSn5+Pi0tLdTV1ZGdnf3LZwiQkpLC\nkydPGBoacloLV/OYvyau5jaf2WzGbDajKIrDuoLz74dGo3H42eDgIKtXr1bvffHiBdXV1eTk5JCS\nksLExAQAgYGBNDU18ebNG169esXNmzdpamqipqYGo9FIS0sLeXl5XL58mcjISLWv362fEEKIv0/+\nSgshhFhQVFQUjx8/ZmpqCovFwsmTJ3n27JnTfT8HHzt37uTBgwcADAwM8OHDB3Q6ndN98wUEBGAy\nmdRrNzc3p2ATwMvLi4GBAWZmZpiYmKCjowNFUdixYwfPnz9nbGwMu91OWVkZt2/fdqi7Z88eGhoa\naGhocBmctbW18e3bN2w2Gw0NDej1eqcxR0VFUV9fj8ViwWw2k5WVRWdnJ7t376axsZGZmRnMZjOv\nX792aj8iIoLGxkYA3r59y/nz5x3mGhYWxsePH+nr6wOgpqaGyspKgoODGR0d5fPnzwDq9l1XQkND\nuXDhAteuXWN4ePi3zzAhIYHW1lZGR0fZvn27U1t2u13ty2QyYTQaiYyMdFgTjUaDxWJh48aNjI2N\nqVt8a2truXfv3i/HOV9kZCTNzc1YLBamp6fJy8tjZGRELW9tbeXAgQMkJyfj7e1Ne3s7FouFu3fv\nUl1dTUJCAqWlpYyNjTE+Ps6BAwcICgqiqKiI6Oho9VAls9mM3W5Xs7xCCCEWJ8msCiGEWFBcXBzd\n3d2kp6djtVqJiYkhKSmJwcFBhwznz9nOc+fOUVpaysOHD1EUhYsXL7JmzZrfnuwaFxdHW1sbMTEx\nwFzmMykpSW3jh6CgIPR6PYcOHcLf3199zzUkJIQTJ06Qk5ODzWYjNDR0wa+M+Zmvry+nT59mZGSE\n6Oho0tLSMJlMDv1nZmbS19dHcnIyFouF1NRUNWv36dMnEhMT8fLyYtOmTera/KhfWlrK2bNnuXPn\nDsuXL1cPkpo/14qKCk6dOoXVamXdunVUVlai0WioqqrizJkzuLm5sW3btt+uZWBgIFlZWZSXl1Nd\nXe3yGcJcxlin0xEcHOyyHUVRmJqa4ujRo8zOzlJeXo6Hh4dD37GxseTn53Pjxg0qKyspKSlhdnaW\nwMBALl26xNOnTxdc9/j4eLq6ukhOTsZut5Obm8uGDRvUMaSnp1NcXExzczM+Pj4YDAZMJhPHjx+n\nuLiYxMREtFothYWFeHl5kZGRQWpqKu7u7vj7+5OcnAzMbcOOi4tbcDxCCCH+LsX+u4+3hRBCiD9s\nZmaGY8eOUV9f77Rt9E9oa2ujtraW69ev//G+/xaz2UxmZia3bt3C29vbqdzVCcH/ZkVFRRQWFi54\nmrIQQoi/S7YBCyGEWFSWLl1KQUGBegLvnzY/A/r/oLOzE4PBQEZGhstA9X9NV1cX/v7+EqgKIcS/\ngGRWhRBCCCGEEEIsOpJZFUIIIYQQQgix6EiwKoQQQgghhBBi0ZFgVQghhBBCCCHEoiPBqhBCCCGE\nEEKIRUeCVSGEEEIIIYQQi44Eq0IIIYQQQgghFp1/ALY/PM57zRXKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113fb2438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "theano_accuracy_report(exp03.network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:data/beatles/models/chords_keys_synth_theano_03.model: saved model\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:10:10 theanets.graph:470 data/beatles/models/chords_keys_synth_theano_03.model: saved model\n"
     ]
    }
   ],
   "source": [
    "exp03.save('data/beatles/models/chords_keys_synth_theano_03.model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.feedforward:layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 theanets.layers.feedforward:41 layer Input \"in\": 192 inputs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 theanets.layers.base:316 layer Feedforward \"hid1\": (in:out)192 -> 100, logistic, 19300 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"hid2\": (hid1:out)100 -> 100, logistic, 10100 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 theanets.layers.base:316 layer Feedforward \"hid2\": (hid1:out)100 -> 100, logistic, 10100 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.layers.base:layer Feedforward \"out\": (hid2:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 theanets.layers.base:316 layer Feedforward \"out\": (hid2:out)100 -> 12, softmax, 1212 parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:network has 30612 total parameters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 theanets.graph:114 network has 30612 total parameters\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.dataset:144 valid: 1083 of 1083 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.dataset:train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.dataset:144 train: 3249 of 3249 mini-batches of (32, 192); (32,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- patience = 5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:378 -- patience = 5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- validate_every = 10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:379 -- validate_every = 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- min_improvement = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:380 -- min_improvement = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:381 -- max_gradient_norm = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:382 -- max_gradient_elem = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:383 -- learning_rate = 0.0001\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- momentum = 0.9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:384 -- momentum = 0.9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:-- nesterov = False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:385 -- nesterov = False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.adaptive:220 -- rms_halflife = 14\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.adaptive:-- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.adaptive:221 -- rms_regularizer = 1e-08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling evaluation function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:11 downhill.base:112 compiling evaluation function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:compiling RMSProp function\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:12 downhill.base:118 compiling RMSProp function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 0 loss=3.702273 err=3.702273 acc=0.085440 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:13 downhill.base:226 validation 0 loss=3.702273 err=3.702273 acc=0.085440 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 1 loss=1.021887 err=1.021887 acc=0.708526\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:18 downhill.base:226 RMSProp 1 loss=1.021887 err=1.021887 acc=0.708526\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 2 loss=0.832367 err=0.832367 acc=0.749224\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:22 downhill.base:226 RMSProp 2 loss=0.832367 err=0.832367 acc=0.749224\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 3 loss=0.785123 err=0.785123 acc=0.760669\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:27 downhill.base:226 RMSProp 3 loss=0.785123 err=0.785123 acc=0.760669\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 4 loss=0.754592 err=0.754592 acc=0.768133\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:31 downhill.base:226 RMSProp 4 loss=0.754592 err=0.754592 acc=0.768133\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 5 loss=0.729357 err=0.729357 acc=0.773365\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:34 downhill.base:226 RMSProp 5 loss=0.729357 err=0.729357 acc=0.773365\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 6 loss=0.709758 err=0.709758 acc=0.778655\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:39 downhill.base:226 RMSProp 6 loss=0.709758 err=0.709758 acc=0.778655\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 7 loss=0.690384 err=0.690384 acc=0.783465\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:43 downhill.base:226 RMSProp 7 loss=0.690384 err=0.690384 acc=0.783465\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 8 loss=0.671167 err=0.671167 acc=0.789302\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:47 downhill.base:226 RMSProp 8 loss=0.671167 err=0.671167 acc=0.789302\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 9 loss=0.654722 err=0.654722 acc=0.795468\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:50 downhill.base:226 RMSProp 9 loss=0.654722 err=0.654722 acc=0.795468\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 10 loss=0.638636 err=0.638636 acc=0.799845\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:54 downhill.base:226 RMSProp 10 loss=0.638636 err=0.638636 acc=0.799845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 1 loss=0.678616 err=0.678616 acc=0.791147 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:55 downhill.base:226 validation 1 loss=0.678616 err=0.678616 acc=0.791147 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 11 loss=0.623418 err=0.623418 acc=0.804779\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:58:59 downhill.base:226 RMSProp 11 loss=0.623418 err=0.623418 acc=0.804779\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 12 loss=0.607773 err=0.607773 acc=0.810550\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:03 downhill.base:226 RMSProp 12 loss=0.607773 err=0.607773 acc=0.810550\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 13 loss=0.593460 err=0.593460 acc=0.814706\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:07 downhill.base:226 RMSProp 13 loss=0.593460 err=0.593460 acc=0.814706\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 14 loss=0.580413 err=0.580413 acc=0.818687\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:11 downhill.base:226 RMSProp 14 loss=0.580413 err=0.580413 acc=0.818687\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 15 loss=0.566488 err=0.566488 acc=0.822988\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:16 downhill.base:226 RMSProp 15 loss=0.566488 err=0.566488 acc=0.822988\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 16 loss=0.554938 err=0.554938 acc=0.826172\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:20 downhill.base:226 RMSProp 16 loss=0.554938 err=0.554938 acc=0.826172\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 17 loss=0.544403 err=0.544403 acc=0.829017\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:24 downhill.base:226 RMSProp 17 loss=0.544403 err=0.544403 acc=0.829017\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 18 loss=0.532372 err=0.532372 acc=0.833501\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:28 downhill.base:226 RMSProp 18 loss=0.532372 err=0.532372 acc=0.833501\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 19 loss=0.522735 err=0.522735 acc=0.836299\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:32 downhill.base:226 RMSProp 19 loss=0.522735 err=0.522735 acc=0.836299\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 20 loss=0.512385 err=0.512385 acc=0.838811\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:36 downhill.base:226 RMSProp 20 loss=0.512385 err=0.512385 acc=0.838811\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 2 loss=0.642438 err=0.642438 acc=0.806036 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:36 downhill.base:226 validation 2 loss=0.642438 err=0.642438 acc=0.806036 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 21 loss=0.502806 err=0.502806 acc=0.841754\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:40 downhill.base:226 RMSProp 21 loss=0.502806 err=0.502806 acc=0.841754\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 22 loss=0.494350 err=0.494350 acc=0.844235\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:44 downhill.base:226 RMSProp 22 loss=0.494350 err=0.494350 acc=0.844235\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 23 loss=0.484886 err=0.484886 acc=0.846958\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:48 downhill.base:226 RMSProp 23 loss=0.484886 err=0.484886 acc=0.846958\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 24 loss=0.477612 err=0.477612 acc=0.848641\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:53 downhill.base:226 RMSProp 24 loss=0.477612 err=0.477612 acc=0.848641\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 25 loss=0.470577 err=0.470577 acc=0.851036\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 21:59:57 downhill.base:226 RMSProp 25 loss=0.470577 err=0.470577 acc=0.851036\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 26 loss=0.462639 err=0.462639 acc=0.852864\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:02 downhill.base:226 RMSProp 26 loss=0.462639 err=0.462639 acc=0.852864\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 27 loss=0.455272 err=0.455272 acc=0.856124\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:06 downhill.base:226 RMSProp 27 loss=0.455272 err=0.455272 acc=0.856124\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 28 loss=0.449502 err=0.449502 acc=0.857807\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:11 downhill.base:226 RMSProp 28 loss=0.449502 err=0.449502 acc=0.857807\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 29 loss=0.441883 err=0.441883 acc=0.860606\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:14 downhill.base:226 RMSProp 29 loss=0.441883 err=0.441883 acc=0.860606\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 30 loss=0.434831 err=0.434831 acc=0.862165\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:18 downhill.base:226 RMSProp 30 loss=0.434831 err=0.434831 acc=0.862165\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 3 loss=0.638559 err=0.638559 acc=0.810826 *\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:19 downhill.base:226 validation 3 loss=0.638559 err=0.638559 acc=0.810826 *\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 31 loss=0.428790 err=0.428790 acc=0.863463\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:23 downhill.base:226 RMSProp 31 loss=0.428790 err=0.428790 acc=0.863463\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 32 loss=0.422871 err=0.422871 acc=0.865723\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:28 downhill.base:226 RMSProp 32 loss=0.422871 err=0.422871 acc=0.865723\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 33 loss=0.417483 err=0.417483 acc=0.867743\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:33 downhill.base:226 RMSProp 33 loss=0.417483 err=0.417483 acc=0.867743\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 34 loss=0.412464 err=0.412464 acc=0.868475\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:38 downhill.base:226 RMSProp 34 loss=0.412464 err=0.412464 acc=0.868475\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 35 loss=0.406479 err=0.406479 acc=0.870907\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:42 downhill.base:226 RMSProp 35 loss=0.406479 err=0.406479 acc=0.870907\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 36 loss=0.400405 err=0.400405 acc=0.872485\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:47 downhill.base:226 RMSProp 36 loss=0.400405 err=0.400405 acc=0.872485\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 37 loss=0.396254 err=0.396254 acc=0.873967\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:51 downhill.base:226 RMSProp 37 loss=0.396254 err=0.396254 acc=0.873967\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 38 loss=0.390471 err=0.390471 acc=0.875871\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:00:56 downhill.base:226 RMSProp 38 loss=0.390471 err=0.390471 acc=0.875871\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 39 loss=0.386346 err=0.386346 acc=0.877150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:00 downhill.base:226 RMSProp 39 loss=0.386346 err=0.386346 acc=0.877150\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 40 loss=0.382261 err=0.382261 acc=0.878891\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:04 downhill.base:226 RMSProp 40 loss=0.382261 err=0.382261 acc=0.878891\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 4 loss=0.658112 err=0.658112 acc=0.809384\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:04 downhill.base:226 validation 4 loss=0.658112 err=0.658112 acc=0.809384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 41 loss=0.377572 err=0.377572 acc=0.879555\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:10 downhill.base:226 RMSProp 41 loss=0.377572 err=0.377572 acc=0.879555\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 42 loss=0.372595 err=0.372595 acc=0.881036\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:14 downhill.base:226 RMSProp 42 loss=0.372595 err=0.372595 acc=0.881036\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 43 loss=0.368699 err=0.368699 acc=0.882615\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:18 downhill.base:226 RMSProp 43 loss=0.368699 err=0.368699 acc=0.882615\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 44 loss=0.364587 err=0.364587 acc=0.883931\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:21 downhill.base:226 RMSProp 44 loss=0.364587 err=0.364587 acc=0.883931\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 45 loss=0.360402 err=0.360402 acc=0.885865\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:25 downhill.base:226 RMSProp 45 loss=0.360402 err=0.360402 acc=0.885865\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 46 loss=0.356985 err=0.356985 acc=0.886759\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:29 downhill.base:226 RMSProp 46 loss=0.356985 err=0.356985 acc=0.886759\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 47 loss=0.352788 err=0.352788 acc=0.888482\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:33 downhill.base:226 RMSProp 47 loss=0.352788 err=0.352788 acc=0.888482\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 48 loss=0.349729 err=0.349729 acc=0.889829\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:38 downhill.base:226 RMSProp 48 loss=0.349729 err=0.349729 acc=0.889829\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 49 loss=0.345766 err=0.345766 acc=0.890425\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:43 downhill.base:226 RMSProp 49 loss=0.345766 err=0.345766 acc=0.890425\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 50 loss=0.341547 err=0.341547 acc=0.891906\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:47 downhill.base:226 RMSProp 50 loss=0.341547 err=0.341547 acc=0.891906\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 5 loss=0.672017 err=0.672017 acc=0.813193\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:47 downhill.base:226 validation 5 loss=0.672017 err=0.672017 acc=0.813193\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 51 loss=0.338920 err=0.338920 acc=0.893792\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:51 downhill.base:226 RMSProp 51 loss=0.338920 err=0.338920 acc=0.893792\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 52 loss=0.334530 err=0.334530 acc=0.894040\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:01:55 downhill.base:226 RMSProp 52 loss=0.334530 err=0.334530 acc=0.894040\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 53 loss=0.331738 err=0.331738 acc=0.894879\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:00 downhill.base:226 RMSProp 53 loss=0.331738 err=0.331738 acc=0.894879\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 54 loss=0.328154 err=0.328154 acc=0.895716\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:05 downhill.base:226 RMSProp 54 loss=0.328154 err=0.328154 acc=0.895716\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 55 loss=0.325105 err=0.325105 acc=0.897274\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:10 downhill.base:226 RMSProp 55 loss=0.325105 err=0.325105 acc=0.897274\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 56 loss=0.322188 err=0.322188 acc=0.898428\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:15 downhill.base:226 RMSProp 56 loss=0.322188 err=0.322188 acc=0.898428\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 57 loss=0.318941 err=0.318941 acc=0.899034\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:19 downhill.base:226 RMSProp 57 loss=0.318941 err=0.318941 acc=0.899034\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 58 loss=0.315622 err=0.315622 acc=0.900429\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:24 downhill.base:226 RMSProp 58 loss=0.315622 err=0.315622 acc=0.900429\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 59 loss=0.312878 err=0.312878 acc=0.901035\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:28 downhill.base:226 RMSProp 59 loss=0.312878 err=0.312878 acc=0.901035\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 60 loss=0.309936 err=0.309936 acc=0.901852\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:32 downhill.base:226 RMSProp 60 loss=0.309936 err=0.309936 acc=0.901852\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 6 loss=0.721838 err=0.721838 acc=0.810538\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:32 downhill.base:226 validation 6 loss=0.721838 err=0.721838 acc=0.810538\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 61 loss=0.307175 err=0.307175 acc=0.902929\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:36 downhill.base:226 RMSProp 61 loss=0.307175 err=0.307175 acc=0.902929\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 62 loss=0.304083 err=0.304083 acc=0.904065\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:39 downhill.base:226 RMSProp 62 loss=0.304083 err=0.304083 acc=0.904065\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 63 loss=0.301888 err=0.301888 acc=0.904440\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:43 downhill.base:226 RMSProp 63 loss=0.301888 err=0.301888 acc=0.904440\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 64 loss=0.298898 err=0.298898 acc=0.905458\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:46 downhill.base:226 RMSProp 64 loss=0.298898 err=0.298898 acc=0.905458\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 65 loss=0.295622 err=0.295622 acc=0.906747\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:50 downhill.base:226 RMSProp 65 loss=0.295622 err=0.295622 acc=0.906747\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 66 loss=0.293157 err=0.293157 acc=0.908124\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:53 downhill.base:226 RMSProp 66 loss=0.293157 err=0.293157 acc=0.908124\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 67 loss=0.290368 err=0.290368 acc=0.908047\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:02:56 downhill.base:226 RMSProp 67 loss=0.290368 err=0.290368 acc=0.908047\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 68 loss=0.289210 err=0.289210 acc=0.909124\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:00 downhill.base:226 RMSProp 68 loss=0.289210 err=0.289210 acc=0.909124\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 69 loss=0.285848 err=0.285848 acc=0.910970\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:03 downhill.base:226 RMSProp 69 loss=0.285848 err=0.285848 acc=0.910970\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 70 loss=0.283695 err=0.283695 acc=0.910836\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:07 downhill.base:226 RMSProp 70 loss=0.283695 err=0.283695 acc=0.910836\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 7 loss=0.762540 err=0.762540 acc=0.810509\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:07 downhill.base:226 validation 7 loss=0.762540 err=0.762540 acc=0.810509\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 71 loss=0.280674 err=0.280674 acc=0.912038\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:11 downhill.base:226 RMSProp 71 loss=0.280674 err=0.280674 acc=0.912038\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 72 loss=0.278904 err=0.278904 acc=0.912037\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:15 downhill.base:226 RMSProp 72 loss=0.278904 err=0.278904 acc=0.912037\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 73 loss=0.276397 err=0.276397 acc=0.914010\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:18 downhill.base:226 RMSProp 73 loss=0.276397 err=0.276397 acc=0.914010\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 74 loss=0.273606 err=0.273606 acc=0.914270\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:22 downhill.base:226 RMSProp 74 loss=0.273606 err=0.273606 acc=0.914270\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 75 loss=0.271579 err=0.271579 acc=0.914404\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:25 downhill.base:226 RMSProp 75 loss=0.271579 err=0.271579 acc=0.914404\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 76 loss=0.270386 err=0.270386 acc=0.915405\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:29 downhill.base:226 RMSProp 76 loss=0.270386 err=0.270386 acc=0.915405\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 77 loss=0.266268 err=0.266268 acc=0.916868\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:32 downhill.base:226 RMSProp 77 loss=0.266268 err=0.266268 acc=0.916868\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 78 loss=0.265151 err=0.265151 acc=0.917126\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:36 downhill.base:226 RMSProp 78 loss=0.265151 err=0.265151 acc=0.917126\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 79 loss=0.263668 err=0.263668 acc=0.918050\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:39 downhill.base:226 RMSProp 79 loss=0.263668 err=0.263668 acc=0.918050\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 80 loss=0.262251 err=0.262251 acc=0.917481\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:44 downhill.base:226 RMSProp 80 loss=0.262251 err=0.262251 acc=0.917481\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 8 loss=0.803825 err=0.803825 acc=0.809614\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:44 downhill.base:226 validation 8 loss=0.803825 err=0.803825 acc=0.809614\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 81 loss=0.259386 err=0.259386 acc=0.918963\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:48 downhill.base:226 RMSProp 81 loss=0.259386 err=0.259386 acc=0.918963\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 82 loss=0.256521 err=0.256521 acc=0.919512\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:51 downhill.base:226 RMSProp 82 loss=0.256521 err=0.256521 acc=0.919512\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 83 loss=0.255650 err=0.255650 acc=0.920599\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:55 downhill.base:226 RMSProp 83 loss=0.255650 err=0.255650 acc=0.920599\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 84 loss=0.253744 err=0.253744 acc=0.920993\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:03:58 downhill.base:226 RMSProp 84 loss=0.253744 err=0.253744 acc=0.920993\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 85 loss=0.251939 err=0.251939 acc=0.921349\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:02 downhill.base:226 RMSProp 85 loss=0.251939 err=0.251939 acc=0.921349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 86 loss=0.249794 err=0.249794 acc=0.922109\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:05 downhill.base:226 RMSProp 86 loss=0.249794 err=0.249794 acc=0.922109\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 87 loss=0.248393 err=0.248393 acc=0.922349\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:10 downhill.base:226 RMSProp 87 loss=0.248393 err=0.248393 acc=0.922349\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 88 loss=0.246391 err=0.246391 acc=0.922176\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:13 downhill.base:226 RMSProp 88 loss=0.246391 err=0.246391 acc=0.922176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 89 loss=0.243599 err=0.243599 acc=0.923638\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:18 downhill.base:226 RMSProp 89 loss=0.243599 err=0.243599 acc=0.923638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:RMSProp 90 loss=0.242535 err=0.242535 acc=0.924561\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:21 downhill.base:226 RMSProp 90 loss=0.242535 err=0.242535 acc=0.924561\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:validation 9 loss=0.851390 err=0.851390 acc=0.805171\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:21 downhill.base:226 validation 9 loss=0.851390 err=0.851390 acc=0.805171\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:downhill.base:patience elapsed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:04:21 downhill.base:402 patience elapsed!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(OrderedDict([('loss', 0.24253518197876961), ('err', 0.24253518197876961), ('acc', 0.92456133717536426)]),\n",
       " OrderedDict([('loss', 0.80382469694782821), ('err', 0.80382469694782821), ('acc', 0.80961449676823638)]))"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# two hidden layers of size 100, dropout regularization\n",
    "exp04 = theanets.Experiment(theanets.Classifier, layers=[X.shape[1], 100, 100, 12])\n",
    "exp04.train(\n",
    "    (X_train.astype(float32),y_train),\n",
    "    (X_valid.astype(float32),y_valid),\n",
    "    algo='pretraining', learning_rate=1e-4, momentum=0.9, input_dropout=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training score: 0.870946953302\n",
      "validation score: 0.810826408126\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAALPCAYAAACE8FZ9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XtwnXWdx/HPSdIrpSJj1o6w2prpYL0xVNNtR6gO09WM\n660LrPQmrnUX1mXVFnFSaQjXpVJHXRWlijoYkTI4BXEvutvITCpxSXeEClRUZikoUmhh1bQWmpKz\nfzDb5Vb6cDnNL+3r9Vdz0pznm6+W9N3fc5JavV6vBwAAAArSNNIDAAAAwFOJVQAAAIojVgEAACiO\nWAUAAKA4YhUAAIDiiFUAAACK09KoJ969e3dWrlyZe++9Ny0tLVm5cmUmTJiQzs7ONDU1Zfr06enu\n7k6tVmvUCAAAAIxSDYvVa6+9NuPHj8/atWtz9913Z/ny5ZkyZUqWL1+e9vb2dHd3p7e3N/PmzWvU\nCAAAAIxSDbsN+K677srcuXOTJNOmTcsDDzyQ//zP/0x7e3uSZO7cuenv72/U5QEAABjFGharM2bM\nyI033pgkufXWW/Pwww/nkUce2fv+iRMnZnBwsFGXBwAAYBRrWKyedNJJmTRpUhYuXJj169dn2rRp\neclLXrL3/Tt37szkyZP3+zz1er1RIwIAAFCohr1m9Wc/+1lmz56dFStW5LbbbsumTZsyderUDAwM\nZNasWenr68ucOXP2+zy1Wi3btjmB3Z/W1sPtqSK7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnqpr\nbT38GR9vWKxOmzYty5Yty5o1azJ27NhcfPHFGR4eTldXV4aGhtLW1paOjo5GXR4AAIBRrGGxesQR\nR+Sb3/zm0x7v6elp1CUBAAA4SDTsNasAAADwfIlVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAA\nAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAA\nKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACg\nOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIoj\nVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5Y\nBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKU3ys1uv1kR4BAACAA6z4\nWG1qquekk8aIVgAAgENI8bGaNGXDhnE59tgJueUWwQoAAHAoGAWxmiS1bN06JqedNsEJKwAAwCFg\nlMQqAAAAh5KWkR6gmnqmTNmTK6/clVqtNtLDAAAA0GCjIlanTNmTTZuEKgAAwKGi+NuAm5vrTlQB\nAAAOMcXH6tFHJ8cdJ1QBAAAOJQ27DXh4eDjnnHNOtmzZkqamplx44YVpbm5OZ2dnmpqaMn369HR3\nd1c4MRWqAAAAh5qGxeqPf/zj7Nq1K1dffXX6+/vzuc99Lnv27Mny5cvT3t6e7u7u9Pb2Zt68eY0a\nAQAAgFGqYbcBjx8/PoODg6nX6xkcHMyYMWNyxx13pL29PUkyd+7c9Pf3N+ryAAAAjGINO1mdOXNm\ndu/enY6Ojvzud7/L5Zdfno0bN+59/8SJEzM4ONioywMAADCKNSxWr7jiisycOTPLli3L1q1b84EP\nfCB79uzZ+/6dO3dm8uTJlZ6rtfXwRo15ULGn6uyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7emEa\nFqu7du3KYYcdliSZPHly9uzZk9e+9rUZGBjIrFmz0tfXlzlz5lR6rm3bnMDuT2vr4fZUkV1VY0/V\n2FN1dlWNPVVnV9XYUzX2VJ1dVWNP1e0r6hsWq0uXLs2KFSuycOHC7NmzJ2eddVZe97rXpaurK0ND\nQ2lra0tHR0ejLg8AAMAo1rBYnTx5ci677LKnPd7T09OoSwIAAHCQaNh3AwYAAIDnS6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUJyW\nRj75ddddl3Xr1iVJHn300dx55535zne+k4svvjhNTU2ZPn16uru7U6vVGjkGAAAAo0xDT1bnz5+f\nnp6e9PT05PWvf326urpy2WWXZfny5bnqqqtSr9fT29vbyBEAAAAYhQ7IbcC33XZb7rrrrpxyyim5\n44470t7eniSZO3du+vv7D8QIAAAAjCIHJFbXrFmTM888M0lSr9f3Pj5x4sQMDg4eiBEAAAAYRRr6\nmtUk+cMf/pAtW7Zk1qxZSZKmpv/v4507d2by5Mn7fY7W1sMbNt/BxJ6qs6tq7Kkae6rOrqqxp+rs\nqhp7qsaeqrOrauzphWl4rG7cuDGzZ8/e+/aMGTMyMDCQWbNmpa+vL3PmzNnvc2zb5vR1f1pbD7en\niuyqGnuqxp6qs6tq7Kk6u6rGnqqxp+rsqhp7qm5fUd/wWN2yZUte+cpX7n27s7MzXV1dGRoaSltb\nWzo6Oho9AgAAAKNMw2N16dKlT3p76tSp6enpafRlAQAAGMUOyDdYAgAAgOdCrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEes\nAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQnJZGPvma\nNWty4403ZmhoKIsXL87MmTPT2dmZpqamTJ8+Pd3d3anVao0cAQAAgFGoYSerN998c2655ZasXbs2\nPT09+fWvf51Vq1Zl+fLlueqqq1Kv19Pb29uoywMAADCKNSxWb7rpphxzzDH5yEc+kjPOOCMnnnhi\n7rjjjrS3tydJ5s6dm/7+/kZdHgAAgFGsYbcBP/zww7n//vuzZs2a/PrXv84ZZ5yRer2+9/0TJ07M\n4OBgoy4PAADAKNawWH3pS1+atra2tLS0ZNq0aRk3blwefPDBve/fuXNnJk+eXOm5WlsPb9SYBxV7\nqs6uqrGnauypOruqxp6qs6tq7Kkae6rOrqqxpxemYbH6pje9Kd/61rfy13/913nggQfyyCOPZPbs\n2RkYGMisWbPS19eXOXPmVHqubducwO5Pa+vh9lSRXVVjT9XYU3V2VY09VWdX1dhTNfZUnV1VY0/V\n7SvqGxarb3vb27Jx48acfPLJGR4eTnd3d4466qh0dXVlaGgobW1t6ejoaNTlAQAAGMUa+qNrzj77\n7Kc91tPT08hLAgAAcBBo2HcDBgAAgOdLrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAcVoafYH58+dn0qRJSZI//dM/zemn\nn57Ozs40NTVl+vTp6e7uTq1Wa/QYAAAAjCINjdVHH300SdLT07P3sTPOOCPLly9Pe3t7uru709vb\nm3nz5jVyDAAAAEaZht4GfOedd2bXrl1ZunRpTjvttNx6663ZvHlz2tvbkyRz585Nf39/I0cAAABg\nFGroyeqECROydOnSnHLKKdmyZUs+/OEPP+n9EydOzODgYCNHAAAAYBRqaKxOnTo1r3rVq/b++ogj\njsjPf/7zve/fuXNnJk+evN/naW09vGEzHkzsqTq7qsaeqrGn6uyqGnuqzq6qsadq7Kk6u6rGnl6Y\nhsbqunXr8otf/CLd3d154IEHsnPnzrzlLW/JwMBAZs2alb6+vsyZM2e/z7Ntm9PX/WltPdyeKrKr\nauypGnuqzq6qsafq7Koae6rGnqqzq2rsqbp9RX1DY/Xkk0/OihUrsmjRoiTJJZdckiOOOCJdXV0Z\nGhpKW1tbOjo6GjkCAAAAo1BDY7WlpSWrV69+2uNP/O7AAAAA8FQN/W7AAAAA8HyIVQAAAIojVgEA\nACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAA\noDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA\n4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiPKdYfeSR\nR7Jjx45GzQIAAABJkpaqv/Haa69NT09PhoeHM2/evHz84x9v5FwAAAAcwvZ5svrLX/7ySW/39vbm\nhhtuyD//8z9n/fr1DR8MAACAQ9c+T1avueaaDA0N5e///u/z8pe/PK95zWuydOnStLS0ZPr06Qdy\nRgAAAA4x+4zVrq6u3H333Vm9enVe8YpX5G/+5m+ybdu27N69O695zWsO5IwAAAAcYp71GyxNmzYt\nn/nMZ/K2t70tZ599dvr6+tLW1nagZgMAAOAQtc9YveqqqzJv3ry84x3vyIMPPpjLL788r3jFK3LG\nGWfkhhtuOJAzAgAAcIjZZ6xeffXV+cEPfpDrrrsua9asSZK8/e1vz5o1a/z4GgAAABpqn69ZbW1t\nzT/+4z/mkUceyatf/er//4CWlixcuPCADAcAAMChaZ+xevnll2fDhg0ZO3Zs3vKWtxzImQAAADjE\n7TNWx40bl3nz5h3IWQAAACDJfr4bMAAAAIwEsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxGh6rDz30UN761rfm7rvvzj333JMFCxZk0aJFOe+881Kv1xt9eQAAAEahhsbq\n0NBQzj333EyYMCH1ej2XXHJJli9fnquuuir1ej29vb2NvDwAAACjVENj9dJLL82CBQvS2tqaJNm8\neXPa29uTJHPnzk1/f38jLw8AAMAo1bBYXbduXY488sgcf/zxSZJ6vf6k234nTpyYwcHBRl0eAACA\nUaxWb9ALRxcvXpxarZYkufPOOzN16tT8/Oc/z+23354kWb9+fX7yk5+kq6vrWZ9n6tRky5ZGTAgA\nAECpWhr1xN/+9rf3/nrJkiU5//zzc+mll2ZgYCCzZs1KX19f5syZU+m5tm1zArs/ra2H21NFdlWN\nPVVjT9XZVTX2VJ1dVWNP1dhTdXZVjT1V19p6+DM+3rBYfaparZbOzs50dXVlaGgobW1t6ejoOFCX\nBwAAYBQ5ILHa09PzjL8GAACAZ9Lwn7MKAAAAz5VYBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUA\nAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAA\ngOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAA\niiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAo\njlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4\nYhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKI\nVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNW\nAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDitDTyyR977LGsXLkyW7Zs\nSa1Wy/nnn5+xY8ems7MzTU1NmT59erq7u1Or1Ro5BgAAAKNMQ2P1xhtvTFNTU66++uoMDAzks5/9\nbJJk+fLlaW9vT3d3d3p7ezNv3rxGjgEAAMAo09DbgOfNm5cLLrggSXLfffflJS95Se644460t7cn\nSebOnZv+/v5GjgAAAMAo1PDXrDY3N6ezszMXX3xx3v3ud6der+9938SJEzM4ONjoEQAAABhlavUn\n1mMDbd++Paecckr++Mc/5uabb06SrF+/Pj/5yU/S1dW1z4+bOjXZsuVATAgAAEApGvqa1euvvz4P\nPPBATj/99IwfPz5NTU15/etfn4GBgcyaNSt9fX2ZM2fOfp9n2zanr/vT2nq4PVVkV9XYUzX2VJ1d\nVWNP1dlVNfZUjT1VZ1fV2FN1ra2HP+PjDY3Vjo6OdHZ2ZvHixdmzZ0/OOeecvPrVr05XV1eGhobS\n1taWjo6ORo4AAADAKNTQWB0/fnw+//nPP+3xnp6eRl4WAACAUa7h32AJAAAAniuxCgAAQHHEKgAA\nAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAA\nFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQ\nHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBx\nxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMUR\nqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEes\nAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEK\nAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxWhr1\nxENDQ/nUpz6V3/72t9m9e3f+7u/+Lm1tbens7ExTU1OmT5+e7u7u1Gq1Ro0AAADAKNWwWP3+97+f\nI488MqtXr87vf//7vPe9782MGTOyfPnytLe3p7u7O729vZk3b16jRgAAAGCUathtwB0dHfnoRz+a\nJBkeHk5LS0s2b96c9vb2JMncuXPT39/fqMsDAAAwijUsVidOnJjDDjssO3bsyMc+9rF8/OMfz/Dw\n8JPePzg42KjLAwAAMIo17DbgJLn//vtz5plnZtGiRXnXu96V1atX733fzp07M3ny5ErP09p6eKNG\nPKjYU3V2VY09VWNP1dlVNfZUnV1VY0/V2FN1dlWNPb0wDYvV7du350Mf+lC6u7sze/bsJMmMGTMy\nMDCQWbNmpa+vL3PmzKn0XNu2OYHdn9bWw+2pIruqxp6qsafq7Koae6rOrqqxp2rsqTq7qsaeqttX\n1DcsVi+//PIMDg7msssuy2WXXZYkOeecc3LxxRdnaGgobW1t6ejoaNTlAQAAGMUaFqsrV67MypUr\nn/Z4T09Poy4JAADAQaJh32AJAAAAni+xCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoA\nAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAA\nUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABA\nccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAAADF\nEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByxCgAAQHHEKgAAAMURqwAAABRH\nrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAgAAUByx\nCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxGh6rmzZtypIlS5Ik99xzTxYsWJBFixblvPPO\nS71eb/TlAQAAGIUaGqtf+9rXsnLlygwNDSVJLrnkkixfvjxXXXVV6vV6ent7G3l5AAAARqmGxuqr\nXvWqfOlLX9p7grp58+a0t7cnSebOnZv+/v5GXh4AAIBRqqGx+va3vz3Nzc17337ibb8TJ07M4OBg\nIy8PAADAKNVyIC/W1PT/bbxz585Mnjy50se1th7eqJEOKvZUnV1VY0/V2FN1dlWNPVVnV9XYUzX2\nVJ1dVWNPL8wBjdUZM2ZkYGAgs2bNSl9fX+bMmVPp47ZtcwK7P62th9tTRXZVjT1VY0/V2VU19lSd\nXVVjT9XYU3V2VY09VbevqD8gsVqr1ZIknZ2d6erqytDQUNra2tLR0XEgLg8AAMAo0/BYPfroo7N2\n7dokydSpU9PT09PoSwIAADDKNfznrAIAAMBzJVYBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWAQAA\nKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDiiFUAAACKI1YBAAAojlgFAACg\nOGIVAACA4ohVAAAAiiNWAQAAKI5YBQAAoDhiFQAAgOKIVQAAAIojVgEAACiOWAUAAKA4YhUAAIDi\niFUAAACKI1YBAAAojlgFAACgOGIVAACA4ohVAAAAiiNWDwL1ej31en2kxwAAAHjRiNVRrF6v56ST\nxuTlLz8sTU2P/1q0AgAABwOxOoqdfPLYbNgwLklzkqZs2DAuxx47IbfcIlj3xSk0AACMDmJ1lKrX\n69mwYUyS2hMerWXr1jE57bQJguwpnEIDAMDoIlY5JDiFBgCA0UWsjlK1Wi0nnDCU5ImxVc+UKUO5\n8spdqdVq+/rQQ45TaAAAGH3E6ij23e/uzpQpe/a+PWXKnmzatCvHHSdUAQCA0U2sjmK1Wi1XXrkr\nTU31NDfXnajug1NoAAAYfVpGegBemOOOq+Woo+ppaqo5UX0W3/3u7hx7bHO2bh2T5P9PoYUqAACU\nycnqQUN0PRun0AAAMLo4WeWQ4RQaAABGDyerHIKEKgAAlE6sAgAAUByxCgAAQHHEKgAAAMURqwAA\nABRHrAIAAFAcsQoAAEBxxCoAAADFEasAAAAUR6wCAABQHLEKAABAccQqAAAAxRGrAAAAFEesAk9T\nr9dTr9dHegwAAA5hYhXYq16v56STxuTlLz8sTU2P/1q0AgAwEsQqsNfJJ4/Nhg3jkjQnacqGDeNy\n7LETcsstgvWZOIGuzq4AgOdKrAJJHo+JDRvGJKk94dFatm4dk9NOmyA0nsAJdHV2BQA8X2IV4Dly\nAl2dXQEAz5dYBZIktVotJ5wwlOSJEVHPlClDufLKXanVavv60EOKE+jq7Oq5c7t0dXYFcPATq8Be\n3/3u7kyZsmfv21Om7MmmTbty3HFCFRrJ7dLV2RXAoUOsAnvVarVceeWuNDXV09xcd6L6DJxAV2dX\n1bldujqRJWljAAAVF0lEQVS7Ajh0iFXgSY47rpajjqrn6KPjRHUfnEBXZ1f753bp6uzquXO7NDCa\niVVgH8TEvjiBrs6uYGS4XRo4GIhVgOfBCXR1dvXs3C5dnV1V53Zp4GAgVgFeEH85rs6u9sXt0tXZ\n1f65XRo4WIhVABhhbpeuzq5oBK/trc6uqrGnF4dYBYACuF26Ort6dm6Xrs5re6uzq2rs6cV1wGN1\neHg45557bk499dQsWbIk995774EeAQAKJiSqs6t9cbt0NV7bW51dVWNPL64DHqvr16/P0NBQ1q5d\nm0984hNZtWrVgR4BAOCg5nbp/fPa3ursqhp7evG1HOgL/vSnP80JJ5yQJDn22GNz++23H+gRAAAO\nev93u3RTU82JKjAqHfBY3bFjRyZNmrT37ebm5gwPD6ep6ZkPeX/8m6k58k3+FeLZ3PTbWmqJPVVg\nV9XYUzX2VJ1dVWNP1dlVNfb07H47Lnn00SeHfHNT8rJ6PWPfPEJDFcquqnninq7NKflkLs2UKXvc\n3fA8HfBYnTRpUnbu3Ln37WcL1SQ5es+WAzDV6PanIz3AKGJX1dhTNfZUnV1VY0/V2VU19vTsptTr\nOfro4fz2t81Jkle84rH85jdNouIZ2FU1z7ynltRqk0d4stHpgL9mdebMmenr60uS3HrrrTnmmGMO\n9AgAAJBarZb77mtOvZ7U68l99zWLr32wq2rs6cVVqx/gV/rW6/Wcd955+cUvfpEkueSSSzJt2rQD\nOQIAAACFO+CxCgAAAPtzwG8DBgAAgP0RqwAAABRHrAIAAFCcImN1eHg45557bk499dQsWbIk9957\n70iPVLT58+dnyZIlWbJkST71qU+N9DhFuu666/bu6K/+6q/yxje+MTt27BjpsYr02GOPZcWKFVmw\nYEEWLlyYX/3qVyM9UtE2bdqUJUuWjPQYRXribu65554sWLAgixYtynnnnRffLuHphoaGcvbZZ2fR\nokU55ZRT8qMf/WikRyrS7t2788lPfjKnnnpqFi9enDvvvHOkRyramjVrcuqpp+akk07KddddN9Lj\nFGl4eHjv171Fixblv//7v0d6pKI99NBDeetb35q77757pEcp0lP/XvAf//EfOeuss0ZwotHtgP+c\n1SrWr1+foaGhrF27Nps2bcqqVavy5S9/eaTHKtKjjz6aJOnp6RnhSco2f/78zJ8/P0lywQUX5JRT\nTsmkSZNGeKoy3XjjjWlqasrVV1+dgYGBfO5zn/Pnbx++9rWv5YYbbshhhx020qMU56m7ueSSS7J8\n+fK0t7enu7s7vb29mTdv3ghPWZbvf//7OfLII7N69er8/ve/z/ve976ceOKJIz1Wca699tqMHz8+\na9euzd13352zzjor69atG+mxinTzzTfnlltuydq1a/PHP/4xV1xxxUiPVKQf//jH2bVrV66++ur0\n9/fn85//fL7whS+M9FhFGhoayrnnnpsJEyaM9ChFeurXvosuuig33XRTXvva147wZKNXkSerP/3p\nT3PCCSckSY499tjcfvvtIzxRue68887s2rUrS5cuzWmnnZZNmzaN9EhFu+222/KrX/0qp5xyykiP\nUqx58+blggsuSJLcd999eclLXjLCE5XrVa96Vb70pS85JXwGT93N5s2b097eniSZO3du+vv7R3K8\nInV0dOSjH/1oksdPepqbm0d4ojLdddddmTt3bpJk2rRpeeCBB9wpsw833XRTjjnmmHzkIx/JGWec\n4R8/9mH8+PEZHBxMvV7P4OBgxowZM9IjFevSSy/NggUL0traOtKjFOmpX/tmzpzpbqIXqMhY3bFj\nx5NOvZqbmzM8PDyCE5VrwoQJWbp0ab7+9a/n/PPPzyc+8Qm7ehZr1qzJP/zDP4z0GMVrbm5OZ2dn\nLrroorzrXe8a6XGK9fa3v11Q7MNTd/PEL9QTJ07M4ODgSIxVtIkTJ+awww7Ljh078rGPfSzLli0b\n6ZGKNGPGjNx4441JkltvvTUPP/xw/vjHP47wVGV6+OGHc/vtt+cLX/jC3r8j8HQzZ87M7t2709HR\nkXPPPTeLFy8e6ZGKtG7duhx55JE5/vjjk0SAPYOnfu175zvfOYLTHByKjNVJkyZl586de98eHh5O\nU1ORo464qVOn5j3vec/eXx9xxBHZtm3bCE9Vpj/84Q/ZsmVLZs2aNdKjjAqrVq3KD3/4w3R1deWR\nRx4Z6XEY5Z743/CdO3dm8uTJIzhNue6///6cdtpped/73pe/+Iu/GOlxinTSSSdl0qRJWbhwYdav\nX7/3ax9P99KXvjTHH398WlpaMm3atIwbNy4PP/zwSI9VnCuuuCIzZ87MD3/4w3zve99LZ2dndu/e\nPdJjFWfdunXp7+/PkiVLcuedd6azszPbt28f6bE4yBVZgDNnzkxfX1+Sx//V9Jhjjhnhicq1bt26\nrFq1Kkn23grl1oxntnHjxsyePXukxyje9ddfnzVr1iR5/NaoWq3mH4t4wWbMmJGBgYEkSV9fX978\n5jeP8ETl2b59ez70oQ/l7LPPzl/+5V+O9DjF+tnPfpbZs2fnO9/5Tt7xjnektbU1Y8eOHemxivSm\nN70pGzZsSPL43xF27dqVl770pSM8VXl27dq19zWGkydPztDQkLvUnsG3v/3t9PT0pKenJ695zWvy\n6U9/Oi972ctGeiwOckV+g6U///M/z0033ZRTTz01yePfmINndvLJJ2fFihVZtGhRksd3JSye2ZYt\nW/LKV75ypMcoXkdHRzo7O7N48eLs2bMn55xzjr8I7ketVhvpEYr1f7vp7OxMV1dXhoaG0tbWlo6O\njhGerDyXX355BgcHc9lll+Wyyy5L8viJz7hx40Z4srJMmzYty5Yty5o1azJ27NhcdNFFIz1Ssd72\ntrdl48aNOfnkkzM8PJzu7m7/vXoGS5cuzYoVK7Jw4cLs2bMnZ511VsaPHz/SYzGKPfHPWa1W8+fu\nBajV3XAOAABAYRzBAQAAUByxCgAAQHHEKgAAAMURqwAAABRHrAIAAFAcsQoAAEBxxCoAxent7U1P\nT0+SZMmSJSM8zQtzwQUX5LrrrsuDDz6Yv/3bv33W3/tcP9fbbrvtGT9myZIlGRgYeE7P9XysXbs2\n11xzTZJkxYoVuf/++/f5e9etW5cVK1Y8r+u8mJ/PHXfckdWrV78ozwVAY4lVAIqye/fufO1rX8vC\nhQuTJBs3bhzhiV6Y//th8H/yJ3+Sr371q8/6e1/Mz/VA/BD6U089Ne9///uTJDfffHOGh4cbNs+L\n9fm87nWvy9atW/PLX/7yRXk+ABqnZaQHAGB0+OpXv5of/OAHeeyxx3L88cfn7LPPzm9+85t8+MMf\nzpFHHplx48blPe95T9atW5ff/e53OfHEE7NkyZKcc845uf/++9PS0pJly5blhBNOyBe/+MXceuut\n2bp1axYvXpwFCxbsvc4NN9yQWbNmpbm5ORdddFGS5P3vf3+uueaazJ49O69//euzffv2fPKTn8xX\nvvKVvSewnZ2d+bM/+7PMnz8/119/fb71rW9leHg4r3vd69Ld3Z2xY8dW+jxvvvnmfOUrX0mSbN26\nNW984xtz0UUX5cEHH9z7uY4fPz5XXHFFPv3pT2fjxo157LHHMn/+/Hzwgx9MvV7PpZdemh/96Ed5\n2cteljFjxuQNb3hDfvOb3+QDH/hAfvSjH+W+++7LihUr8j//8z8ZP358Lrroolx77bVP+lz7+vry\nxS9+MXv27MnRRx+dCy+8MEcccURuuummrFq1KmPGjMn06dOf9XN56KGH8sEPfjDLli3LiSee+Iz/\nG37+859PvV7PsmXLkjx+QnrCCSfkne98597n6ezszJgxY7J58+bs2LEjH/nIR/Le9743X/ziF1Or\n1TJ27Ng8+OCDOf300/Ptb387mzdvzqc//ekMDw/nqKOOymc+85nU6/Xcc889WbJkSe6///7MmTMn\nF1544dNmXr16ddavX5+Wlpa8//3vzwc+8IG973vsscfS3d2du+66K9u3b8+0adPypS99KUNDQ1m+\nfHm2b9+eJDnzzDNz4okn5pvf/Gauv/76NDU15Q1veEMuuOCCJMm73/3ufOMb38iqVasq/X8CgJHh\nZBWA/err68sdd9yR7373u7nuuuuydevW3HDDDUmSLVu25DOf+Uy++c1vpl6v58EHH8z3vve9LFu2\nLBdeeGHmzJmTG264If/0T/+UT33qU3nooYeSJENDQ/mXf/mXJ4Vqktx4441585vfnCRZuXJlkuy9\n1fR3v/tdTj/99Fx//fVpbm5+0sfVarXUarX86le/yrXXXpu1a9fm+uuvz5FHHpmvf/3rz+nz3bRp\nUy644IL827/9Wx599NF85zvfedLn+o1vfCPXXHNNarVa1q1bl2uvvTa9vb35r//6r/z7v/97br/9\n9vzrv/5rvvzlL+fee+992vOff/756ejoyPe///2ceeaZufzyy5/0uT788MP57Gc/+7/t3VlIVH8b\nwPHv0Zl2MrOysrQVwyKxtCzBpWmxotI0tbpQaIEoI7CFqKyLDHIiIamLFiHB9qaJRKPdImwwKy3C\nAiOpQbRSg0lLZ+a8F9J5nWZM/jfl/32fz92Zc37rGRye8/zOTwoKCrh27RpRUVEcOXKE9vZ2du3a\nRV5eHiaTiUGDBnU7hq9fv7Jx40YyMzOZN29et/cwKSmJ4uJiAFpbW3ny5AkLFixwm9vGxkYuX75M\nYWEhubm5fP78Wct2bty4UcscDxgwgB07dnD48GFu3LhBcHAwZrMZRVGor6/n+PHjlJSU8PDhQ2pr\na13aKS0t5fnz5xQXF3P58mVMJpMWgKqqyvPnz+nbty8XLlzg9u3bfP/+nbKyMu7cucOYMWMwmUwY\njUYqKytxOBycPHkSk8mEyWTCy8uLhoYGAMLDw7l///4/+k4IIYT48ySzKoQQokfl5eVUV1ezcuVK\nAH78+MGYMWOYOXMmfn5+jB49Wrs2JCQEL6/OZ6EWi4WcnBwAxo4dS2hoKFVVVSiKQmhoqMe26urq\nGDlyZLd96a4cdAY0FouFuro6UlJSgM6geOrUqf9ovHPmzCEwMBCAFStWcOnSJRYsWOAy1vLycmpq\nanjy5AkAbW1tvH37ltraWhYtWoS3tzc+Pj4YDAa3+isqKsjLywMgJiaGmJgYl/NVVVXU19dr76M6\nHA6GDBnC27dvGTFiBJMmTQIgOTmZQ4cOeRzDgQMHGD58OAsXLtT66+keLl++nICAACoqKrBarcTG\nxqLX693qW7VqFV5eXvj7+zNjxgwqKys9tvvmzRv8/f2ZMmUKgJaxNZlMhIeHM3jwYAACAwNpbm52\nKfv06VOWLFmCXq9Hr9djNpu1c4qiEB4ejo+PD0VFRbx79466ujpaW1sJCwvj6NGjNDQ0EBsby6ZN\nm/D29iYsLIykpCQMBgNr167F398fgEGDBqGqKi0tLQwZMsTjOIQQQvx9EqwKIYTokdPpJD09nYyM\nDKAza6fT6WhubqZv377adYqi0K9fP+1YVVWXelRVxeFwALiU60pRFLesaVc/l/P++g5jR0eH1tf4\n+HgtU/nt2zetzZ/u3bvHsWPHADAYDGRmZrqc1+n++/PodDq1/nTts9PpZOfOncyfPx+ApqYmBg4c\niNFodHl309NY9Hq9y9zU1tYyceJE7djhcDBjxgxtOfKPHz/49u0b9fX1LuV+PhTwZMOGDZSVlVFU\nVMTatWu7vYcASUlJ3Lhxg/r6ere58DSOrnPiaWxd2Ww2bDYbiqK4zCu4fz90Op3LZx8/fmTo0KHa\ntXfv3iU/P5/09HSSkpJoaWkBICgoiNLSUh49esT9+/cpKCigtLSUEydOUFVVRVlZGevXr+fIkSNE\nRERobf1u/oQQQvx98ldaCCFEjyIjI7l+/Tqtra3Y7Xa2bNnC7du33a77NfiYPXs2V65cAeDDhw88\ne/aMsLAwt+u6CgwMxGq1asfe3t5uwSaAr68vHz58oL29nZaWFiorK1EUhVmzZnHnzh2amppQVZUD\nBw5QWFjoUnbevHmYzWbMZrPH4MxisfDp0yecTidms5mYmBi3PkdGRnLx4kXsdjs2m401a9ZQXV3N\n3LlzKSkpob29HZvNxoMHD9zqDw8Pp6SkBIDHjx+zb98+l7GGhoby4sUL3r9/D8CJEycwGo0EBwfz\n5csXXr9+DaAt3/UkJCSE/fv3c/z4cRoaGn57D+Pj4ykvL+fLly9Mnz7drS5VVbW2rFYrVVVVRERE\nuMyJTqfDbrczfvx4mpqatCW+p06d4sKFC932s6uIiAhu3bqF3W6nra2N9evX09jYqJ0vLy9n8eLF\nJCYm4ufnR0VFBXa7nfPnz5Ofn098fDzZ2dk0NTXR3NzM4sWLmTx5Mlu3biUqKkrbVMlms6Gqqpbl\nFUII0TtJZlUIIUSP4uLiqKmpISUlBYfDQXR0NAkJCXz8+NElw/lrtnPv3r1kZ2dz9epVFEUhJyeH\nYcOG/XZn17i4OCwWC9HR0UBn5jMhIUGr46fJkycTExPD0qVLCQgI0N5znTJlCps3byY9PR2n00lI\nSEiP/zLmVyNGjGD79u00NjYSFRXFqlWrsFqtLu2npaXx/v17EhMTsdvtJCcna1m7V69esWzZMnx9\nfZkwYYI2Nz/LZ2dns2fPHs6dO0f//v21jaS6jvXQoUNs27YNh8PBqFGjMBqN6HQ68vLy2L17N97e\n3kybNu23cxkUFMSaNWs4ePAg+fn5Hu8hdGaMw8LCCA4O9liPoii0traycuVKOjo6OHjwID4+Pi5t\nx8bGsmHDBs6cOYPRaGTnzp10dHQQFBREbm4uN2/e7HHe58+fz8uXL0lMTERVVTIyMhg3bpzWh5SU\nFLKysrh16xbDhw/HYDBgtVpZt24dWVlZLFu2DL1eT2ZmJr6+vqSmppKcnEy/fv0ICAggMTER6FyG\nHRcX12N/hBBC/F2K+rvH20IIIcQf1t7ezurVq7l48aLbstE/wWKxcOrUKU6fPv3H2/5bbDYbaWlp\nnD17Fj8/P7fznnYI/jfbunUrmZmZPe6mLIQQ4u+SZcBCCCF6lT59+rBp0yZtB94/rWsG9P9BdXU1\nBoOB1NRUj4Hq/5qXL18SEBAggaoQQvwLSGZVCCGEEEIIIUSvI5lVIYQQQgghhBC9jgSrQgghhBBC\nCCF6HQlWhRBCCCGEEEL0OhKsCiGEEEIIIYTodSRYFUIIIYQQQgjR60iwKoQQQgghhBCi1/kPr0TT\npZqY2ksAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115ed5c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "theano_accuracy_report(exp04.network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:theanets.graph:data/beatles/models/chords_keys_synth_theano_04.model: saved model\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I 2015-09-22 22:10:21 theanets.graph:470 data/beatles/models/chords_keys_synth_theano_04.model: saved model\n"
     ]
    }
   ],
   "source": [
    "exp04.save('data/beatles/models/chords_keys_synth_theano_04.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that neural network models are able to squeeze even more from the data. By training a few models using theano (up to to hidden layers with autoencoder pretraining and input dropout regularization) we were able to improve the accuracy by 10% points relative to the logistic regression (71% to 81%).\n",
    "\n",
    "The error analysis still show that the models have mostly problems with discriminating keys close on the circle of fifths."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "We could see that by using features designed with a bit care and a few rather simple models we were able to get past 80% performance of classifying key from chord sequences. This is quite a nice result for a few evenings of work.\n",
    "\n",
    "### Where to go from now?\n",
    "\n",
    "- we could better analyze the input data which exhibited errors and looks if there is anything to improve\n",
    "- we could try to tune the models more\n",
    "- we could try RNN models trained directly on the sequences of chords for whole songs (no fixed-size window contexts)\n",
    "- we could try to apply the models to new datasets (other bands or gengres) and look for the performace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}